Commit Graph

329 Commits

Author SHA1 Message Date
Gabe Ruttner 6c3a954553 Merge branch 'feat-durable-execution' into feat/durable-complex-tests 2026-03-13 11:20:29 -07:00
Gabe Ruttner 1244039e64 Fix--durable-feedback (#3240)
* feat: duration takes ms

* fix: nondeterminism err clarity

* fix: cleanup

* chore: lint

* chore: feedback

* fix: typescript

* fix: python
2026-03-13 10:04:03 -07:00
Gabe Ruttner 583bb6f4ba Feat--unify-running-and-evicted (#3248)
* feat: unify running and evicted counts with filter

* chore: gen

* chore: gen lint

* fix: opsies...

* chore: gen

* chore: feedback
2026-03-13 05:57:07 -07:00
matt 1986435d6e Feat: Simple TTL Cache (#3220)
* feat: basic ttl cache

* feat: use ttl cache for buffered completions

* fix: don't inherit from dict

* feat: ts

* fix: stop eviction job on shutdown

* fix: add cache size
2026-03-12 17:08:40 -04:00
matt 6fb50ce004 Feat: Wait wrapper signature improvements (#3241)
* fix: make memo private for now

* feat: first pass at returning structured data back

* fix: json marshalling, types

* feat: wiring

* fix: sleep for return type

* chore: add comments

* fix: tests for `expr_to_timedelta`

* fix: change examples

* feat: improve asserts

* fix: pass payloads through

* fix: add hacky fallbacks to avoid breaking changes

* feat: add memo now

* feat: add test for memo now

* fix: hack for old engine test

* fix: rm restore button

* fix: one more test

* fix: ts impls

* revert: unwind engine changes (woohoo!)

* fix: payload validation

* fix: ts

* chore: gen

* feat: sleep until

* fix: return wait result on replay

* fix: data get, pydantic warning

* fix: test assert

* fix: ts memo key, make memo private

* chore: rm comment, gen again

* chore: merge and gen
2026-03-12 12:53:03 -04:00
mrkaye97 c0204fbd8b Merge branch 'main' into feat-durable-execution 2026-03-12 12:24:32 -04:00
Gabe Ruttner 3bd565c0e8 fix: durable evict at capacity (#3221)
* fix: durable evict at capacity

* tests: capacity eviction

* chore: lint

* tests: change fixture
2026-03-11 17:16:30 -07:00
Jishnu c77bfa9274 feat: add logs client for TS SDK (#3237)
* feat: add logs client for typescript

* docs: add ts logs docs

* fix: lint

* bump: version, changelog

* fix: lint
2026-03-11 17:30:04 -04:00
Gabe Ruttner 98212cb2de Merge branch 'main' into feat-durable-execution 2026-03-10 09:01:01 -04:00
Mohammed Nafees ab6cae4420 Test PgBouncer against our code-level partitioning logic (#3178)
* test partition drops with pgbouncer

* dedicated DDLPool to bypass pgbouncer

* fix generate

* add env var

* min env var

* update documentation

* no go mod redundancy
2026-03-10 12:24:07 +01:00
matt 918f18f127 Fix: Clean up durable exec todos (#3214)
* fix: comments

* fix: more todo comments

* fix: otel instrumentor

* fix: rm unused attrs

* chore: remove comment

---------

Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>
2026-03-09 16:42:23 -04:00
Julius Park 6c1bd9a074 Fix broken docs links (#3217)
* initial commit

* idk why those files got updated

* the gh action for checking links isn't viable :/

* try again

* fix child-spawning

* fix more

* fix some more

* fix lint

* remove gh workflow for now
2026-03-09 16:25:31 -04:00
Gabe Ruttner ab84a0b9b0 refactor: simplify abort signal handling in durable listeners (#3215)
* refactor: simplify abort signal handling in durable listeners

* rm: hatchet cloud

* fix: cancellation reason

* fix: tests
2026-03-09 09:35:12 -07:00
Gabe Ruttner 468de0c584 feat: handle server initiated eviction on workers (#3200)
* refactor: ts in sync with python

* fix: waits respect abort signal

* chore: gen, lint

* fix: update key extraction logic in durable workflow example

* fix: flake

* fix: racey test

* chore: gen

* feat: send eviction to engine

* feat: handle eviction on workers

* fix: tests and invocation scoped eviction

* fix: cleanup scope

* refactor: cleanup how we do action keys in ts

* chore: gen, lint

* fix: test

* chore: lint

* fix: mock

* fix: cleanup

* chore: lint

* chore: copilot feedback

* fix: reconnect durable listener

* chore: feedback

* fix: bump worker status on wait and reconnect

* refactor: rename eviction notice

* fix: return error on query failure

* chore: lint

* chore: gen

* refactor: remove early completion

* Revert "refactor: remove early completion"

This reverts commit f756ce37b0.

* chore: remove unused param
2026-03-09 05:16:46 -07:00
Gabe Ruttner f667b7a630 chore: lint 2026-03-08 17:58:53 -04:00
Gabe Ruttner 7e01f7ba94 Merge branch 'main' into feat-durable-execution 2026-03-08 17:50:42 -04:00
Gabe Ruttner 2a98243cd4 chore: update eslint and other packages (#3202)
* fix: eslint for local dev

* chore: eslint v8 -> 10

* fix: flyby bug

* chore: remove many anys

* chore: update dependencies

* chore: lint
2026-03-08 13:11:51 -07:00
Gabe Ruttner fca52697a8 refactor: ts in sync with python (#3199)
* refactor: ts in sync with python

* fix: waits respect abort signal

* chore: gen, lint

* fix: update key extraction logic in durable workflow example

* fix: flake

* fix: racey test

* chore: gen
2026-03-07 05:42:42 -08:00
Gabe Ruttner 1cd9660835 rip: remove unneeded durable event log update (#3186)
* rip: update

* refactor: start cleaning up proto defs

* refactor: finish cleaning up proto definitions

* fix: rm the kind

* refactor: rewire the server to have different methods for the different paths

* refactor: more intermediate work

* fix: variables

* fix: wire the kind through

* refactor: get it to compile

* chore: start fixing python

* fix: first pass at fixing python

* chore: rm unused sql

* fix: rm invocation count, rework some logic

* fix: alias (why does sqlc not catch this)

* fix: panics

* fix: add faster timeout to durable spawn test

* fix: task id bug

* refactor: more cleanup of types

* refactor: rm stale entries logic

* fix: rework getOrCreate logic

* fix: clean up a bunch more unneeded stuff

* fix: bug

* fix: python code

* fix: dag matches bug

* fix: add parent to dag to make it more broken, add timeout

* fix: more involved tests

* fix: dag waits

* fix: tests

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>
2026-03-07 04:02:48 -08:00
Julius Park 2386b442eb Update change logs for cron change again (#3192) 2026-03-06 15:30:22 -05:00
Julius Park bb14620fce Update change logs for cron change (#3155)
* initial commit

* update dates
2026-03-06 15:06:06 -05:00
Gabe Ruttner f177f11290 Feat ts durable execution (#3134)
* chore: generate

* feat: durable execution in typescript

* fix: cancellation logs

* chore: lint

* chore: generate

* chore: address copilot review

* chore: update docstring

* chore: lint

* refactor: how memos are constructed

* chore: generate ts

* feat: bulk spawn

* feat: compat

* chore: gen

* chore: cleanup and lint

* fix: durable spawn types

* fix: build

* fix: graceful termination

* flyby: namespaces

* fix: graceful term

* chore: lint

* fix: compat

* chore: lint

* fix: legacy race

* fix: last tests

* chore: lint

* fix: bulk replay e2e test

* fix: relax assertion

* fix: bulk
2026-03-06 10:00:31 -08:00
matt 609ec0a10e [Python] Fix: Lifespan not shutting down properly on worker drain (#3175)
* fix: wait for in-flight tasks to complete before running lifespan shutdown

* feat: add test

* chore: version

* fix: add wait for tasks in deprecated / legacy mode too

* fix: wrong spot
2026-03-06 12:50:03 -05:00
Gabe Ruttner 0a55f73979 fix: ts bulk and event e2e (#3187)
* fix: bulk replay e2e test

* fix: relax assertion

* fix: bulk

* fix: just apply the namespace

* chore: lint
2026-03-06 08:49:08 -08:00
Gabe Ruttner 6b06669a13 fix: ts-sdk namespace on wait for conditions (#3182)
* chore(ts-sdk): pass namespace into condition transformers

- Add optional namespace param to conditionsToPb and taskConditionsToPb
- Apply namespace to userEventKey in UserEventCondition via applyNamespace
- Pass config.namespace from context, worker-internal, and legacy step
  when building conditions so wait/event keys are namespaced correctly

Made-with: Cursor

* release: 1.15.2

* fix: imports
2026-03-06 06:20:32 -08:00
Mohammed Nafees 00b53529bd Add missing Go SDK examples for worker affinity and manual slot release (#3179)
* add missing Go SDK examples for affinity and manual slot release

* fix gen
2026-03-06 01:51:27 +01:00
Gabe Ruttner 12bf7514c5 feat: durable-bulk-spawn (#3173)
* feat: durable bulk spawn

* feat: bulk db operations

* chore: lint

* chore: generate

* fix: sqlc hack

* cleanup

* chore: lint

* refactor: remove duplicated code

* refactor: simplify input

* ops: rules

* fix: add missing workflows to worker

* chore: add todo

* refactor: single execution path

* comment

* refactor: remove extra class

* chore: lint

* chore: lint

* refactor: batch ack

* fix: handle empty refs

* chore: feedback

* fix: durable output

* chore: lint

* chore: lint

* chore: test fix and gen

* revert: actually use bulk

* refactor: simplify path

* chore: empty commit

Made-with: Cursor
2026-03-05 16:08:13 -08:00
Jishnu 09b6001fc8 docs: python webhooks docs, update current docs (#3159) 2026-03-05 18:19:12 -05:00
matt 1bffb66bb3 Feat: Branching off branches (#3150)
* feat: add branch point table

* chore: gen

* feat: id for ordering

* feat: check for `isBranchPoint` and handle branching

* feat: wire up branching over the api

* fix: api, gen

* fix: gen

* fix: branch

* feat: remove duped parent node and branch ids

* fix: rm branch count, dupe of latest id

* refactor: resolve naming

* fix: base case

* fix: test, + it's literally always a caching issue omg

* fix: docs

* chore: lint

* refactor: make branch resolution more efficient

* feat: stable sort, add a bunch of tests

* fix: confusing naming

* fix: naming

* chore: gen

* fix: update

* fix: failing test

---------

Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>
2026-03-05 16:31:01 -05:00
Gabe Ruttner a6f5fb6871 chore: rename durable fork to branch (#3174) 2026-03-05 09:56:46 -08:00
Gabe Ruttner 78c547b6dd chore: feedback 2026-03-05 09:56:24 -05:00
Gabe Ruttner 18b32cab91 fix: assert eviction for each 2026-03-05 09:51:09 -05:00
Gabe Ruttner b07e31db10 fix: skip eviction on old engine 2026-03-05 08:52:40 -05:00
Gabe Ruttner 217aa2bec2 fix: register the examples on the worker 2026-03-05 08:04:42 -05:00
Gabe Ruttner d4df0d91b7 feat: add durable complex test cases
Add comprehensive test suite for durable complex workflows including:
- DAG workflows with mixed ephemeral/durable tasks
- Concurrency control and slot retention
- Execution timeout handling
- On-failure callback tests
- Rate limiting
- Retry behavior

Made-with: Cursor
2026-03-05 07:45:09 -05:00
Gabe Ruttner 5b58eed1ce tests: compat testing (#3144)
* tests: compat testing

* fix: new engine old sdk

* feat: ruby ts

* fix: namespaced tests, conditions

* chore: lint ts

* fix: python compat

* chore: lint

* fix: warn on version mismatch

* chore: lint

* fix: child spawn

* refactor: address review

* chore: lint

* chore: lint

* chore: lint

* refactor: address reviews

* chore: lint

* fix: typeguard

* tests: docker-compose

* fix: docker compose

* fix: labels?

* chore: skip new tests

* fix: backwards compat

* fix: grpc proxy
2026-03-04 18:12:45 -08:00
matt 46b3041eb1 Fix: Go unexported type (#3160)
* fix: internal types

* fix: rm unused helper
2026-03-04 17:40:13 -05:00
Gabe Ruttner 163918ae98 fix: package ts (#3161)
* fix: package

* fix base url...
2026-03-04 14:36:30 -08:00
Gabe Ruttner 5babd09679 docs: chunky overhaul (#3129)
* improve overall docs structure and things
2026-03-04 14:33:15 -08:00
Gabe Ruttner 65e44d6f63 feat: OLAP status priority functions and query updates (#3156)
* feat: OLAP status priority functions and query updates

- Add v1_status_priority / v1_status_from_priority for v1_readable_status_olap
- Use priority-based aggregation in OLAP task status update queries (EVICTED
  below terminal statuses)
- Migration v1_0_84 and schema v1-olap.sql

Made-with: Cursor

* test: durable eviction tests — replay, cancel after eviction, restore idempotency

- test_eviction_plus_replay, test_evictable_cancel_after_eviction, test_restore_idempotency
- Tighter poll interval for faster test runs

Made-with: Cursor

* fix: flakes

* feedback
2026-03-04 11:11:09 -08:00
mrkaye97 51b3af0601 fix: lint 2026-03-04 14:03:03 -05:00
mrkaye97 5d735b4c41 fix: linting issues 2026-03-04 13:59:05 -05:00
mrkaye97 e29459f58a chore: merge main 2026-03-04 13:38:37 -05:00
matt 6c29e48204 Feat: Dynamic worker label assign (#3137)
* feat: initial wiring work on desired labels

* feat: initial wiring

* chore: gen python

* fix: use the whole desired label thing instead

* fix: more wiring, improve types

* fix: sql type

* fix: len check

* chore: gen python

* fix: initial plural label work

* fix: store the labels properly on the task

* fix: skip cache on override

* fix: bug

* fix: scoping bug whoops

* chore: lint

* fix: send labels back over the api correctly

* feat: python test

* fix: lint

* fix: comment

* fix: override

* fix: namespaces, ugh

* fix: no need for error here

* chore: version

* feat: ruby, go, ts

* feat: versions

* fix: appease the rubocop

* chore: lint

* chore: bundle install

* fix: tests

* chore: lint

* chore: lint more

* fix: ts test

* fix: rb

* chore: gen

* chore: reset gemfile

* chore: reset changelog

* fix: pgroup

* fix: tests, part i

* Revert "chore: reset changelog"

This reverts commit b63bf7d3e5.

* Revert "chore: reset gemfile"

This reverts commit bb848bb6f0.

* fix: go -> golang mapping hack

* fix: go enums

* fix: appease the cop

* fix: namespace

* chore: gen
2026-03-04 11:03:58 -05:00
mrkaye97 a4f65454f3 Merge branch 'main' into feat-durable-execution 2026-03-03 17:24:23 -05:00
Jishnu 508edde607 feat: automatic doc generator for TS SDK (#3047)
* add: typedoc config

* refactor: typeconfig, name feature client modules

* add: custom hatchet md plugin, feature client docs

* docs: filters client

* docs: add worker, workflows defintions

* fix: client and context docs, update typedoc

* fix: lint, add runnables doc

* add: additional md formatter methods, module docs

* add: geneated mdx, core generate and format logic

* refactor: new function changes

* fix: lint, lockfile

* fix: lint, strip generics in classes for docs

* refactor: use single line comments for deprecated

* fix: markdown escaping by typedoc

* docs: regenerate docs with new changes

* docs: refactor methods on alphabetical order

* refactor: non method headers to normal text

* docs: regenerate docs with 3 type depth
2026-03-03 12:07:41 -05:00
Julius Park 5e4bb9d151 Add seconds granularity to cron jobs (#3136) 2026-03-02 18:07:15 -05:00
matt 77f769bd66 Feat: Durable Memoization (#3112)
* feat: initial engine / db work

* chore: gen python

* feat: initial python work

* feat: wiring

* feat: initial test

* fix: scope memo key to the run id (we might not even need this)

* chore: gen, docstring

* fix: mandatory result type

* fix: docs

* fix: log a warning and don't cache

* fix: add test for non-unique keys, fix some bugs

* chore: lint

* chore: naming

* fix: more naming

* fix: comment

* chore: gen

* fix: docs

* fix: naming, ugh

* chore: gen

* chore: gen

* fix: always send event log entry and get ack

* fix: initial rework of memo to only use stream

* fix: start reworking python side

* fix: finish wiring up memo put

* chore: fmt

* chore: comment for monday

* fix: start reworking signature

* fix: docstring

* fix: union type for send event to improve typing a bunch

* fix: memo key

* chore: gen

* fix: rm unused query

* fix: comments
2026-03-01 16:27:56 -08:00
Gabe Ruttner e2eb3f05e9 Merge branch 'main' into feat-durable-execution 2026-03-01 19:11:13 -05:00
Gabe Ruttner 5a79618824 chore: deprecate ts v0 (#2960)
* rip ts v0: keep only TypeScript SDK changes

Cherry-pick TypeScript changes from rip--ts-v0 onto main,
discarding all non-TS changes (Go, Python, proto, SQL, etc.)

Made-with: Cursor

* merge: main into branch

* chore: rm redundant examples

* fix: tests

* fix: tests

* tests: speed em up

* tests: unskip

* tests: add missing p1s

* version: bump to 1.14.0

* chore: generate

* fix: release version

* chore: generate

* chore: rm legacy examples

* tests: timeouts...

* chore: remove unused config

* refactor: minimize dependency on legacy client

* refactor: rename V1Worker to InternalWorker and update references

* chore: lint

* tests: one big worker

* chore: lint

* chore: generate and test fixture tweaks

* tests: debug logs

* tests: stable order

* fix: abortable sleep

* tests: log artifacts on CI failure

* tmp: tmate

* chore: lint

* test: remove tmate

* chore: generate

* tmate

* tmate alwasy

* fix: log level

* fix: casing

* chore: remove tmate

* chore: consistent naming

* fix: test

* test: skip flaky test

* fix: casing

* chore: gen

* chore: lint

* revert: dependency version changes
2026-03-01 16:06:56 -08:00