Commit Graph

6 Commits

Author SHA1 Message Date
matt
05399ebf39 Feat: Durable event log wiring (#2956)
* feat: initial protos

* chore: lint

* fix: work on improving naming

* chore: rename session id to invocation count

* feat: scaffold implementation of durabletask rpc

* fix: one more session rename

* feat: initial work on the server scaffolding

* chore: gen protos for python

* feat: initial durable task client

* feat: initial durable context work for python

* fix: pass client through to runner

* fix: clean up type checking errors

* fix: cruft

* feat: initial work wiring up durable events

* fix: get -> getorcreate

* feat: query + wiring for updating latest node id

* fix: simplify, bump latest node ids in the same query

* chore: note

* feat: wire up sleeps with internal signal matches

* chore: gen

* fix: callback data writes

* feat: cache previous events

* fix: wire up external id writes

* feat: got sleeps sorta working!

* fix: tenant and external id wiring

* chore: comments

* fix: clean up some types a bit

* feat: add run triggering params to proto to allow for spawning children

* feat: first pass at child spawning

* feat: start wiring up child spawning

* fix: use `triggerWriter` for spawn

* feat: update trigger proto def

* chore: regen python

* feat: start wiring up spawning correctly with all opts

* refactor: share trigger code

* chore: remove log lines, lint

* fix: add triggered run external id

* feat: start wiring up child key storage better

* chore: gen again

* fix: gen, colname

* fix: trigger opts panicking

* hack: get things working for now

* feat: shared rpc message

* chore: fix imports

* feat: add tenant id to tables

* fix: improve ingest logic

* refactor: shared trigger opt type

* fix: send tenant id through everywhere

* chore: fix log file insert on conflict

* fix: repo

* fix: generate external id upstream

* feat: add columns to the match

* feat: first pass at durable waits on the controllers instead of the dispatcher

* fix: types

* feat: wire up callbacks

* fix: invoc counts

* fix: typing, lint

* driveby: more constants for message ids

* refactor: struct for callback keys everywhere

* fix: bugs, passing tests

* fix: return errnorows

* fix: schema

* fix: remove current callback flow

* feat: new message types

* fix: remove key from callback model

* fix: rm unused queries

* refactor: start reworking flow

* fix: start working on feedback

* fix: query

* fix: wire up external ids

* revert: drive by

* refactor: rm extra interface

* chore: move listener, lint

* refactor: remove old listener, rename

* refactor: consolidate migrations

* fix: immediately send already-satisfied callbacks

* fix: union

* chore: rm unused queries

* fix: check if entry already exists before re-spawning / signaling

* fix: node id incrementation

* fix: rm json dump

* fix: don't pass node id

* fix: store latest invocation, update query

* fix: upsert logic

* Revert "fix: upsert logic"

This reverts commit cf7c609c1d.

* fix: change logic slightly

* fix: split up get and create queries

* fix: err

* fix: pass node ids around properly

* fix: invocation handling

* fix: callback bug

* fix: naming

* fix: rm cruft method, dynamic kind

* fix: wire up memo payload and kind stuff

* fix: propagate trigger opts

* fix: child spawn signaling + olap wiring

* fix: extract output method

* feat: improve test coverage a bit

* fix: child spawning

* feat: another test

* fix: query fixes, overwrite

* fix: match bug

* fix: proto indexes, regen

* fix: eviction comment

* fix: warning for non-async durable tasks

* fix: rm contracts import

* fix: basic locking, rm sync durable tasks

* fix: invocation counts, etc.

* chore: add fixme

* fix: rm unused invocation count param from callback response

* fix: rm dispatcher id from the callback

* fix: di test

* Revert "fix: rm dispatcher id from the callback"

This reverts commit 26e6c82797.

* fix: migration

* fix: use optimistictx

* fix: lift grpc codes out of trigger repo

* fix: span names

* fix: rm comment

* fix: consolidate kind types, batching, not-null kinds

* fix: null bug

* fix: satisfied claim bug, simplify queries

* fix: add back payload storage

* fix: match bug, simplification

* fix: factor out trigger opts to the dispatcher level

* fix: factor out conditions

* fix: rm unused structs

* fix: rm dupes

* fix: migration

* refactor: switch case helpers

* fix: panic

* fix: couple warnings

* fix: lint

* fix: generate external ids properly

* refactor: return trigger task data from helper

* fix: handle matches correctly for dag spawns

* fix: add validators, one more uuid type

* chore: gen

* chore: bump pytest-asyncio to latest

* fix: store the worker instead of the dispatcher, then look up the dispatcher

* fix: store dispatcher id on the worker

* chore: lint
2026-02-16 12:23:58 -05:00
matt
c8d5144ed4 [Python] Feat: Dependency Injection, Improved error handling (#2067)
* feat: first pass at dependency injection

* feat: add DI example + tests

* feat: finish up tests

* feat: docs

* chore: gen

* chore: lint

* chore: changelog + ver

* fix: split up paragraphs

* refactor: improve impl

* chore: gen

* feat: inject input + ctx into deps

* chore: gen

* [Python] Feat: More use of `logger.exception` (#2069)

* feat: add more instances of `logger.exception`

* chore: ver

* chore: changelog

* fix: one more error log

* chore: gen

* chore: gen

* chore: lint

* fix: improve shutdown

* chore: changelog

* unwind: exit handler

* feat: task run error

* feat: improve error serde with more context

* chore: changelog

* fix: changelog

* chore: gen

* fix: rm celpy + lark dep, casing issues

* chore: changelog

* fix: respect log levels over the API

* fix: changelog

* refactor: rename log forwarder

* fix: circular import
2025-08-11 23:10:44 -04:00
matt
cca0999eea [Python]: Fixing logging bugs, fixing duped sleep key bug (#2040)
* feat: add flag to disable log capture

* fix: sleep bug with duped key

* fix: allow formatters to be passed through

* feat: support filters too

* fix: cruft

* chore: gen

* feat: changelog

* fix: lint

* [Python] Fix: Don't retry gRPC requests on 4xx (#2024)

* fix: dont retry on 4xx

* chore: ver

* fix: sleep conditions with index

* fix: bug in sleep conditions

* chore: gen
2025-07-23 15:15:31 -04:00
Matt Kaye
7679732b15 Fix: Skipping conditions with multiple parents (#1948)
* fix: skipping bug

* fix: move `waits` -> `conditions`

* fix: refs

* chore: ver

* feat: add skipped task to test

* feat: start implementing or groups in wait for

* feat: test of or groups on durable context

* fix: lint

* chore: gen

* fix: lint

* fix: branching hell
2025-07-03 16:50:57 -04:00
Matt Kaye
2f33dd4dbd Feat: Misc. Python improvements + Streaming Improvements (#1846)
* fix: contextvars explicit copy

* feat: fix a ton of ruff errors

* fix: couple more ruff rules

* fix: ignore unhelpful rule

* fix: exception group in newer Python versions for improved handling

* fix: workflow docs

* feat: context docs

* feat: simple task counter

* feat: config for setting max tasks

* feat: graceful exit once worker exceeds max tasks

* fix: optional

* fix: docs

* fix: events docs + gen

* chore: gen

* fix: one more dangling task

* feat: add xdist in ci

* fix: CI

* fix: xdist fails me once again

* fix: fix + extend some tests

* fix: test cleanup

* fix: exception group

* fix: ugh

* feat: changelog

* Add Ruff linter callout to post

* refactor: clean up runner error handling

* feat: improved errors

* fix: lint

* feat: hacky serde impl

* fix: improve serde + formatting

* fix: logging

* fix: lint

* fix: unexpected errors

* fix: naming, ruff

* fix: rm cruft

* Fix: Attempt to fix namespacing issue in event waits (#1885)

* feat: add xdist in ci

* fix: attempt to fix namespacing issue in event waits

* fix: namespaced worker names

* fix: applied namespace to the wrong thing

* fix: rm hack

* drive by: namespacing improvement

* fix: delay

* fix: changelog

* fix: initial log work

* fix: more logging work

* fix: rm print cruft

* feat: use a queue to send logs

* fix: sentinel value to stop the loop

* fix: use the log sender everywhere

* fix: make streaming blocking, remove more thread pools

* feat: changelog

* fix: linting issues

* fix: broken test

* chore: bunch more generated stuff

* fix: changelog

* fix: one more

* fix: mypy

* chore: gen

* Feat: Streaming Improvements (#1886)

* Fix: Filter list improvements (#1899)

* fix: uuid validation

* fix: improve filter filtering

* fix: inner join

* fix: bug in workflow cached prop

* chore: bump

* fix: lint

* chore: changelog

* fix: separate filter queries

* feat: improve filter filtering

* fix: queries and the like

* feat: add xdist in ci

* feat: streaming test + gen

* feat: add index to stream event

* fix: rm langfuse dep

* fix: lf

* chore: gen

* feat: impl index for stream on context

* feat: tweak protos

* feat: extend test

* feat: send event index through queue

* feat: first pass + debug logging

* debug: fixes

* debug: more possible issues

* feat: generate new stream event protos

* feat: first pass at using an alternate exchange for replaying incoming stream events

* fix: exchange create timing

* fix: rm unused protos

* chore: gen

* feat: python cleanup

* fix: revert rabbit changes

* fix: unwind a bunch of cruft

* fix: optional index

* chore: gen python

* fix: event index nil handling

* feat: improve test

* fix: stream impl in sdk

* fix: make test faster

* chore: gen a ton more stuff

* fix: test

* fix: sorting helper

* fix: bug

* fix: one more ordering bug

* feat: add some tests for buffering logic

* feat: hangup test

* feat: test no buffering if no index sent

* fix: regular mutex

* fix: pr feedback

* fix: conflicts
2025-06-25 10:11:01 -04:00
Gabe Ruttner
8e80faf2d6 Fe overhaul docs (#1640)
* api changes

* doc changes

* move docs

* generated

* generate

* pkg

* backmerge main

* revert to main

* revert main

* race?

* remove go tests
2025-04-30 14:10:09 -07:00