mrkaye97
f8e787cd89
Merge branch 'main' into feat-durable-execution
2026-02-19 19:35:40 -05:00
Gabe Ruttner
8c9fa7fd82
feat: add migration for worker slot config index ( #3062 )
2026-02-19 12:25:36 -08:00
matt
7e3e3b8fc0
Feat: Non-determinism errors ( #3041 )
...
* fix: retrieve payloads in bulk
* fix: hash -> idempotency key
* feat: initial hashing work
* feat: check idempotency key if entry exists
* fix: panic
* feat: initial work on custom error for non-determinism
* fix: handle nondeterminism error properly
* feat: add error response, pub message to task controller
* chore: lint
* feat: add node id field to error proto
* chore: rm a bunch of unhelpful cancellation logs
* fix: conflict issues
* fix: rm another log
* fix: send node id properly
* fix: improve what we hash
* fix: improve error handling
* fix: python issues
* fix: don't hash or group id
* fix: rm print
* feat: add python test
* fix: add timeout
* fix: improve handling of non determinism error
* fix: propagate node id through
* fix: types, test
* fix: make serializable
* fix: no need to cancel internally anymore
* fix: hide another internal log
* fix: add link to docs
* fix: copilot
* fix: use sha256
* fix: test cleanup
* fix: add error type enum
* fix: handle exceptions on the worker
* fix: clean up a bunch of cursor imports
* fix: cursor docstring formatting
* fix: simplify idempotency key func
* fix: add back cancellation logs
* feat: tests for idempotency keys
* fix: add a couple more for priority and metadata
* chore: gen
* fix: python reconnect
* fix: noisy error
* fix: improve log
* fix: don't run durable listener if no durable tasks are registered
* fix: non-null idempotency keys
2026-02-18 11:27:02 -05:00
matt
eaf6bba824
Refactor: Remove separate callback table ( #3045 )
...
* fix: remove callback table
* fix: type
* fix: type
* fix: wiring everything up
* fix: result payload for replays
* chore: lint
* feat: set fillfactor
* chore: gen
* fix: simplify v1 match changes
* fix: simplify v1 match wiring
* fix: rm print line
* fix: some more wiring
* fix: wiring
* chore: comments
* chore: gen, proto naming
* chore: comments
* fix: rm comment
* fix: broken listener
2026-02-17 13:25:08 -05:00
mrkaye97
7ad5bfdf89
Merge branch 'main' into feat-durable-execution
2026-02-17 08:45:41 -05:00
Gabe Ruttner
2fdc47a6af
feat: multiple slot types ( #2927 )
...
* feat: adds support for multiple slot types, primarily motivated by durable slots
---------
Co-authored-by: mrkaye97 <mrkaye97@gmail.com >
2026-02-17 05:43:47 -08:00
matt
05399ebf39
Feat: Durable event log wiring ( #2956 )
...
* feat: initial protos
* chore: lint
* fix: work on improving naming
* chore: rename session id to invocation count
* feat: scaffold implementation of durabletask rpc
* fix: one more session rename
* feat: initial work on the server scaffolding
* chore: gen protos for python
* feat: initial durable task client
* feat: initial durable context work for python
* fix: pass client through to runner
* fix: clean up type checking errors
* fix: cruft
* feat: initial work wiring up durable events
* fix: get -> getorcreate
* feat: query + wiring for updating latest node id
* fix: simplify, bump latest node ids in the same query
* chore: note
* feat: wire up sleeps with internal signal matches
* chore: gen
* fix: callback data writes
* feat: cache previous events
* fix: wire up external id writes
* feat: got sleeps sorta working!
* fix: tenant and external id wiring
* chore: comments
* fix: clean up some types a bit
* feat: add run triggering params to proto to allow for spawning children
* feat: first pass at child spawning
* feat: start wiring up child spawning
* fix: use `triggerWriter` for spawn
* feat: update trigger proto def
* chore: regen python
* feat: start wiring up spawning correctly with all opts
* refactor: share trigger code
* chore: remove log lines, lint
* fix: add triggered run external id
* feat: start wiring up child key storage better
* chore: gen again
* fix: gen, colname
* fix: trigger opts panicking
* hack: get things working for now
* feat: shared rpc message
* chore: fix imports
* feat: add tenant id to tables
* fix: improve ingest logic
* refactor: shared trigger opt type
* fix: send tenant id through everywhere
* chore: fix log file insert on conflict
* fix: repo
* fix: generate external id upstream
* feat: add columns to the match
* feat: first pass at durable waits on the controllers instead of the dispatcher
* fix: types
* feat: wire up callbacks
* fix: invoc counts
* fix: typing, lint
* driveby: more constants for message ids
* refactor: struct for callback keys everywhere
* fix: bugs, passing tests
* fix: return errnorows
* fix: schema
* fix: remove current callback flow
* feat: new message types
* fix: remove key from callback model
* fix: rm unused queries
* refactor: start reworking flow
* fix: start working on feedback
* fix: query
* fix: wire up external ids
* revert: drive by
* refactor: rm extra interface
* chore: move listener, lint
* refactor: remove old listener, rename
* refactor: consolidate migrations
* fix: immediately send already-satisfied callbacks
* fix: union
* chore: rm unused queries
* fix: check if entry already exists before re-spawning / signaling
* fix: node id incrementation
* fix: rm json dump
* fix: don't pass node id
* fix: store latest invocation, update query
* fix: upsert logic
* Revert "fix: upsert logic"
This reverts commit cf7c609c1d26e6c82797
2026-02-16 12:23:58 -05:00
mrkaye97
eaac2b09fb
Merge branch 'main' into feat-durable-execution
2026-02-16 07:59:50 -05:00
Gabe Ruttner
7875d78057
Feat: Official Ruby SDK ( #3004 )
...
* feat: initial ruby sdk
* fix: run listener
* fix: scope
* feat: rest feature clients
* fix: bugs
* fix: concurrent register
* fix: tests and ergonomics
* docs: all of them
* chore: lint
* feat: add RBS
* feat: add GitHub Actions workflow for Ruby SDK with linting, testing, and publishing steps
* chore: lint
* refactor: simplify load path setup for Hatchet REST client and remove symlink creation
* fix: cert path
* fix: test
* fix: blocking
* fix: ensure Hatchet client is only initialized once across examples
* fix: tests
* remove: unused example
* fix: bubble up errors
* test: skip flaky for now
* remove: lifespans
* fix: durable context bugs
* fix: bulk replay
* fix: tests
* cleanup: generate tooling
* fix: integration test
* chore: lint
* release: 0.1.0
* chore: remove python comments
* refactor: remove OpenTelemetry configuration and related unused options
* fix: default no healthcheck
* chore: lockfile
* feat: register as ruby
* chore: lint
* chore: update py/ts apis to include ruby
* chore: docs pass
* chore: lint
* chore: generate
* chore: cleanup
* chore: generate examples
* tests: add e2e tests
* tests: cache examples dependencies
* fix: namespace
* fix: namespace
* fix: namespaces
* chore:lint
* fix: improve cancellation workflow polling logic and add error handling
* revert: py/ts versions
2026-02-15 14:32:15 -08:00
mrkaye97
740af439ce
Merge branch 'main' into feat-durable-execution
2026-02-12 09:27:40 -05:00
Mohammed Nafees
4fd7b94751
Add support for Svix webhooks ( #2996 )
...
* support Svix webhooks
* add migration
* use http status codes
* comment fix
* custom svix verification logic
* copilot comments
* copilot comments
2026-02-11 16:41:36 +01:00
mrkaye97
3faa5adfc7
fix: id and inserted at on fields that need payloads
2026-02-06 16:38:58 -05:00
matt
7d66b3806b
Feat: Durable event log models ( #2940 )
...
* feat: add new models
* feat: migration
* chore: generate
* fix: linter
* fix: couple types
* Feat: Initial work on CRUD operations for durable events (#2943 )
* feat: initial query work
* feat: first pass at durable events repo + queries
* feat: add new payload type for durable event data
* chore: gen
* fix: payload key
* fix: lint
2026-02-05 13:58:19 -05:00
Jishnu
ed43cae0a2
feat: Extend webhook support for scope_expression and payload ( #2874 )
...
* add: scope_expression and payload columns for v1_webhook
* refactor: insert or update sql cmds for v1_webhook
* feat: update api clients, openapi schema for new webhook body
* refactor: receiver and transformer for v1 webhook
* add: python sdk changes
* feat: ts sdk changes
* feat: add FE for webhook new params
* fix: scope expression empty payload
* add: support for scope and payload for go client
* fix: lint
* fix: error message UI on webhook
* fix: lint
* fix: migraiton conflict, build failure
* fix: error handling
* update docs, add tests
* fix: lint, test file name
2026-02-04 12:44:52 -05:00
abelanger5
d56dee4266
feat: durable user event log ( #2861 )
...
* placeholder
* feat: db tables for user events (#2862 )
* feat: db tables for user events
* move event payloads to payloads table, fix env var loading
* fix: address pr review comments
* missed save
* feat: optimistic scheduling (#2867 )
* feat: db tables for user events
* move event payloads to payloads table, fix env var loading
* refactor: small changes to prepare optimistic txs
* feat: optimistic scheduling
* address pr review comments
* rm comments
* fix: rampup test race condition
* fix: goleak
* feat: grpc-side triggers
* fix: config and sem logic
* fix: respect optimistic scheduling env var
* add optimistic to testing matrix, remove pg-only mode
* fix cleanup of pubbuffers
* merge migrations
* last testing fixes
2026-02-02 18:04:02 -05:00
matt
a3fe89ef03
Feat: Workflow input JSON schema in trigger preview ( #2851 )
...
* feat: add input json schema to workflow version
* feat: add json schema to putworkflow proto
* feat: wire up writes of the json schema
* chore: gen python
* feat: send json schema from the python code
* feat: wiring
* feat: pass json schema into code editor
* feat: pass prop
* fix: clean up validation stuff
* feat: allow zod `input` as optional ts field
* fix: try except logic hack
* fix: rename input -> inputValidator for consistency
* chore: gen api
* fix: improve hack slightly
* chore: changelogs, versions
* feat: zod example
* chore: rework api a bit
* fix: tsc, allow schema to update
* fix: improve loading state
* fix: api cleanup, sqlc cleanup
* fix: initial mount
* chore: lint
* chore: lint
* chore: tsc
* fix: lint
* fix: unwind unneeded change
* [Python] Feat: Default additional metadata (#2876 )
* Add doc about autoscaling workers (#2864 )
* add doc for autoscaling workers
* oldest also in running stats
* chore(deps): bump google.golang.org/api from 0.262.0 to 0.263.0 (#2869 )
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client ) from 0.262.0 to 0.263.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases )
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md )
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.262.0...v0.263.0 )
---
updated-dependencies:
- dependency-name: google.golang.org/api
dependency-version: 0.263.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump github.com/getsentry/sentry-go from 0.41.0 to 0.42.0 (#2870 )
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go ) from 0.41.0 to 0.42.0.
- [Release notes](https://github.com/getsentry/sentry-go/releases )
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md )
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.41.0...v0.42.0 )
---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
dependency-version: 0.42.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump hatchet-sdk in /examples/python/quickstart (#2871 )
Bumps hatchet-sdk from 1.22.10 to 1.22.11.
---
updated-dependencies:
- dependency-name: hatchet-sdk
dependency-version: 1.22.11
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: default additional meta
* feat: wiring
* chore: changelog, version
* fix: copy
* feat: add default meta to stubs
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: migration ver
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-29 11:38:25 -05:00
Gabe Ruttner
b7ec0bc270
fix: big int alignment for cleanup function ( #2877 )
...
* fix function
* Update cmd/hatchet-migrate/migrate/migrations/20260128120000_v1_0_71.sql
Co-authored-by: matt <mrkaye97@gmail.com >
* leeeeent
---------
Co-authored-by: matt <mrkaye97@gmail.com >
2026-01-29 05:19:39 -08:00
Mohammed Nafees
6eba6fa91f
Billing changes ( #2643 )
...
* make changes for billing
* progress around redesign
* meter callback
* modify limits
* upcoming subscription
* fix lint
* fix payment methods
* fix build
* PR comments
* address PR comments
* update cloud contracts
* fix migration name
* fix json serialization error
* loader and fixed for managed compute
* PR comments
* upgrade Go version
* fix migration name
* fix CI
* fix lint CI
* golangci-lint fix
* dedicated subscription
2026-01-19 12:15:11 +01:00
abelanger5
6920ec1b61
revert: UI version removal ( #2756 )
...
* revert: UI version removal
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2026-01-06 14:16:42 -05:00
abelanger5
dd9c36c315
refactor: remove v0 paths from codebase ( #2728 )
...
* refactor: remove v0 paths from codebase
* remove uiVersion references
2025-12-30 09:57:00 -05:00
matt
735742c466
Revert "Revert "chore: run list query optimizations ( #2670 )" ( #2708 )" ( #2720 )
...
This reverts commit 2f301e55cf
2025-12-26 10:11:02 -07:00
Mohammed Nafees
58758d35b2
Publish COULD_NOT_SEND_TO_WORKER OLAP event due to worker backlog ( #2710 )
...
* could not send to worker OLAP event
* fix lint and PR comments
* submodule GHA
* remove submodule
* no gitsubmodule
* fix migration
* revert sdk workflows
* revert sdk workflows
* revert sdk workflows
2025-12-26 09:35:15 -07:00
matt
2f301e55cf
Revert "chore: run list query optimizations ( #2670 )" ( #2708 )
...
This reverts commit 87b57febe8
2025-12-23 17:10:47 -05:00
matt
b65c6de53f
Feat: Hatchet Metrics Monitoring, I ( #2699 )
...
* Revert "Revert "Feat: Hatchet Metrics Monitoring, I (#2480 )" (#2698 )"
This reverts commit b87150767ahello@mnafees.me >
2025-12-23 20:14:14 +01:00
Gabe Ruttner
87b57febe8
chore: run list query optimizations ( #2670 )
...
* add missing tenant index
* fix span name
* parallelize
* instrument tenant id attribute
* feedback
* cleanup migrations
* rename migration
* correct version
* cleanup
2025-12-23 08:59:13 -08:00
matt
bc7f341d13
Fix: Dynamically-sized chunks on payload read ( #2700 )
...
* feat: function for pulling chunk size in bytes
* chore: schema
* feat: queries
* chore: rework function
* feat: impl
* fix: ptr
* fix: ptr
* feat: olap side
* fix: handle nulls
2025-12-22 18:28:04 -05:00
matt
b87150767a
Revert "Feat: Hatchet Metrics Monitoring, I ( #2480 )" ( #2698 )
...
This reverts commit fdc075ec6f
2025-12-22 16:26:14 -05:00
matt
fdc075ec6f
Feat: Hatchet Metrics Monitoring, I ( #2480 )
...
* feat: queries + task methods for oldest running task and oldest task
* feat: worker slot and sdk metrics
* feat: wal metrics
* repository stub
* feat: add meter provider thingy
* pg queries
* fix: add task
* feat: repo methods for worker metrics
* feat: active workers query, fix where clauses
* fix: aliasing
* fix: sql, cleanup
* chore: cast
* feat: olap queries
* feat: olap queries
* feat: finish wiring up olap status update metrics
* chore: lint
* chore: lint
* fix: dupes, other code review comments
* send metrics to OTel collector
* last autovac
* flag
* logging updates
* address PR comments
---------
Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com >
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
2025-12-23 01:04:02 +05:30
matt
a4e7584c18
Fix: Last bits of payload job cleanup ( #2690 )
...
* feat: offset cleanup queries
* feat: wire it up
* fix: extend lease interval
* feat: new function for diffing rows
* feat: queries
* fix: signature, start wiring up
* feat: wire up diffing logic at the end
* chore: longer lease
* chore: fix type
* fix: couple bugs
* fix: columns
* fix: dupe
* chore: rm unused function
* fix: err
* fix: reduce memory footprint by a bunch
2025-12-22 12:43:08 -05:00
abelanger5
fe6583fc41
fix: rare cases of duplicate writes causing stuck updates ( #2681 )
...
* fix: rare cases of duplicate writes causing stuck updates
* update to go 1.25
* fix: add version to sdk-go.yml
2025-12-18 13:05:01 -05:00
matt
2226a3eaa4
Chore: Remove payload WAL + corresponding tables 🥳 ( #2645 )
...
* chore: nuke the wal and cutover qi tables
* fix: migration name
2025-12-17 11:27:59 -05:00
matt
bf849d415c
Fix: Payload List Index Performance ( #2669 )
...
* feat: update core func
* feat: migration to update function
* fix: pass batch size through
* fix: limit
* feat: materialized cte
* chore: comment
2025-12-16 12:35:29 -05:00
matt
23db2a4fac
Fix: Pagination by bounds ( #2654 )
...
* fix: pagination missing rows
* fix: separate functions
* fix: return both bounds from query
* fix: wiring
* fix: func
* fix: order col
* fix: bug
* fix: math is hard
* fix: more math
* fix: math and math and math
* fix: slightly more math
* fix: placeholders 🤦
* fix: where clause
* fix: math!
* fix: schema
* refactor: try with `CEIL`
* fix: mathin up a storm
* fix: I was actually a math major in college, who knew
* fix: copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* fix: copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-12-15 13:07:51 -05:00
matt
0a947924fa
Feat: Parallelize replication from PG -> External ( #2637 )
...
* feat: chunking query
* feat: first pass at range chunking
* fix: bug bashing
* fix: function geq
* fix: use maps.Copy
* fix: olap func
* feat: olap side
* refactor: external id
* fix: order by
* feat: wire up env vars
* fix: pass var through
* fix: naming
* fix: append to returnErr properly
* fix: use eg.Go
2025-12-10 17:11:03 -05:00
matt
3ff672ebe4
Fix: Don't reset offset if a new process acquires lease ( #2628 )
...
* fix: don't reset offset if a new process acquires the lease
* fix: copy paste
* feat: migration, fix queries
* fix: more queries
* fix: down migration
* fix: comment
* feat: finish wiring up everything else
* fix: placeholder initial type
* fix: zero values everywhere
* fix: param ordering
* fix: handle no rows
* fix: zero values
* fix: limit
* fix: simplify
* fix: better defaults
2025-12-09 19:01:51 -05:00
matt
9e14814acb
Feat: OLAP Payload Cutover Job ( #2618 )
...
* feat: migration
* feat: queries
* feat: overwrite queries
* fix: bug
* feat: first pass
* fix: more olap job wiring
* fix: signature
* fix: refs to a bunch of funcs
* feat: job
* fix: table names
* fix: span name
* chore: lint
* fix: redundant error check
* fix: naming
* fix: handle nil external id
* fix: order payload partitions descending
* fix: param for limiting which partitions get processed
* fix: olap
2025-12-09 12:33:07 -05:00
matt
7e48ac7d02
Fix: Leasing for payload job ( #2609 )
...
* refactor: acquire a lease instead of an advisory lock
* refactor: partition dates
* fix: single query to acquire / extend
* fix: explicit alias
* fix: unwind
* fix: hwere clause
* fix: handle no rows
* fix: lease bug
* fix: rm debug
* fix: comment for clarity
* fix: syntax that doesn't actually matter
* fix: error
2025-12-05 13:55:59 -05:00
matt
18940869ae
Feat: Job for payload cutovers to external ( #2586 )
...
* feat: initial payload cutover job
* refactor: fix a couple things
* feat: start wiring up writes
* feat: only run job if external store is enabled
* fix: add some notes, add loop
* feat: function for reading out payloads
* fix: date handling, logging
* feat: remove wal and immediate offloads
* feat: advisory lock
* feat: partition swap logic
* fix: rm debug
* fix: add todo
* fix: sql cleanup
* fix: sql cleanup, ii
* chore: nuke a bunch of WAL stuff
* chore: more wal
* feat: trigger for crud opts
* feat: drop trigger + function in swapover
* feat: move autovac to later
* feat: use unlogged table initially
* feat: update migration
* fix: drop trigger
* fix: use insert + on conflict
* fix: types
* refactor: clean up a bit
* fix: panic
* fix: detach partition before dropping
* feat: configurable batch size
* feat: offset tracking in the db
* feat: explicitly lock
* fix: down migration
* fix: bug
* fix: offset handling
* fix: try explicit ordering of the insert
* fix: lock location
* fix: do less stuff after locking
* fix: ordering
* fix: dont drop and recreate if temp table exists
* fix: explicitly track completed status
* fix: table name
* fix: dont use unlogged table
* fix: rm todos
* chore: lint
* feat: configurable delay
* fix: use date as pk instead of varchar
* fix: daily job
* fix: hack check constraint to speed up partition attach
* fix: syntax
* fix: syntax
* fix: drop constraint after attaching
* fix: syntax
* fix: drop triggers properly
* fix: factor out insert logic
* refactor: factor out loop logic
* refactor: factor out job preparation work
* fix: ordering
* fix: run the job more often
* fix: use `WithSingletonMode`
* fix: singleton mode sig
* fix: env var cleanup
* fix: overwrite sig
* fix: re-enable immediate offloads with a flag
* fix: order, offload at logic
* feat: add count query to compare
* fix: row-level triggers, partition time bug
* fix: rm todo
* fix: for true
* fix: handle lock not acquired
* fix: handle error
* fix: comment
2025-12-05 10:54:26 -05:00
Mohammed Nafees
8842a2a9cf
Case on conflict for v1_statuses_olap entry ( #2528 )
...
* case on conflict for v1_statuses_olap
* fix sql
2025-11-14 17:18:35 +01:00
abelanger5
e1fdeeaf1c
fix: payload performance ( #2441 )
...
* change some olap flush settings
* increase timeouts for payload wal
* fix: improve performance of payload wal metrics
* slight updates
* more small tweaks
* undo some olap changes, don't offload some payloads
* remove double reads
* try reducing wal poll limit
* analyze v1_dag
* move partition method
2025-10-23 17:45:49 -04:00
matt
c6e154fd03
Feat: OLAP Payloads ( #2410 )
...
* feat: olap payloads table
* feat: olap queue messages for payload puts
* feat: wire up writes on task write
* driveby: add + ignore psql-connect
* fix: down migration
* fix: use external id for pk
* fix: insert query
* fix: more external ids
* fix: bit more cleanup
* feat: dags
* fix: the rest of the refs
* fix: placeholder uuid
* fix: write external ids
* feat: wire up messages over the queue
* fix: panic
* Revert "fix: panic"
This reverts commit c0adccf2ea36f425f3c1fe430840bdb42a5cacbb95da7de64f8fda64adc4
2025-10-20 09:09:49 -04:00
abelanger5
b16be655be
feat: stateful polling intervals ( #2417 )
...
* initial pass on stateful intervals
* pr review comments + add evict expired idempotency keys
* fix: goroutine leak and name vars better
* fix some cleanup logic
2025-10-15 11:40:22 -04:00
abelanger5
d08bf6aac5
fix: optimize concurrency slot trigger method ( #2391 )
...
* fix: optimize concurrency slot trigger method
* fix: migration ordering
---------
Co-authored-by: mrkaye97 <mrkaye97@gmail.com >
2025-10-08 11:50:54 -04:00
matt
c48a3211b5
Feat: Immediate Payload Offloads ( #2375 )
...
* feat: modify operations
* feat: attempt 1 at doing the cutover + the offload in the same query
* fix: operation write
* debug: add some print lines
* fix: check constraint
* fix: select records to offload properly
* fix: fn
* feat: add second table to hold queued cutovers
* fix: start reworking queries
* fix: select
* fix: missing cols
* fix: for update
* fix: query name for finalize
* feat: cut over query finalizer
* feat: query for writes into cutover queue
* feat: add query for cut over polling
* feat: add cutover job
* fix: rm operations
* feat: write cutover queue items at the same time as setting payload keys
* fix: simplify into single query
* fix: revert debug
* chore: lint
* fix: don't remove operation column yet
* feat: refactor into struct of opts and make job intervals configurable
* fix: add analyze for payload table
* fix: schema copy paste
* fix: drop fk
* feat: add an index to help with poll performance for a short while
* fix: simplify poll ordering
* fix: simplify more
* fix: ctx
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
* Feat: Task Event and DAG Payloads (#2370 )
* feat: initial work on task event payloads
* fix: iterator
* feat: wire up task events
* fix: backwards compat
* fix: migrations
* fix: duplication
* fix: col
* fix: add timestamptz col
* fix: overwrite
* fix: rm debugging
* fix: revert debugging
* fix: rm unused cols
* fix: spelling
* fix: use `current_timestamp` as default
* feat: dual writes for payloads
* fix: improve debug lines
* debug: add log
* debug: always write
* fix: make annoying log debug level
* fix: rm debug lines
* fix: add comment
* feat: dag payloads
* fix: index
* fix: migration ver
* fix: error msg
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* fix: create, then set default
* fix: inserted at copy paste
* fix: n+1 query
* fix: another n+1 query
* fix: rm unused singleton retrieve
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
---------
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2025-10-08 11:22:34 -04:00
matt
dfc5074057
Fix: Payload fallbacks, WAL conflict handling, WAL eviction ( #2372 )
...
* fix: improve error handling
* fix: add default operation
* fix: dont write operation
* fix: refactor offload to always evict
* fix: err check
* fix: err
2025-10-03 14:50:46 -04:00
matt
4730273bce
Fix: Relax check constraint to allow null payloads ( #2366 )
...
* fix: relax check constraint
* fix: tweak logs
* fix: constraint in schema
2025-09-30 12:24:38 -04:00
matt
c759da79aa
Feat: Partition pruning for ListTaskParentOutputs, lookup index for v1_payload_wal ( #2294 )
...
* fix: small query rework, partition pruning for `ListTaskParentOutputs`
* feat: migration for adding index
* fix: copilot comments
2025-09-12 13:18:08 -04:00
matt
92843bb277
Feat: Payload Store Repository ( #2047 )
...
* feat: add table for storing payloads
* feat: add payload type enum
* feat: gen sqlc
* feat: initial sql impl
* feat: add payload store repo to shared
* feat: add overwrite
* fix: impl
* feat: bulk op
* feat: initial wiring of inputs for task triggers
* feat: wire up dag matches
* feat: create V1TaskWithPayload and use it everywhere
* fix: couple bugs
* fix: clean up types
* fix: overwrite
* fix: rm input from replay
* fix: move payload store to shared repo
* fix: schema
* refactor: repo setup
* refactor: repos
* fix: gen
* chore: lint
* fix: rename
* feat: naming, write dag inputs
* fix: more naming, trigger bug
* fix: dual writes for now
* fix: pass in tx
* feat: initial work on offloader
* feat: improve external offloader
* fix: some refs
* add withExternalHandler
* fix: improve impl of external store
* feat: implement offloading, fix other impls
* feat: add query to update JSON
* fix: implement offloading + updating records in payloads table
* feat: add WAL table
* feat: add queries for polling WAL and evicting
* feat: wire up writes into WAL
* fix: get job working
* refactor: improve types
* fix: infinite loop
* feat: improve offloading logic to run in two separate txes
* refactor: rework how overrides work
* fix: lint
* fix: migration number
* fix: migration
* fix: migration version
* fix: revert back to reading payloads out
* fix: fall back to previous input, part i
* fix: input fallback
* fix: add back input to replay
* fix: input fallback in dispatcher
* fix: nil check
* feat: advisory locks, part i
* fix: no skip locked
* feat: hash partitioned wal table
* fix: modify queries a bit, tweak crud enum
* fix: pk order, function to find tenants
* feat: wal processing
* fix: only write wal if an external store is enabled, fix offloading logic
* fix: spacing
* feat: schema cleanup
* fix: rm external store loc name
* fix: set content to null when offloading
* fix: cleanup, naming
* fix: pass overwrite payload store along
* debug: add some logging
* Revert "debug: add some logging"
This reverts commit 43e71eadf1
2025-09-12 09:53:01 -04:00
matt
f385964fcc
Fix: Scheduled runs race w/ idempotency key check ( #2077 )
...
* feat: create table for storing key
* feat: is_filled col
* feat: idempotency repo
* fix: handle filling
* fix: improve queries
* feat: check if was created already before triggering
* fix: handle partitions
* feat: improve schema
* feat: initial idempotency key claiming impl
* fix: db
* fix: sql fmt
* feat: crazy query
* fix: downstream
* fix: queries
* fix: query bug
* fix: migration rename
* fix: couple small issues
* feat: eviction job
* fix: copilot comments
* fix: index name
* fix: rm comment
2025-09-12 07:54:42 -04:00
Gabe Ruttner
59fe6c110e
feat: improved onboarding part 1 ( #2186 )
...
* feat: analytics events
* improved forms
* store state
* lint
* cleanup tenant name
* nits
* add environment to the form
* environment tag
* include env with tenant
* lint
* fix gen
* address comments
* feedback
* fix: layout
* navigation state
* rm dep
* lint
* address review
* lint
* lint
* fix: build
2025-08-25 11:14:34 -07:00
