Mohammed Nafees
cf5c5989ff
add vars to tune concurrency poller ( #2428 )
2025-10-23 11:36:12 -04:00
abelanger5
1f35782b59
fix: move err check to before len check ( #2437 )
2025-10-21 19:24:19 -04:00
matt
c6e154fd03
Feat: OLAP Payloads ( #2410 )
...
* feat: olap payloads table
* feat: olap queue messages for payload puts
* feat: wire up writes on task write
* driveby: add + ignore psql-connect
* fix: down migration
* fix: use external id for pk
* fix: insert query
* fix: more external ids
* fix: bit more cleanup
* feat: dags
* fix: the rest of the refs
* fix: placeholder uuid
* fix: write external ids
* feat: wire up messages over the queue
* fix: panic
* Revert "fix: panic"
This reverts commit c0adccf2ea36f425f3c1fe430840bdb42a5cacbb95da7de64f8fda64adc4
2025-10-20 09:09:49 -04:00
Mohammed Nafees
8f57989730
fix race condition in child spawn ( #2429 )
2025-10-17 16:56:41 +02:00
Mohammed Nafees
e2b1f1353e
Fix OTel span attribute naming convention ( #2409 )
...
* rename spans according to convention
* low cardinality
2025-10-16 18:43:40 +02:00
Mohammed Nafees
d9268c7270
Cleanup job for old and invalid entries ( #2378 )
...
* auto run table cleanup
* batched cleanup of tables
* address PR comments
* fix timeout
* update queries
* fix shouldContinue
* also call cleanup for v1_workflow_concurrency_slot
* fix comment
* comment fix
2025-10-16 16:51:08 +02:00
matt
aa38c6d2df
fix: payload fallback for child runs ( #2421 )
2025-10-15 16:16:51 -04:00
abelanger5
b16be655be
feat: stateful polling intervals ( #2417 )
...
* initial pass on stateful intervals
* pr review comments + add evict expired idempotency keys
* fix: goroutine leak and name vars better
* fix some cleanup logic
2025-10-15 11:40:22 -04:00
matt
5b5adcb8ed
Feat: Scheduled run detail view, bulk cancel / replay with pagination helper ( #2416 )
...
* feat: endpoint for listing external ids
* feat: wire up external id list
* chore: regen api
* feat: py sdk wrapper
* fix: since type
* fix: log
* fix: improve defaults for statuses
* feat: docs
* feat: docs
* fix: rm extra file
* feat: add id column to scheduled runs
* feat: side panel for scheduled runs
* fix: side panel header pinned
* fix: border + padding
* chore: gen
* chore: lint
* chore: changelog, version
* fix: spacing of cols
* fix: empty webhook resource limit
* fix: tsc
* fix: sort organizations and tenants alphabetically
2025-10-15 11:36:45 -04:00
Mohammed Nafees
a750ce950d
Introduce vars to tune ANALYZE job gocron run intervals ( #2407 )
...
* introduce cars to tune ANALYZE job gocron run intervals
* update config doc
* fix assignment
2025-10-10 11:02:10 +02:00
Mohammed Nafees
0695db820c
Use UTC for all pgx connections and check for database TZ ( #2398 )
...
* set utc for all pgx sessions
* helper func
* also accept IANA Etc/UTC
2025-10-09 10:54:27 +02:00
matt
d677cb2b08
feat: gzip compression for large payloads, persistent OLAP writes ( #2368 )
...
* debug: remove event pub
* add additional spans to publish message
* debug: don't publish payloads
* fix: persistent messages on olap
* add back other payloads
* remove pub buffers temporarily
* fix: correct queue
* hacky partitioning
* add back pub buffers to scheduler
* don't send no worker events
* add attributes for queue name and message id to publish
* add back pub buffers to grpc api
* remove pubs again, no worker writes though
* task processing queue hashes
* remove payloads again
* gzip compression over 5kb
* add back task controller payloads
* add back no worker requeueing event, with expirable lru cache
* add back pub buffers
* remove hash partitioned queues
* small fixes
* ignore lru cache top fn
* config vars for compression, disable by default
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2025-10-08 11:44:04 -04:00
matt
c48a3211b5
Feat: Immediate Payload Offloads ( #2375 )
...
* feat: modify operations
* feat: attempt 1 at doing the cutover + the offload in the same query
* fix: operation write
* debug: add some print lines
* fix: check constraint
* fix: select records to offload properly
* fix: fn
* feat: add second table to hold queued cutovers
* fix: start reworking queries
* fix: select
* fix: missing cols
* fix: for update
* fix: query name for finalize
* feat: cut over query finalizer
* feat: query for writes into cutover queue
* feat: add query for cut over polling
* feat: add cutover job
* fix: rm operations
* feat: write cutover queue items at the same time as setting payload keys
* fix: simplify into single query
* fix: revert debug
* chore: lint
* fix: don't remove operation column yet
* feat: refactor into struct of opts and make job intervals configurable
* fix: add analyze for payload table
* fix: schema copy paste
* fix: drop fk
* feat: add an index to help with poll performance for a short while
* fix: simplify poll ordering
* fix: simplify more
* fix: ctx
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
* Feat: Task Event and DAG Payloads (#2370 )
* feat: initial work on task event payloads
* fix: iterator
* feat: wire up task events
* fix: backwards compat
* fix: migrations
* fix: duplication
* fix: col
* fix: add timestamptz col
* fix: overwrite
* fix: rm debugging
* fix: revert debugging
* fix: rm unused cols
* fix: spelling
* fix: use `current_timestamp` as default
* feat: dual writes for payloads
* fix: improve debug lines
* debug: add log
* debug: always write
* fix: make annoying log debug level
* fix: rm debug lines
* fix: add comment
* feat: dag payloads
* fix: index
* fix: migration ver
* fix: error msg
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* fix: create, then set default
* fix: inserted at copy paste
* fix: n+1 query
* fix: another n+1 query
* fix: rm unused singleton retrieve
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
---------
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2025-10-08 11:22:34 -04:00
matt
8fd90a29a6
Feat: Pausable Crons ( #2395 )
...
* feat: update query, patch route
* feat: api for update
* fix: simplify ui a bit
* feat: wire up fe
* feat: improve copy, spinners
* fix: invert naming to avoid horrible double negative
* fix: improve handling of optional types
* fix: last bits of naming
* feat: persist enabled flag across workflow versions properly
* fix: update spinner
2025-10-08 11:12:14 -04:00
matt
dfc5074057
Fix: Payload fallbacks, WAL conflict handling, WAL eviction ( #2372 )
...
* fix: improve error handling
* fix: add default operation
* fix: dont write operation
* fix: refactor offload to always evict
* fix: err check
* fix: err
2025-10-03 14:50:46 -04:00
Mohammed Nafees
ed40a82dbb
Include tenant_id in OTel spans wherever possible ( #2382 )
2025-10-03 18:16:16 +02:00
matt
bb1de91254
fix: run analyze every 3 hours ( #2380 )
2025-10-03 09:49:35 -04:00
icbd
c8bd3cf93c
Add ApplyNamespace for BulkRunWorkflow ( #2374 )
2025-10-02 13:33:46 +02:00
matt
202d6b0184
Candidate Fix: WAL Write Dupes ( #2369 )
...
* fix: wal dupes
* fix: make unique key align with payload pk
2025-09-30 17:11:35 -04:00
matt
4730273bce
Fix: Relax check constraint to allow null payloads ( #2366 )
...
* fix: relax check constraint
* fix: tweak logs
* fix: constraint in schema
2025-09-30 12:24:38 -04:00
abelanger5
2edeeb10ea
feat: max channels for rabbitmq ( #2365 )
...
* feat: max conns for rabbitmq
* rename conns -> chans
2025-09-30 08:49:45 -04:00
Mohammed Nafees
806931f8c6
ignore tenants with deletedAt non null ( #2364 )
2025-09-30 08:01:23 -04:00
abelanger5
43eb47b881
fix(go-sdk): async start step run action event ( #2351 )
...
* fix(go-sdk): async start step run action event
* use inner ctx
2025-09-26 14:03:37 -04:00
matt
c1e5fa828b
FE Polish, VI: Make badges dynamically sized, use slate instead of fuchsia for queued, display ms on dates ( #2352 )
...
* fix: rm w-full from badges
* fix: use slate instead of fuchsia for queued
* fix: use `date-funs` to parse + display dates
* fix: worker styling
* debug: redocly
* debug: redocly
* debug: redocly
* debug: redocly
* debug: redocly
* debug: redocly
* feat: direct workflow id getter
* fix: dedupe
* fix: return workflow type
* chore: lint
* chore: lint
* fix: overflow
* fix: include tenant in query for safety
2025-09-26 13:59:06 -04:00
matt
126ff3771b
Fix: Rename metrics queries, always refetch queue metrics, change default refetch interval, configurable WAL poll limit ( #2346 )
...
* fix: improve query naming + change refetch interval for queue metrics
* fix: only enable queue metrics query when it's open
* feat: change default
* fix: change storage key to clear cache
* fix: queue metrics loading state
* feat: introduce poll limit with default of 1000
* feat: set WAL poll limit
* debug: log line to test
* fix: export helper
* fix: rm debug
2025-09-25 20:44:39 -04:00
abelanger5
3fe1cdde37
fix: improve DAG status updates ( #2343 )
...
* fix: improve DAG status updates
* fix: error msg
2025-09-25 12:22:35 -04:00
matt
d6afe55338
FE Polish V: Searchable workflows, additional metadata tab ( #2342 )
...
* feat: search for workflows
* fix: allow fts for workflow name
* feat: add back metadata tab to details
* feat: dynamic copy
* chore: lint
2025-09-25 10:58:48 -04:00
matt
ee17433ac9
Fix: Event getter backwards compat ( #2337 )
...
* feat: query for getting event in v1
* feat: extend populator to fetch v1 event
* fix: rm debug
* fix: simplify join
* fix: ctx
2025-09-24 17:10:20 -04:00
matt
df1dab945b
Hotfix: Stable ordering for flattened tasks + child tasks ( #2334 )
...
* fix: stable ordering with flattened tasks / child tasks
* fix: `Equal` method
2025-09-24 12:46:27 -04:00
matt
63c644577a
Feat: Add error level logs if we fall back to the task input for monitoring ( #2328 )
...
* feat: logs on fallback to input from task
* drive-by: couple more status badge colors
* fix: durable sleep matches
2025-09-23 15:48:30 -04:00
Mohammed Nafees
cf21550502
Update docs to use Go SDK v1 ( #2313 )
...
* SDK fixes
* go docs generate
* simple changes
* more docs changes
* bulk docs done
* cancellation example
* concurrency example
* conditional example
* cron example
* dag example
* durable event example
* durable sleep example
* on failure example
* priority example
* rate limit example
* retries example
* run-no-wait example
* on event example
* run with results example
* running your task example
* scheduled runs example
* streaming example
* workers example
* timeouts example
* sticky example
* docker go
* fix lint and build
* update migration doc
* fix lint
* fix some docs
* fix docker mdx
* remove local lint
* go stub example
* make changes
* child spawning
* migration code examples
* fix child workflow example
2025-09-23 19:19:27 +02:00
abelanger5
06c4bb4f24
feat: support dynamic rate limit durations ( #2320 )
2025-09-21 17:01:11 -04:00
abelanger5
7088925ea6
fix: skip locked on queue updates ( #2321 )
2025-09-21 17:01:02 -04:00
matt
30e0c0b89e
Fix: Payload WAL dupes ( #2319 )
...
* fix: payload wal
* fix: spacing
2025-09-18 21:54:23 -04:00
matt
8b8ded655d
Fix: Update payload properly on replay ( #2317 )
...
* fix: overwrite payloads when task is in an initially e.g. cancelled state
* fix: add distinct to payload writes to limit conflict resolution
* feat: first pass at test
* fix: tenant in warning
* fix: lint, more assertions
* fix: bug
* fix: my pet peeve
2025-09-18 20:42:39 -04:00
matt
1108cf29d3
Fix: Payloads OLAP backwards compat ( #2316 )
...
* fix: backwards compat
* fix: copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Revert "fix: copilot"
This reverts commit 47f4f8f64d175728472+Copilot@users.noreply.github.com >
2025-09-18 11:53:16 -04:00
matt
3ebf76b638
Fix: Event filtering edge case ( #2311 )
...
* fix: bugs in python tests + extend them to assert more
* fix: bug
* chore: lint
* fix: typo
* fix: longer timeout
* debug: more iterations
* debug: try fixing test again
* fix: allocate memory initially
2025-09-18 09:48:01 -04:00
Mohammed Nafees
da6c23ede4
Allow RabbitMQ to be used with Hatchet Lite ( #2128 )
...
* allow rabbitmq to be used with lite
* fix docker image
* fix env vars
* WIP
* fix mq kind selection
* lite specific env vars
* failure check
* changes to logic
* fix failing tests
* fix e2e test
2025-09-16 22:34:53 +02:00
Mohammed Nafees
c1ebeb518a
Fix GetDetails in Runs feature client of Go SDK v1 ( #2297 )
...
* fix GetDetails underlying API call
* use Get instead
2025-09-16 12:52:31 -04:00
matt
bdedab653a
Fix: WAL partition poll function type ( #2301 )
...
* fix: type
* fix: cast to int32
* debug: add logging
* debug: more logs
* Revert "debug: more logs"
This reverts commit 2ff8033f89a7aaa05b9c
2025-09-16 12:44:55 -04:00
matt
8ae760dd15
fix: revert partition pruning ( #2295 )
2025-09-12 17:33:13 -04:00
Mohammed Nafees
47a065df83
Add panic handler to Go SDK ( #2293 )
2025-09-12 19:49:51 +02:00
matt
c759da79aa
Feat: Partition pruning for ListTaskParentOutputs, lookup index for v1_payload_wal ( #2294 )
...
* fix: small query rework, partition pruning for `ListTaskParentOutputs`
* feat: migration for adding index
* fix: copilot comments
2025-09-12 13:18:08 -04:00
matt
92843bb277
Feat: Payload Store Repository ( #2047 )
...
* feat: add table for storing payloads
* feat: add payload type enum
* feat: gen sqlc
* feat: initial sql impl
* feat: add payload store repo to shared
* feat: add overwrite
* fix: impl
* feat: bulk op
* feat: initial wiring of inputs for task triggers
* feat: wire up dag matches
* feat: create V1TaskWithPayload and use it everywhere
* fix: couple bugs
* fix: clean up types
* fix: overwrite
* fix: rm input from replay
* fix: move payload store to shared repo
* fix: schema
* refactor: repo setup
* refactor: repos
* fix: gen
* chore: lint
* fix: rename
* feat: naming, write dag inputs
* fix: more naming, trigger bug
* fix: dual writes for now
* fix: pass in tx
* feat: initial work on offloader
* feat: improve external offloader
* fix: some refs
* add withExternalHandler
* fix: improve impl of external store
* feat: implement offloading, fix other impls
* feat: add query to update JSON
* fix: implement offloading + updating records in payloads table
* feat: add WAL table
* feat: add queries for polling WAL and evicting
* feat: wire up writes into WAL
* fix: get job working
* refactor: improve types
* fix: infinite loop
* feat: improve offloading logic to run in two separate txes
* refactor: rework how overrides work
* fix: lint
* fix: migration number
* fix: migration
* fix: migration version
* fix: revert back to reading payloads out
* fix: fall back to previous input, part i
* fix: input fallback
* fix: add back input to replay
* fix: input fallback in dispatcher
* fix: nil check
* feat: advisory locks, part i
* fix: no skip locked
* feat: hash partitioned wal table
* fix: modify queries a bit, tweak crud enum
* fix: pk order, function to find tenants
* feat: wal processing
* fix: only write wal if an external store is enabled, fix offloading logic
* fix: spacing
* feat: schema cleanup
* fix: rm external store loc name
* fix: set content to null when offloading
* fix: cleanup, naming
* fix: pass overwrite payload store along
* debug: add some logging
* Revert "debug: add some logging"
This reverts commit 43e71eadf1
2025-09-12 09:53:01 -04:00
matt
f385964fcc
Fix: Scheduled runs race w/ idempotency key check ( #2077 )
...
* feat: create table for storing key
* feat: is_filled col
* feat: idempotency repo
* fix: handle filling
* fix: improve queries
* feat: check if was created already before triggering
* fix: handle partitions
* feat: improve schema
* feat: initial idempotency key claiming impl
* fix: db
* fix: sql fmt
* feat: crazy query
* fix: downstream
* fix: queries
* fix: query bug
* fix: migration rename
* fix: couple small issues
* feat: eviction job
* fix: copilot comments
* fix: index name
* fix: rm comment
2025-09-12 07:54:42 -04:00
matt
ced1ad43aa
Feat: Filters UI, Events page refactor, Misc. other fixes ( #2276 )
...
* feat: initial filters page work
* feat: cols
* feat: start implementing filters in the hook
* fix: rm some effects
* fix: filter pagination
* fix: placeholder data, query key
* refactor: factor out popover + columns
* fix: tsc
* fix: empty state
* fix: move filters tab down
* fix: remove run id from scheduled runs table
* fix: rm unused
* fix: rm tmp change
* fix: rm one more
* fix: lint
* fix: colname
* feat: declarative checkmark, col key to name mapping
* fix: add more col key to name mappings
* fix: rm log
* fix: rm id checkboxes for events and filters
* refactor: start fixing event cols
* refactor: events hook
* fix: event id
* fix: clickable id
* fix: unwind example
* feat: delete mutation
* feat: form + schema
* feat: update + create mutations
* feat: filter detail
* feat: filter create
* feat: modify cols
* feat: filter detail side panel
* fix: json fallback
* fix: cursors, etc
* fix: cursor
* fix: invalidate
* fix: modal copy
* chore: lint
* fix: openapi spec
2025-09-11 13:41:12 -04:00
Gabe Ruttner
9459dad14d
Feat improve auth error handling ( #1893 )
...
* common errors
* rate limits
* add IP extractor to api server
* use echo rate limit middleware func
* use rate limit for webhooks as well
---------
Co-authored-by: Mohammed Nafees <hello@mnafees.me >
2025-09-11 18:30:07 +02:00
Mohammed Nafees
ebb49cb1a0
error out instead of panic ( #2274 )
2025-09-09 17:48:55 +02:00
matt
cf59a7bcd9
Feat: Worker slot Prom metrics ( #2195 )
...
* feat: add slots to prom metrics
* feat: available
* fix: extension instead
* fix: docs
* fix: rm unused query changes
* fix: rm unused struct
* fix: labels
* feat: improve total slots
* fix: pr feedback
* fix: docs
* Revert "fix: docs"
This reverts commit 7fe105da92
2025-09-08 14:07:44 -04:00
Mohammed Nafees
9b0ec2618e
Go SDK v1 feature client changes ( #2160 )
...
* feature client changes
* remove code duplication
* func name should make sense
* add simple compile gh workflow
2025-09-08 17:10:24 +02:00
