Commit Graph

125 Commits

Author SHA1 Message Date
abelanger5 1f2096313d feat: v1 engine (#1318) 2025-03-11 14:57:13 -04:00
Gabe Ruttner 46c4b367b1 fix: propagate retry (#1258) 2025-02-10 10:40:24 -08:00
Gabe Ruttner 5cda236e5d fix: addl meta unmarshal deep obj (#1245)
* fix: addl meta unmarshal

* fix: ci

* fix: ci

* fix: ci
2025-02-04 10:59:36 -08:00
Gabe Ruttner 0e91542d87 wip: backoff state (#1225)
* wip: backoff state

* fix: retry state and step run start condition

* fix: missing key

* fix: gen

* chore: squash migration

* chore: rm todos

* ops: upgrade proto
2025-01-28 19:16:12 +00:00
Gabe Ruttner 13024c09bd Feat canceled state (#1228)
* feat: add cancel state to event list

* ops: db conns

* feat: add cancelled status to wfr

* feat: mark cancelled workflow runs
2025-01-28 10:31:04 -08:00
abelanger5 769fed7d97 feat(go-sdk): adds preset labels on workers for autoscaling (#1195)
* feat(go-sdk): adds preset labels on workers for autoscaling

* fix: env var consistency

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
2025-01-28 14:58:41 +00:00
Matt Kaye 0f1cbd98ca Fix: Change which exceptions retry (#1194)
* fix: change retryable exceptions

* fix: add back resource exhausted
2025-01-22 14:54:32 -05:00
Sean Reilly 9e961ac196 Feature add version info (#1154)
* adding a /version endpoint for the engine and a /api/v1/version endpoint for the API

* make the security optional so we don't get redirected for having auth

* lint

* upgrade protoc to the latest available version on brew

* use useQuery and clean up html
2025-01-06 10:50:04 -08:00
abelanger5 a237f90450 fix: circuit breaker for dispatcher reassignment (#1144) 2024-12-20 16:00:23 -05:00
abelanger5 e12e700980 feat: CANCEL_NEWEST strategy and make cancel in progress more reliable (#1127) 2024-12-18 01:40:14 +00:00
Sean Reilly cbc2526c0b add a monitoring probe (#1108)
* add a monitoring probe

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-12-17 15:55:50 -05:00
abelanger5 94d14336aa feat(go-sdk): blocking worker (#1106) 2024-12-12 20:42:13 -05:00
abelanger5 6a0d8c762d feat: shared meta (#1102) 2024-12-12 20:42:01 -05:00
abelanger5 3d616c42b1 feat(go-sdk): expose user data (#1107) 2024-12-12 20:41:37 -05:00
Sean Reilly 279f68c3a4 add pagination for workflows (#1109)
* add some more detail to the list worklflow API endpoint and add pagination and a search function to the frontend that searches the workflow names

* switch

* remove the column filter stuff

* cleanup

* lint and clean up query

* fix linter warning

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-12-12 19:33:43 +00:00
abelanger5 b0c6c7cd46 feat(go-sdk): cron and schedules API, minor fixes (#1083)
* feat(go-sdk): cron and schedules API, minor fixes

* try to improve code block and docs

* revert pre-commit

* fix: generate

* fix: put overflow in right place

* remove branch specs
2024-12-04 21:18:05 +00:00
abelanger5 f821379997 fix: remove deprecated swagger-cli (#1080)
* fix: remove deprecated swagger-cli

* fix: use node:18-alpine
2024-12-03 16:01:46 +00:00
Gabe Ruttner 23201665a6 Feat: dynamic crons fixes (#1073) 2024-11-26 10:47:30 -05:00
Gabe Ruttner 574eb0b67e feat: dynamic crons (#1000)
* wip: stub schedule page

* wip: stub list

* fix: 2025 bug...

* feat: wip cron list

* feat: addl meta

* feat: expose metadata column

* feat: sort and created at

* cron to recurring

* scheduled: with statuses

* fix: links

* feat: expose schedule ids

* feat: delete run

* fix: remove search

* feat: filterable scheduled

* fix: remove broken features

* chore: lint

* rm metadata for now

* chore: lint

* chore: recurring to cron job

* fix: review comments

* fix: populator

* wip cron changes

* fix: ids are helpful

* fix: populator

* wip

* wip: create crons, stub scheduled

* wip: create schedule

* wip add trigger buttons to all the pages

* wip: reusable trigger form

* fix: hash

* fixes: cron bugs

* fixes: cron sort

* fix: out of order migrations

* fix: add internalRetryCount

* feat: api things survive version transitions

* feat: table things

* feat: delete disabled for non api

* feat: prevent delete non api

* feat: filters

* require cron name for api

* default name

* fix: migrations

* frontend improvements and migrations

* fix: pagination

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-11-21 16:18:24 -05:00
Sean Reilly 31e425a858 lets make retry configurable and do not retry for unavailable because the retry is slower than regular heartbeat (#1046)
Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-11-21 13:39:31 -05:00
abelanger5 197bdd1f88 feat: exponential backoff (#1062)
* initial migration

* feat: exp backoff, fix linting

* fix utc issue and cleanup
2024-11-21 13:39:02 -05:00
Gabe Ruttner 3850964a98 feat: initial doc pages (#1020)
* generate initial cloud client

* feat: initial doc pages

* feat: cloud register id, action filtering

* feat:cloud register

* fix: env var

* chore:lint

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-11-08 07:46:43 -08:00
Alexander Belanger 5b59af076e fix: cancellation status propagation and minimap view 2024-11-07 11:13:14 -05:00
Sean Reilly b456382429 add multiple rate limiter in grpc using a token bucket (#984)
* add multiple rate limiter in grpc using a token bucket

* PR feedback

* add in client retry for go client

* update test files

* remove log line only retry on ResourceExhausted and Unavailable

* add some concurrency limits so we don't swamp ourselves

* add some logging for when we are getting backed up

* lets not queue up when we are too full to prevent OOM problems

* fix spelling

* add config options for maximum concurrent and how long to wait for flush , let the wait for flush setting be used as back pressure and a signal to writers that we are slowing up

* lots of changes to buffering

* fix data race

* add some comments explaing how this works, change errors to be ResourceExhausted now that we have client retry and limit how many gofuncs we can create on cleanup and wait for them to finish before we exit

* hooking up the config values so they go to the right place

* Update config.go to default to 1 ms waitForFlush

* disable grpc_retry for client streams

* explicitly set the limit if it is 0

* weirdness because we were using an older version of the lib

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-11-01 11:48:23 +00:00
Gabe Ruttner 44addbb47e Feat scheduled improvements (#992)
* wip: stub schedule page

* wip: stub list

* fix: 2025 bug...

* feat: wip cron list

* feat: addl meta

* feat: expose metadata column

* feat: sort and created at

* cron to recurring

* scheduled: with statuses

* fix: links

* feat: expose schedule ids

* feat: delete run

* fix: remove search

* feat: filterable scheduled

* fix: remove broken features

* chore: lint

* rm metadata for now

* chore: lint

* chore: recurring to cron job

* fix: review comments

* fix: populator
2024-11-01 07:16:20 -04:00
Gabe Ruttner 4932e7f863 Feat sdk runtime (#942)
* feat: runtime signature

* feat: add sdk runtime to worker model

* feat: post runtime

* feat: expose sdk version on worker

* feat: go inf

* chore: gen

* chore: migrations and generation

* fix: simpler runtime

* feat: hatchet sdk ver

* fix: rm debug line
2024-10-28 13:47:12 -07:00
Gabe Ruttner 7cd08077d5 feat: improved sdk ack (#931)
* feat: add step run event reasons

* feat: ack

* fix: remove rejected reason

* fix: merge

* fix: correct buffer

* fix: consistent message

* chore: rm todo
2024-10-15 15:52:42 +00:00
abelanger5 19e151e29a fix: RunWorkflow and SpawnWorkflow should respond with consistent APIs (#965) 2024-10-15 11:09:58 -04:00
Sean Reilly 29721cd1f0 Feat bulk workflows (#940)
Adds support for inserting workflows in bulk via the API and an optional buffered insert on the engine.
2024-10-14 15:35:29 -04:00
abelanger5 b4c861d7a1 patch: release semaphore slots before jobs controller (#927)
* fix: don't need acks on queue checks

* patch: release semaphores early

* proper list on high queue depth

* fix: don't release on started
2024-10-02 11:36:05 -04:00
Gabe Ruttner 7d7e43d4e1 feat: pauseable workflows (#879)
* feat: pause workflow state

* feat: dont run paused workflows

* feat: skipped paused

* implement unpaused behavior for workflow runs

* fix: frontend

* fix: more frontend

* fix: imports

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-09-29 10:58:10 -04:00
abelanger5 925b2654c8 feat: workflow run metrics view (#912)
* feat: add callbacks for workflow run completed

* add tenant id to resolve row

* add finishedBefore, finishedAfter to workflow runs query

* add more callbacks

* feat: tenant ids and loggers in callback

* feat: workflow run metrics frontend

* fix: frontend build
2024-09-27 07:38:15 -04:00
abelanger5 a1a10b4073 feat: dynamic rate limits (#904)
* wip: step run expressions on rate limits

* feat: dynamic rate limits

* chore: v0.47.0

* chore: address changes from PR review

* fix: improved error handling

* address pr review

* better error messages for step run cels, remove debug logs

* fix: hash

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
2024-09-26 22:00:34 +00:00
abelanger5 840e590312 fix: frontend improvements (#905)
* fix: set time range properly on reload

* fix: small button to show queue counts for now
2024-09-24 19:08:41 -04:00
Sean Reilly 5811929928 feat: bulk inserts of events (#887)
* progress commit of bulk inserts

* in_flight: Add changes to metering finish the bulk insert

* remove an attempt to overide enforce limits

* merge in PR fixes

* update docs to add in an additional section in the User guide to describe pushing single events and pushing multiple events

* run lint fix

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-09-23 09:19:39 -07:00
Sean Reilly 15c50f46b5 Partial PR - need to generate SDK - Add endpoint to get the total free worker slots for a worker and the … (#857)
* Add endpoint to get the total free worker slots for a worker and the max runs

* update to use WorkerSempahoreCount instead of checking stepRunId

* modify the query for the new table and change the interface

* bump golangci-lint make changes to name of returned data

* revert the simple example

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-09-19 10:11:16 -07:00
abelanger5 d23e5d9963 feat: expression-based concurrency keys (#889)
* feat: expression-based concurrency keys

* fix: build

* fix: typos

* fix: gen

* fix: migration

* fix: remove print statements

* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
Steinway Wu 44d03af852 fix: propagating additional metadata for child workflows (#882)
* fix: propagating additional metadata for child workflows

* add unit test
2024-09-19 13:28:46 +00:00
Gabe Ruttner af9ed49f1e fix: events list view (#878)
* fix: filter by event id

* fix: run count

* feat: filter by id api

* feat: filter by Event Id

* chore: default page is runs

* feat: cancel event runs

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-09-16 16:46:31 +00:00
Gabe Ruttner c64c62f66a feat: improved workflow run details page (#821)
* wip: rip prisma

* wip

* wip

* fix: lint

* wip

* wip

* gen

* wip

* wip

* fix trigger

* hide overview

* revert db changes

* feat: wrap up frontend changes and perf

* chore: generate

* chore: frontend build

* fix: workflow transformer

* fix: avoid race conditions on simultaneous parent completions

* fix: 2025 started

* feat: toast for replay/cancel

* fix: toast

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-09-16 15:39:49 +00:00
abelanger5 891514b461 feat: queue v4 (#842)
* wip: v4 of queue

* fix: correct query for updating counts

* tmp: save migration files

* feat: wrap up initial queue

* fix compilation

* fix: reassigns
2024-09-06 16:12:22 -04:00
Gabe Ruttner 53be615d5f Enhancement webhook usability (#807)
* feat: secret copier

* feat: improved form

* fix: quotes

* wip: improved flow

* feat: health check logging

* fix: page design

* fix: hard delete, no upsert

* fix: reset modal state

* fix: empty text

* fix: worker state

* fix: update only token

* fix: dont delete name

* fix: logs component

* fix: sort order

* chore: build

* fix: webhook worker cleanup

* chore: squash migrations

* Update api-contracts/openapi/paths/webhook-worker/webhook-worker.yaml

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>

* chore: rename

* fix: wrong query

---------

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>
2024-08-23 10:09:09 -04:00
Gabe Ruttner 9bea55438a Fix webhook healthcheck race (#797)
* fix: race

* fix: partition no rows

* chore: move to workers tab

* feat: redirect empty worker path to all

* chore: add worker type and webhook id

* fix: upsert webhook worker

* fix: update by webhookId

* fix: only stub on create

* feat: url on worker

* chore: migration version

* fix: move

* fix: upsert

* fix: upert

* chore: fix migration

* fix: migrations

* chore: generate
2024-08-21 19:23:24 +00:00
abelanger5 84f7334a06 feat: add windows for metrics and selector (#794)
* feat: add windows for metrics and selector

* better placeholder
2024-08-20 15:29:32 +00:00
abelanger5 dd50515aeb fix: cancelling status on frontend (#779)
* fix: cancelling status on frontend

* fix: slot grid and dedupe on slots
2024-08-13 17:01:26 +00:00
Gabe Ruttner 4ea4712d4d refactor: performance and throughput (#756)
Refactors the queueing logic to be fairly balanced between actions, with each action backed as a separate FIFO queue. Also adds support for priority queueing and custom queues, though those aren't exposed on the API layer yet. Improves throughput to be > 5000 tasks/second on a single queue. 

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-08-12 14:38:47 +00:00
Viktor Szépe 0948598749 Fix typos (#775) 2024-08-10 10:58:33 +00:00
Gabe Ruttner b802f9f45f feat: stream by addl meta (#751)
* feat: prop schedule and run

* wip

* fix: filter wfrid

* feat: hangup

* chore: rm debug log

* chore: func name

* fix: cancelled payload

* fix: load

* fix: cleanup the cahce

* fix: single proto

* fix: key -> val

* chore: case

* chore: rm dead code

* chore: rm dead code

* feat: go and docs

* fix: docs
2024-07-29 19:09:51 +00:00
abelanger5 a245151d91 feat: add workflow kind to workflow versions (#750)
* feat: support workflow kinds

* chore: generate
2024-07-29 12:07:34 -07:00
abelanger5 9efd9368fd feat: deduplicated enqueue error and additional context methods (#747)
* feat: additional context fields and dedupe error

* fix: case on error properly
2024-07-26 18:32:56 +00:00