d23e5d9963
feat: expression-based concurrency keys ( #889 )
...
* feat: expression-based concurrency keys
* fix: build
* fix: typos
* fix: gen
* fix: migration
* fix: remove print statements
* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
44d03af852
fix: propagating additional metadata for child workflows ( #882 )
...
* fix: propagating additional metadata for child workflows
* add unit test
2024-09-19 13:28:46 +00:00
53be615d5f
Enhancement webhook usability ( #807 )
...
* feat: secret copier
* feat: improved form
* fix: quotes
* wip: improved flow
* feat: health check logging
* fix: page design
* fix: hard delete, no upsert
* fix: reset modal state
* fix: empty text
* fix: worker state
* fix: update only token
* fix: dont delete name
* fix: logs component
* fix: sort order
* chore: build
* fix: webhook worker cleanup
* chore: squash migrations
* Update api-contracts/openapi/paths/webhook-worker/webhook-worker.yaml
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* chore: rename
* fix: wrong query
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-08-23 10:09:09 -04:00
9bea55438a
Fix webhook healthcheck race ( #797 )
...
* fix: race
* fix: partition no rows
* chore: move to workers tab
* feat: redirect empty worker path to all
* chore: add worker type and webhook id
* fix: upsert webhook worker
* fix: update by webhookId
* fix: only stub on create
* feat: url on worker
* chore: migration version
* fix: move
* fix: upsert
* fix: upert
* chore: fix migration
* fix: migrations
* chore: generate
2024-08-21 19:23:24 +00:00
0948598749
Fix typos ( #775 )
2024-08-10 10:58:33 +00:00
9efd9368fd
feat: deduplicated enqueue error and additional context methods ( #747 )
...
* feat: additional context fields and dedupe error
* fix: case on error properly
2024-07-26 18:32:56 +00:00
fd947cb5bc
feat: go worker assignment ( #741 )
...
* feat: create worker with label
* feat: worker context
* feat: dynamic labels
* feat: affinity
* fix: ptr
* fix: nil labels
* feat: sticky dag
* feat: sticky docs
* feat: sticky children
* chore: lint
* fix: tests
* fix: possibly nil workerId
* chore: cleanup unneeded pointers
2024-07-26 10:19:11 -07:00
62e02495c9
fix(webhooks): remove unused workflows field ( #665 )
2024-07-02 15:24:23 +01:00
8a991c1cb0
fix(go-sdk): add schedule timeout ( #673 )
...
* fix(go-sdk): add schedule timeout
* chore: lint
2024-06-30 11:38:39 -04:00
b88989d825
fix(webhooks): send ActionPayload as string ( #657 )
2024-06-26 23:20:50 +00:00
a51681ddc6
fix(webhooks): use PUT for healthcheck ( #644 )
2024-06-26 10:39:11 -04:00
5ffa3d1839
fix(worker): deprecate On in favor of RegisterWorkflow ( #627 )
2024-06-25 22:26:04 +00:00
1490d88954
feat: webhook workers ( #542 )
...
Adds serverless support via the concept of webhook workers. Allows any webhook to be registered as a serverless endpoint for executing a step.
2024-06-25 17:06:43 -04:00
7c3ddfca32
feat: api server extensions ( #614 )
...
* feat: allow extending the api server
* chore: remove internal packages to pkg
* chore: update db_gen.go
* fix: expose auth
* fix: move logger to pkg
* fix: don't generate gitignore for prisma client
* fix: allow extensions to register their own api spec
* feat: expose pool on server config
* fix: nil pointer exception on empty opts
* fix: run.go file
2024-06-19 09:36:13 -04:00
e09ee98df5
feat: expose retry count ( #524 )
...
* feat: expose retry count
* feat: expose retry count go
* docs: accessing retry count
* fix: import
* fix: tests
* fix: docs formatting
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-06-07 15:27:38 +00:00
5f93a41e8b
fix(go-sdk): propagate namespace to child workflows ( #562 )
2024-06-06 12:36:50 -04:00
e6617e1381
feat: refresh timeout ( #495 )
...
* feat: show timeout on step run
* feat: refresh timeout
* fix: increment timeout from last time
* feat: add step run event for timeout refresh
* chore: generate
* feat: add refreshing timeouts docs section
* chore: linting
* feat: go client
* chore: generate
* chore: generate
* fix: test context
* chore: generate
* fix: rm txn
* fix: validator
* chore: lint
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-05-16 12:23:36 -04:00
48d06b931a
feat: client releasable slots ( #476 )
...
* feat: add release slot proto
* feat: add semaphore release state and methods
* feat: go sdk and example
* docs: manual slot release
* chore: linting
* fix: broken test
* fix: unlink step run on manual release
* feat: release slot event
* fix: test
* fix: revert e2e test changes
* chore: remove debug line
* fix: place step run query in same tx
* fix: change migration release version
---------
Co-authored-by: Alexander Belanger <belanger@sas.upenn.edu >
2024-05-14 15:22:46 -04:00
7543a0c2a5
add jobs which always run on failure ( #445 )
...
* (wip) prisma schema
* feat: on-failure steps
* chore: address changes from PR review
* chore: bump migration number
2024-05-06 15:39:22 -04:00
671a7e1474
feat(go-sdk): capture panics and send to alerter ( #403 )
2024-04-22 05:35:40 -04:00
4ce1dd8632
feat: multi-workflow runs listener on a single endpoint
...
* new api-contract for workflow run events
* feat: initial implementation for new subscribe listener
* fix: sync issues and send workflow runs immediately
* refactor: add context to all engine db queries, fix deadlocking query
* fix: use new ctx for deleting dispatcher and ticker
* add cancellation reasons
* fix: docs linting
---------
Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com >
2024-04-18 20:55:11 -04:00
f43f32283c
feat(py/go): namespaces ( #354 )
...
* feat: namespaced python
* wip: namespaced go
* fix: service name
* fix: tests
* feat: client WithNamespace
* feat: namespace example
* feat: namespaced event triggers
* docs: namespace docs
* chore: linting
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-15 11:26:05 -07:00
3c112d9df8
fix(workflow): remove workflow timeout in favor of step timeout ( #366 )
2024-04-10 23:53:00 +07:00
066b3c5b71
feat(engine): initial rate-limiting engine implementation ( #324 )
...
* feat(engine): initial rate-limiting engine implementation
* fixes and implement go sdk rate limiting
2024-04-02 10:53:03 -04:00
d8b6843dec
feat: streaming events ( #309 )
...
* feat: add stream event model
* docs: how to work with db models
* feat: put stream event
* chore: rm comments
* feat: add stream resource type
* feat: enqueue stream event
* fix: contracts
* feat: protos
* chore: set properties correctly for typing
* fix: stream example
* chore: rm old example
* fix: async on
* fix: bytea type
* fix: worker
* feat: put stream data
* feat: stream type
* fix: correct queue
* feat: streaming payloads
* fix: cleanup
* fix: validation
* feat: example file streaming
* chore: rm unused query
* fix: tenant check and read only consumer
* fix: check tenant-steprun relation
* Update prisma/schema.prisma
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* chore: generate protos
* chore: rename migration
* release: 0.20.0
* feat(go-sdk): implement streaming in go
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-04-01 15:46:21 -04:00
7b7fbe3668
fix: update Requeue and Reassign logic to fix performance degradation when many events are queued ( #310 )
...
Logic for requeueing and reassigning did not limit the number of step runs to requeue, so when events accumulate with no worker present it causes memory to spike along with a very high query latency on the database. This commit limits the number of step runs returned in the requeue and reassign queries, and also properly locks step run rows for these queries so only a step run in a PENDING or PENDING_ASSIGNMENT state can be requeued.
It also improves performance of the `AssignStepRunToWorker` query and ensures that `maxRuns` on workers are always respected through the introduction of a `WorkerSemaphore` model. The value gets decremented when a step run is assigned and incremented when a step run is in a final state.
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
* Update controller.go
---------
Co-authored-by: steebchen <contact@luca-steeb.com >
2024-04-01 12:33:18 -04:00
77e5d2b77c
feat(go-sdk): spawnWorkflow method and get up to speed with other sdks ( #297 )
...
* feat(go-sdk): spawnWorkflow method and get up to speed with other sdks
* fix: manual trigger example
* fix: linting errors
* fix: double serialization from go sdk
* fix: spawn workflow logic and procedural example
* test(e2e): add procedural test
* fix: panic in e2e test
* fix: e2e test preparation
* fix: api server url in test.yml
* fix: load test server url
* chore: make num children configurable
* address pr review
2024-03-29 14:07:39 -07:00
9b68115fb5
refactor: cleanup functions in api + worker ( #192 )
2024-03-02 00:37:02 +07:00
7709bcb175
fix(engine): requeue improvements ( #200 )
...
* fix: queue improvements
* fix: bugs
* fix: e2e tests
* fix: diff
2024-02-27 12:02:31 -05:00
6ea38a99f2
feat: support maxRuns parameter on workers ( #195 )
...
* feat: round robin queueing
* feat: configurable max runs per worker
* fix: address PR review
* docs for max runs and group round robin
2024-02-26 00:48:46 -05:00
2d625fec81
feat: round robin queueing ( #194 )
2024-02-26 00:16:40 -05:00
df3f540748
feat: add retries to the engine and SDKs ( #171 )
...
This PR adds support for retrying failed step runs against the engine and SDKs. This was tested up to 30 retries per step run, with both failure and success at the 30th step run. Each SDK now has a `retries` configurable param for steps when declaring a workflow.
2024-02-16 13:00:22 -05:00
00111d823c
test(load): add load tests CLI & e2e tests ( #157 )
2024-02-16 23:47:34 +07:00
d63b66a837
feat: concurrency groups ( #135 )
...
* first pass at moving controllers around
* feat: concurrency limits for strategy CANCEL_IN_PROGRESS
* fix: linting
* chore: bump python sdk version
2024-01-30 00:00:28 -05:00
14a971bfd7
feat: improve versioning to use checksums ( #129 )
...
* feat: improve versioning to use checksums
* fix: remove json ignore for scheduled triggers
* update migration with random strings for checksum
2024-01-26 20:47:15 -05:00
c8f87599c7
chore: ignore or fix linter errors ( #116 )
...
* chore: ignore or fix linter errors
* chore: wrap up lint errors
* chore: sqlc generation
---------
Co-authored-by: Alexander Belanger <belanger@sas.upenn.edu >
2024-01-21 22:01:20 -05:00
52fde1e704
feat: dag-style execution ( #108 )
...
* feat: dag-style execution
* docs: update to reflect new context
* ensure no cycles
* remove example cycle
* linting
* lint and small fixes
* update deferred rollback
* last rollback handling
* unset max issues
* fix requeue edge case
2024-01-16 11:31:24 -05:00
0ad77d50cb
chore(lint): run golangci-lint --fix ( #105 )
2024-01-16 01:34:07 +07:00
752d5b0ab7
feat: support passing inputs to scheduled workflows ( #104 )
...
* fix: usage of RegisterAction
* make registered actions callable
* chore: update yaml example
* docs: register action documented
* feat: support input to scheduled workflows
* add worker
* add client
* docs: add input to schedule workflow
* chore: generate with updated protoc
* chore: sqlc generate
2024-01-12 00:27:34 -05:00
1f7baacb94
Fix usage of RegisterAction and update docs ( #103 )
...
* fix: usage of RegisterAction
* make registered actions callable
* chore: update yaml example
* docs: register action documented
2024-01-11 18:42:20 -05:00
4d03592aec
chore: telemetry improvements ( #101 )
...
* fix: simple example
* chore: telemetry improvements
- Adds opentelemetry integration for the engine
- Adds standard logger with json and more readable output formats
* remove env from nodemon config files
2024-01-11 13:09:06 -05:00
62445dc37f
feat: support one-time scheduled workflows ( #84 )
...
* feat: support one-time scheduled workflows
* refactor: move schedule out of workflow trigger def
* docs: add scheduling workflows section
* docs: update creating workflow
* only cancel schedules that are in the future
2024-01-08 10:03:32 -05:00
7d1cf5400f
feat: add middleware to worker and services ( #82 )
...
* feat: add middleware support
* docs: add middleware to docs
* chore: remove commented code
* address review comments
2024-01-05 08:35:19 -05:00
65c165caf6
fix: slack example and rendering bug ( #79 )
2024-01-04 08:28:49 -05:00
ff30027ae7
feat: update go-sdk workflow syntax to make it less verbose ( #78 )
...
* feat: update go-sdk workflow syntax to make it less verbose
* docs: make triggers and definitions more clear
2024-01-03 17:14:12 -05:00
76d38d1af9
fix: allow rendering step runs with - in name ( fixes #69 ) ( #74 )
...
* fix: allow rendering step runs with - in name (fixes #69 )
* remove debug lines
* include worker fix
2024-01-03 14:48:18 -05:00
b069347851
fix: check duplicate functions correctly ( #66 )
2024-01-03 13:49:14 -05:00
c7a4c9045a
fix: go-sdk improvements and set default timeouts ( #64 )
2024-01-03 13:13:37 -05:00
49693a5ad6
fix: duplicate workflows showing up in workflow list ( #59 )
...
* fix: duplicate workflows showing up in workflow list
* update formatting
* fix: action map should have proper action id for fn
2024-01-03 09:20:21 -05:00
bf16ba3d30
feat: better support for workflow versions ( #55 )
2024-01-02 16:15:58 -05:00
abelanger5