334ce758ec
feat: queued metrics, along with bug fixes for round robin queueing ( #340 )
2024-04-04 17:45:10 -04:00
28e3a18b82
ci(auto-merge): add auto-merge for dependabot ( #321 )
2024-04-05 00:13:20 +07:00
d6004bbe0c
fix: stale queries in update semaphore ( #333 )
...
* fix: stale queries in update semaphore
* fix: set correct alias for semaphore update
2024-04-04 12:58:32 -04:00
2d61324643
fix: rest client ( #338 )
...
* fix: rest client
* release: py 0.21.1
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-04 09:04:30 -07:00
40dc7712b0
fix: single child element ( #334 )
...
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-03 16:08:13 -07:00
7ab7290eec
feat: UI tweaks ( #332 )
...
* fix: help button treatment
* feat: revise worker docs
* feat: worker empty state
* fix: docs build
* chore: fix linting
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-03 07:35:24 -07:00
d659b08fb4
fix: no semaphore for old workers
2024-04-02 12:47:38 -07:00
2cd61a6dc0
docs: global rate limits ( #330 )
...
* chore: regen protos
* feat: admin put rate limit
* fix: client type
* feat: rate limit example
* feat: workflow config
* feat: step config
* release: py 0.21.0
* feat: rate limit draft
* add behavior description
* chore: cleanup examples
* feat: finalized revision
* chore: linting
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-02 17:48:02 +00:00
d01736c15a
feat(py-sdk): add support for put_workflow extending from base workflow ( #329 )
...
* feat(py-sdk): add put_workflow method and extend from base class
* feat: add cron_input to create workflow API
2024-04-02 13:29:16 -04:00
91b0dda46e
feat(py): global rate limits ( #328 )
...
* chore: regen protos
* feat: admin put rate limit
* fix: client type
* feat: rate limit example
* feat: workflow config
* feat: step config
* release: py 0.21.0
* fix: RateLimitDuration in init
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-02 13:01:22 -04:00
0da379e423
feat(dashboard): improve dashboard settings for user and tokens ( #275 )
...
* feat: expires at as Date
* wip: change passwords
* feat: user can change their password
* chore: cleanup
* chore: linting
* chore: fix build issue
* fix: protoc version
* chore: resolve feedback
* fix: ref
* fix: password inputs
* chore: update generated
* fix: update respository
* feat: only show change password if user has password
* fix: linting
* fix: validation string
* chore: regen api
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-02 12:50:20 -04:00
066b3c5b71
feat(engine): initial rate-limiting engine implementation ( #324 )
...
* feat(engine): initial rate-limiting engine implementation
* fixes and implement go sdk rate limiting
2024-04-02 10:53:03 -04:00
5066547ce6
feat: cancel in progress ( #325 )
...
* chore: bad project path
* fix: remove tickerId for replay
* feat: cancel from request
* feat: cancel button in UI
* chore: rm comment
* fix: build error
* chore: reason case
* chore: reason string
* fix: linting
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-02 01:16:27 +00:00
d731a13afd
docs: streaming ( #323 )
...
* fix: linting
* docs: rename page
* docs: streaming ts/py
* fix: linting
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-04-01 14:52:52 -07:00
d8b6843dec
feat: streaming events ( #309 )
...
* feat: add stream event model
* docs: how to work with db models
* feat: put stream event
* chore: rm comments
* feat: add stream resource type
* feat: enqueue stream event
* fix: contracts
* feat: protos
* chore: set properties correctly for typing
* fix: stream example
* chore: rm old example
* fix: async on
* fix: bytea type
* fix: worker
* feat: put stream data
* feat: stream type
* fix: correct queue
* feat: streaming payloads
* fix: cleanup
* fix: validation
* feat: example file streaming
* chore: rm unused query
* fix: tenant check and read only consumer
* fix: check tenant-steprun relation
* Update prisma/schema.prisma
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* chore: generate protos
* chore: rename migration
* release: 0.20.0
* feat(go-sdk): implement streaming in go
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-04-01 15:46:21 -04:00
7b7fbe3668
fix: update Requeue and Reassign logic to fix performance degradation when many events are queued ( #310 )
...
Logic for requeueing and reassigning did not limit the number of step runs to requeue, so when events accumulate with no worker present it causes memory to spike along with a very high query latency on the database. This commit limits the number of step runs returned in the requeue and reassign queries, and also properly locks step run rows for these queries so only a step run in a PENDING or PENDING_ASSIGNMENT state can be requeued.
It also improves performance of the `AssignStepRunToWorker` query and ensures that `maxRuns` on workers are always respected through the introduction of a `WorkerSemaphore` model. The value gets decremented when a step run is assigned and incremented when a step run is in a final state.
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
* Update controller.go
---------
Co-authored-by: steebchen <contact@luca-steeb.com >
2024-04-01 12:33:18 -04:00
e385e098af
chore(deps): bump github.com/go-co-op/gocron/v2 from 2.1.2 to 2.2.9 ( #319 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-01 17:14:45 +07:00
dada779f6d
ci(github): add dependabot config with actions and gomod ( #306 )
...
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-04-01 09:34:50 +00:00
cdf203dc3e
feat: new listen strategy with worker-side heartbeats ( #308 )
2024-03-31 22:45:10 -04:00
8183dd509a
test(rampup): add load ramp up test ( #273 )
...
* test(rampup): add load ramp up test
* disable debug logging
* actual implementation
* refactor
* max acceptable schedule
* check for non-executed events
* fixes
* chore: set log level to error in engine tests
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-03-31 19:14:30 -04:00
cb15fff91f
chore(docs): fix formatting in child-workflows.mdx ( #307 )
2024-03-30 17:17:41 -04:00
bfa867408f
fix: await gather
2024-03-29 14:27:48 -07:00
c1b1edcfad
Update child-workflows.mdx
2024-03-29 17:14:53 -04:00
be3388f7f0
fix: linting errors on docs
2024-03-29 14:07:39 -07:00
d917ebdda0
docs: child workflows ( #302 )
...
* docs: spawn and join
* lint: prettier
* docs: insert video for launch post
* chore: address feedback
* docs: clean up go example
* Update child-workflows.mdx
2024-03-29 14:07:39 -07:00
d426c9316d
feat(py-sdk): spawn and join ( #299 )
...
* chore: expose full client to context
* chore: update protos
* feat: add parent options to trigger
* feat: example fanout worker
* fix: typehint
* feat: add spawn workflow to context
* wip: polling and streaming
* fix: correct request
* chore: rm logging
* feat: working polling
* feat: functional polling
* chore: separate connection
* feat: functional streaming and listening
* fix: get result payload
* fix: increased initial poll interval
* release (py): 0.19.0
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-03-29 14:07:39 -07:00
77e5d2b77c
feat(go-sdk): spawnWorkflow method and get up to speed with other sdks ( #297 )
...
* feat(go-sdk): spawnWorkflow method and get up to speed with other sdks
* fix: manual trigger example
* fix: linting errors
* fix: double serialization from go sdk
* fix: spawn workflow logic and procedural example
* test(e2e): add procedural test
* fix: panic in e2e test
* fix: e2e test preparation
* fix: api server url in test.yml
* fix: load test server url
* chore: make num children configurable
* address pr review
2024-03-29 14:07:39 -07:00
0daa62974a
feat(frontend): basic list of child workflows ( #293 )
2024-03-29 14:07:39 -07:00
ad9e9aedbf
feat(engine/api): implementation of child/parent workflow runs ( #292 )
2024-03-29 14:07:39 -07:00
70a05f86cf
feat: contracts for spawn/join feature ( #290 )
...
* feat: contracts for spawn/join feature
* add parent step run ids to model
* add parent ids to schedule request
2024-03-29 14:07:39 -07:00
e02a326821
chore(deps): bump github.com/jackc/pgx/v5 from 5.5.0 to 5.5.4 ( #304 )
...
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx ) from 5.5.0 to 5.5.4.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jackc/pgx/compare/v5.5.0...v5.5.4 )
---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-03-29 13:19:08 -04:00
4929483913
chore(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 ( #303 )
...
Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0.
---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-29 13:04:26 -04:00
9c31fdf0f6
fix: assigned step runs not timing out or reassigned ( #301 )
...
* fix: assigned step runs not timing out or reassigned
* fix: regenerate sqlc
2024-03-27 21:03:24 -04:00
1a5e9e07a0
fix: list methods should select distinct step run ids ( #300 )
...
* fix: list methods should select distinct step run ids
* fix: remove unnecessary distinct ons
2024-03-27 20:59:09 -04:00
5f1f59b1e6
fix: onboarding redirect and expired link bugs ( #286 )
...
Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com >
2024-03-25 12:41:28 -04:00
1dccc387b0
feat(py): schedule_workflows ( #287 )
...
* feat: schedule workflows
* docs: delay events
* fix: linting
* release: py 0.18.0
2024-03-22 20:46:20 -04:00
82cb1c0936
chore(docs): add and run prettier ( #284 )
2024-03-22 23:04:19 +07:00
df52bad9e8
fix: timeout zero values ( #283 )
2024-03-21 18:21:07 -04:00
092f54c64f
refactor: separate api and engine repositories, change ticker logic ( #281 )
...
* refactor: separate api and engine repositories, change ticker logic
* fix: nil error blocks
* fix: run migration on load test
* fix: generate db package in load test
* fix: test.yml
* fix: add pnpm to load test
* fix: don't lock CTEs with columns that don't get updated
* fix: update heartbeat for worker every 4 seconds, not 5
* chore: remove dead code
* chore: update python sdk
* chore: add back telemetry attributes
2024-03-21 14:10:34 -04:00
f82cfb4eef
feat(repository): cache engine-relevant methods ( #270 )
2024-03-21 17:09:59 +00:00
617a306b13
docs: fix worker start
2024-03-20 18:04:56 -07:00
65224753c1
fix(go-sdk): support tls strategy of none, with docs ( #269 )
...
* fix(go-sdk): support tls strategy of none, with docs
* chore: errorf -> sprintf in examples
* Apply suggestions from code review
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
* fix: remove time from example
---------
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
2024-03-18 14:02:53 -04:00
d91a8d7bc3
fix: handle inefficient assignments ( #263 )
2024-03-16 17:09:35 +07:00
d9360520de
chore: add better telemetry to database ( #268 )
...
* chore: add better telemetry to database
* fix: span end on query
2024-03-15 15:08:40 -04:00
d577b5f34c
fix(cli): make server config cleanup properly ( #267 )
2024-03-14 17:46:01 +07:00
d7e6e4d8c6
fix: worker locking on requeues ( #265 )
...
* fix: worker locking on requeues
* chore: add alerter to dispatcher
2024-03-13 21:50:02 -04:00
f1d193a32d
fix(docs): adapt home sdk redirect ( #262 )
2024-03-13 22:22:40 +07:00
c66f97c856
fix: deadlocks on workers and tickers ( #241 )
...
* chore: add sentry support to engine
* fix: deadlocks on workers and tickers
* refactor: reduce prisma calls in engine
* trigger
* fix: remove some tenant lookups
* feat: dlx and renamed taskqueue -> msgqueue
* refactor: get group key run logic
* fix: retry counts on messages and concurrency edge cases
* fix: rabbitmq integration tests
* feat: add consumer timeouts
---------
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
2024-03-12 00:45:18 -04:00
a982ff26dd
Update pyproject.toml
2024-03-11 13:21:02 -04:00
5ddc9f80d8
feat(python-sdk): logs api endpoint ( #253 )
2024-03-11 13:20:38 -04:00
abelanger5