d8b6843dec
feat: streaming events ( #309 )
...
* feat: add stream event model
* docs: how to work with db models
* feat: put stream event
* chore: rm comments
* feat: add stream resource type
* feat: enqueue stream event
* fix: contracts
* feat: protos
* chore: set properties correctly for typing
* fix: stream example
* chore: rm old example
* fix: async on
* fix: bytea type
* fix: worker
* feat: put stream data
* feat: stream type
* fix: correct queue
* feat: streaming payloads
* fix: cleanup
* fix: validation
* feat: example file streaming
* chore: rm unused query
* fix: tenant check and read only consumer
* fix: check tenant-steprun relation
* Update prisma/schema.prisma
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* chore: generate protos
* chore: rename migration
* release: 0.20.0
* feat(go-sdk): implement streaming in go
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-04-01 15:46:21 -04:00
7b7fbe3668
fix: update Requeue and Reassign logic to fix performance degradation when many events are queued ( #310 )
...
Logic for requeueing and reassigning did not limit the number of step runs to requeue, so when events accumulate with no worker present it causes memory to spike along with a very high query latency on the database. This commit limits the number of step runs returned in the requeue and reassign queries, and also properly locks step run rows for these queries so only a step run in a PENDING or PENDING_ASSIGNMENT state can be requeued.
It also improves performance of the `AssignStepRunToWorker` query and ensures that `maxRuns` on workers are always respected through the introduction of a `WorkerSemaphore` model. The value gets decremented when a step run is assigned and incremented when a step run is in a final state.
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
* Update controller.go
---------
Co-authored-by: steebchen <contact@luca-steeb.com >
2024-04-01 12:33:18 -04:00
e385e098af
chore(deps): bump github.com/go-co-op/gocron/v2 from 2.1.2 to 2.2.9 ( #319 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-01 17:14:45 +07:00
dada779f6d
ci(github): add dependabot config with actions and gomod ( #306 )
...
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-04-01 09:34:50 +00:00
cdf203dc3e
feat: new listen strategy with worker-side heartbeats ( #308 )
2024-03-31 22:45:10 -04:00
8183dd509a
test(rampup): add load ramp up test ( #273 )
...
* test(rampup): add load ramp up test
* disable debug logging
* actual implementation
* refactor
* max acceptable schedule
* check for non-executed events
* fixes
* chore: set log level to error in engine tests
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-03-31 19:14:30 -04:00
cb15fff91f
chore(docs): fix formatting in child-workflows.mdx ( #307 )
2024-03-30 17:17:41 -04:00
bfa867408f
fix: await gather
2024-03-29 14:27:48 -07:00
c1b1edcfad
Update child-workflows.mdx
2024-03-29 17:14:53 -04:00
be3388f7f0
fix: linting errors on docs
2024-03-29 14:07:39 -07:00
d917ebdda0
docs: child workflows ( #302 )
...
* docs: spawn and join
* lint: prettier
* docs: insert video for launch post
* chore: address feedback
* docs: clean up go example
* Update child-workflows.mdx
2024-03-29 14:07:39 -07:00
d426c9316d
feat(py-sdk): spawn and join ( #299 )
...
* chore: expose full client to context
* chore: update protos
* feat: add parent options to trigger
* feat: example fanout worker
* fix: typehint
* feat: add spawn workflow to context
* wip: polling and streaming
* fix: correct request
* chore: rm logging
* feat: working polling
* feat: functional polling
* chore: separate connection
* feat: functional streaming and listening
* fix: get result payload
* fix: increased initial poll interval
* release (py): 0.19.0
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
2024-03-29 14:07:39 -07:00
77e5d2b77c
feat(go-sdk): spawnWorkflow method and get up to speed with other sdks ( #297 )
...
* feat(go-sdk): spawnWorkflow method and get up to speed with other sdks
* fix: manual trigger example
* fix: linting errors
* fix: double serialization from go sdk
* fix: spawn workflow logic and procedural example
* test(e2e): add procedural test
* fix: panic in e2e test
* fix: e2e test preparation
* fix: api server url in test.yml
* fix: load test server url
* chore: make num children configurable
* address pr review
2024-03-29 14:07:39 -07:00
0daa62974a
feat(frontend): basic list of child workflows ( #293 )
2024-03-29 14:07:39 -07:00
ad9e9aedbf
feat(engine/api): implementation of child/parent workflow runs ( #292 )
2024-03-29 14:07:39 -07:00
70a05f86cf
feat: contracts for spawn/join feature ( #290 )
...
* feat: contracts for spawn/join feature
* add parent step run ids to model
* add parent ids to schedule request
2024-03-29 14:07:39 -07:00
e02a326821
chore(deps): bump github.com/jackc/pgx/v5 from 5.5.0 to 5.5.4 ( #304 )
...
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx ) from 5.5.0 to 5.5.4.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md )
- [Commits](https://github.com/jackc/pgx/compare/v5.5.0...v5.5.4 )
---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-03-29 13:19:08 -04:00
4929483913
chore(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 ( #303 )
...
Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0.
---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-29 13:04:26 -04:00
9c31fdf0f6
fix: assigned step runs not timing out or reassigned ( #301 )
...
* fix: assigned step runs not timing out or reassigned
* fix: regenerate sqlc
2024-03-27 21:03:24 -04:00
1a5e9e07a0
fix: list methods should select distinct step run ids ( #300 )
...
* fix: list methods should select distinct step run ids
* fix: remove unnecessary distinct ons
2024-03-27 20:59:09 -04:00
5f1f59b1e6
fix: onboarding redirect and expired link bugs ( #286 )
...
Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com >
2024-03-25 12:41:28 -04:00
1dccc387b0
feat(py): schedule_workflows ( #287 )
...
* feat: schedule workflows
* docs: delay events
* fix: linting
* release: py 0.18.0
2024-03-22 20:46:20 -04:00
82cb1c0936
chore(docs): add and run prettier ( #284 )
2024-03-22 23:04:19 +07:00
df52bad9e8
fix: timeout zero values ( #283 )
2024-03-21 18:21:07 -04:00
092f54c64f
refactor: separate api and engine repositories, change ticker logic ( #281 )
...
* refactor: separate api and engine repositories, change ticker logic
* fix: nil error blocks
* fix: run migration on load test
* fix: generate db package in load test
* fix: test.yml
* fix: add pnpm to load test
* fix: don't lock CTEs with columns that don't get updated
* fix: update heartbeat for worker every 4 seconds, not 5
* chore: remove dead code
* chore: update python sdk
* chore: add back telemetry attributes
2024-03-21 14:10:34 -04:00
f82cfb4eef
feat(repository): cache engine-relevant methods ( #270 )
2024-03-21 17:09:59 +00:00
617a306b13
docs: fix worker start
2024-03-20 18:04:56 -07:00
65224753c1
fix(go-sdk): support tls strategy of none, with docs ( #269 )
...
* fix(go-sdk): support tls strategy of none, with docs
* chore: errorf -> sprintf in examples
* Apply suggestions from code review
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
* fix: remove time from example
---------
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
2024-03-18 14:02:53 -04:00
d91a8d7bc3
fix: handle inefficient assignments ( #263 )
2024-03-16 17:09:35 +07:00
d9360520de
chore: add better telemetry to database ( #268 )
...
* chore: add better telemetry to database
* fix: span end on query
2024-03-15 15:08:40 -04:00
d577b5f34c
fix(cli): make server config cleanup properly ( #267 )
2024-03-14 17:46:01 +07:00
d7e6e4d8c6
fix: worker locking on requeues ( #265 )
...
* fix: worker locking on requeues
* chore: add alerter to dispatcher
2024-03-13 21:50:02 -04:00
f1d193a32d
fix(docs): adapt home sdk redirect ( #262 )
2024-03-13 22:22:40 +07:00
c66f97c856
fix: deadlocks on workers and tickers ( #241 )
...
* chore: add sentry support to engine
* fix: deadlocks on workers and tickers
* refactor: reduce prisma calls in engine
* trigger
* fix: remove some tenant lookups
* feat: dlx and renamed taskqueue -> msgqueue
* refactor: get group key run logic
* fix: retry counts on messages and concurrency edge cases
* fix: rabbitmq integration tests
* feat: add consumer timeouts
---------
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
2024-03-12 00:45:18 -04:00
a982ff26dd
Update pyproject.toml
2024-03-11 13:21:02 -04:00
5ddc9f80d8
feat(python-sdk): logs api endpoint ( #253 )
2024-03-11 13:20:38 -04:00
628180b42c
fix(controller): attempt to fix data race ( #247 )
2024-03-11 16:01:36 +07:00
30fc42174b
chore: fix typo in .gitignore ( #252 )
...
sentitive -> sensitive
2024-03-10 14:21:24 -04:00
5291f3fa7c
Update README.md
2024-03-09 16:18:14 -05:00
0f31758042
Update README.md
2024-03-09 16:16:46 -05:00
6cc8bd1e3a
fix: deadlink
2024-03-08 17:16:25 -08:00
ae00b16cc3
fix: doc dead-link
2024-03-08 09:56:39 -08:00
2d1fa235e6
Update README.md
2024-03-08 12:46:48 -05:00
c2b42508be
fix(typescript-sdk): extract typescript sdk into separate repo ( #239 )
2024-03-08 16:04:38 +07:00
78962596aa
fix(engine): do not panic on regular exit ( #242 )
2024-03-08 14:46:56 +07:00
c7cdc8aa5d
fix(engine/health): listen before serving ( #243 )
2024-03-08 14:46:47 +07:00
f3769a47fa
release: py 0.15.4
2024-03-07 17:02:49 -08:00
49f16ba953
fix: register timeout
2024-03-07 17:02:32 -08:00
38688adce5
feat: sns integrations from frontend ( #249 )
2024-03-07 18:52:08 -05:00
92330002c0
release: py 0.15.3
2024-03-07 14:01:47 -08:00
Gabe Ruttner