4ea4712d4d
refactor: performance and throughput ( #756 )
...
Refactors the queueing logic to be fairly balanced between actions, with each action backed as a separate FIFO queue. Also adds support for priority queueing and custom queues, though those aren't exposed on the API layer yet. Improves throughput to be > 5000 tasks/second on a single queue.
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-08-12 14:38:47 +00:00
b4670af138
Fix qos otel config ( #754 )
...
* feat: otel trace id ratio
* feat: rabbitmq qos
* feat: requeue limit
* fix: tests
2024-07-30 18:11:10 -04:00
b802f9f45f
feat: stream by addl meta ( #751 )
...
* feat: prop schedule and run
* wip
* fix: filter wfrid
* feat: hangup
* chore: rm debug log
* chore: func name
* fix: cancelled payload
* fix: load
* fix: cleanup the cahce
* fix: single proto
* fix: key -> val
* chore: case
* chore: rm dead code
* chore: rm dead code
* feat: go and docs
* fix: docs
2024-07-29 19:09:51 +00:00
ad29edb44f
fix: partitioned semaphore resolver ( #731 )
...
* fix: partition and improve query
* feat: paginate until done
* chore: address comments
* fix: write partitions
2024-07-18 11:06:25 -04:00
b7cec9ec53
feat: soft delete ( #717 )
...
* feat: soft delete workflows and versions
* feat: filter soft deletes wf and wfr
* feat: filter events and step runs
* fix: query
* fix: query
* chore: generate
* wip
* chore: squash migrations
* chore: separate retention into new service
* feat: regularly clean up
* chore: migrations
* fix: tests
* fix: queries
* fix: ambiguous
* fix: refs
* fix: ambiguous id
* fix: remove update from
* fix: soft delete
* fix: cleanup retention scheduler
* fix: has more query
* chore: gen
* fix: query
* fix: table
2024-07-18 09:06:05 -04:00
8f8f3ad287
fix: reduce max throughput of requeue ( #713 )
...
* fix: reduce max throughput of requeue
* fix: reassign query
* fix: move step run timeout to partition model
* fix: partitioning queries and index
* better logs on requeue
* fix: inactive rebalance and get step run for engine query
* fix: correct inactive queries
2024-07-12 14:03:55 -04:00
c2debe62d8
fix: add back deprecated service names and fix webhook worker query ( #660 )
2024-06-27 08:01:02 -04:00
f2c6bc1f44
feat: tenant partitioning ( #649 )
...
* feat: tenant partitioning
* fix: rebalance inactive partitions, split into separate partitioner
* fix: shutdown partitioner scheduler properly
* update config options
* fix: config options linting
2024-06-26 21:06:51 +00:00
a8d42819ea
feat: check security service ( #639 )
...
* feat: check security service
* feat: propegate version
* feat: with ident
* fix: lint
* chore: generate
* fix: change domain
* fix: panic recover
* fix: migrations
* fix: hash
* fix: dont check in tests
2024-06-26 16:26:29 -04:00
d19e299d1e
refactor: make engine runnable with config instead of loader ( #640 )
...
* refactor: make hatchet-engine runnable programmatically
* feat: export teardown name and fn
2024-06-26 08:14:30 -04:00
1490d88954
feat: webhook workers ( #542 )
...
Adds serverless support via the concept of webhook workers. Allows any webhook to be registered as a serverless endpoint for executing a step.
2024-06-25 17:06:43 -04:00
7c3ddfca32
feat: api server extensions ( #614 )
...
* feat: allow extending the api server
* chore: remove internal packages to pkg
* chore: update db_gen.go
* fix: expose auth
* fix: move logger to pkg
* fix: don't generate gitignore for prisma client
* fix: allow extensions to register their own api spec
* feat: expose pool on server config
* fix: nil pointer exception on empty opts
* fix: run.go file
2024-06-19 09:36:13 -04:00
bbc4e58dd9
feat: limits ( #559 )
...
* feat: workflow run limits
* fix: resource exhausted 429
* feat: event limit
* feat: worker limit
* fix: sensible error
* fix: pb
* feat: expose limits api
* feat: default limits
* feat: add enable alert option
* feat: slack and email alerts
* fix: cron interval
* feat: make metered util
* wip: schedules and crons
* chore: squash migration
* fix: select or insert
* fix: remove unfinished meter
* chore: atlas migration
* fix: template format
* fix: shared ErrResourceExhausted
* feat: cache
* fix: limit can be nil
* fix: clarification
* fix: close meter ticker
* fix: friendly error for child workflows
2024-06-07 10:57:57 -07:00
68a79fe071
fix: handle nil input more gracefully ( #486 )
2024-05-13 13:07:41 -04:00
b50ed62924
feat: alerting from slack and email ( #461 )
...
* feat: alerting. implements slack alerting, email, and refactors tenant settings to make them more manageable
* chore: generate
* chore: generate sqlc after migrate
2024-05-08 10:04:58 -04:00
e0d363e796
chore: intercept grpc errors and don't send internal to client ( #370 )
2024-04-10 19:03:18 -04:00
d8b6843dec
feat: streaming events ( #309 )
...
* feat: add stream event model
* docs: how to work with db models
* feat: put stream event
* chore: rm comments
* feat: add stream resource type
* feat: enqueue stream event
* fix: contracts
* feat: protos
* chore: set properties correctly for typing
* fix: stream example
* chore: rm old example
* fix: async on
* fix: bytea type
* fix: worker
* feat: put stream data
* feat: stream type
* fix: correct queue
* feat: streaming payloads
* fix: cleanup
* fix: validation
* feat: example file streaming
* chore: rm unused query
* fix: tenant check and read only consumer
* fix: check tenant-steprun relation
* Update prisma/schema.prisma
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* chore: generate protos
* chore: rename migration
* release: 0.20.0
* feat(go-sdk): implement streaming in go
---------
Co-authored-by: gabriel ruttner <gabe@hatchet.run >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-04-01 15:46:21 -04:00
092f54c64f
refactor: separate api and engine repositories, change ticker logic ( #281 )
...
* refactor: separate api and engine repositories, change ticker logic
* fix: nil error blocks
* fix: run migration on load test
* fix: generate db package in load test
* fix: test.yml
* fix: add pnpm to load test
* fix: don't lock CTEs with columns that don't get updated
* fix: update heartbeat for worker every 4 seconds, not 5
* chore: remove dead code
* chore: update python sdk
* chore: add back telemetry attributes
2024-03-21 14:10:34 -04:00
d9360520de
chore: add better telemetry to database ( #268 )
...
* chore: add better telemetry to database
* fix: span end on query
2024-03-15 15:08:40 -04:00
d7e6e4d8c6
fix: worker locking on requeues ( #265 )
...
* fix: worker locking on requeues
* chore: add alerter to dispatcher
2024-03-13 21:50:02 -04:00
c66f97c856
fix: deadlocks on workers and tickers ( #241 )
...
* chore: add sentry support to engine
* fix: deadlocks on workers and tickers
* refactor: reduce prisma calls in engine
* trigger
* fix: remove some tenant lookups
* feat: dlx and renamed taskqueue -> msgqueue
* refactor: get group key run logic
* fix: retry counts on messages and concurrency edge cases
* fix: rabbitmq integration tests
* feat: add consumer timeouts
---------
Co-authored-by: Luca Steeb <contact@luca-steeb.com >
2024-03-12 00:45:18 -04:00
c7cdc8aa5d
fix(engine/health): listen before serving ( #243 )
2024-03-08 14:46:47 +07:00
105aa08f3f
chore: add sentry support to engine ( #237 )
...
* chore: add sentry support to engine
* chore: address PR comments
2024-03-06 11:50:49 -05:00
f256b258d8
feat: logging from step run executions ( #217 )
...
* fix: job cancellations with shared ctx
* fix: found the bug
* fix: all job runs were getting cancelled
* feat: support logging from executions
* fix: place logging in background
* add back split screen
2024-03-01 17:55:31 -05:00
0d503288ba
fix(engine): cleanup dispatcher and grpc server in parallel ( #207 )
2024-03-01 01:32:24 +07:00
577f432218
feat(engine): readiness & liveness probes ( #197 )
2024-02-28 00:28:11 +07:00
ae4841031b
feat(engine): standalone tests and engine teardown ( #172 )
2024-02-28 00:15:25 +07:00
Gabe Ruttner