Commit Graph

27 Commits

Author SHA1 Message Date
abelanger5 d23e5d9963 feat: expression-based concurrency keys (#889)
* feat: expression-based concurrency keys

* fix: build

* fix: typos

* fix: gen

* fix: migration

* fix: remove print statements

* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
abelanger5 891514b461 feat: queue v4 (#842)
* wip: v4 of queue

* fix: correct query for updating counts

* tmp: save migration files

* feat: wrap up initial queue

* fix compilation

* fix: reassigns
2024-09-06 16:12:22 -04:00
abelanger5 6317f86793 refactor: consolidate partition logic (#826)
* refactor: consolidate partition logic

* fix: race on scheduler

* fix: move partition uuid to db query

* fix: generate
2024-08-27 15:28:53 -04:00
Gabe Ruttner 4ea4712d4d refactor: performance and throughput (#756)
Refactors the queueing logic to be fairly balanced between actions, with each action backed as a separate FIFO queue. Also adds support for priority queueing and custom queues, though those aren't exposed on the API layer yet. Improves throughput to be > 5000 tasks/second on a single queue. 

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-08-12 14:38:47 +00:00
Gabe Ruttner b7cec9ec53 feat: soft delete (#717)
* feat: soft delete workflows and versions

* feat: filter soft deletes wf and wfr

* feat: filter events and step runs

* fix: query

* fix: query

* chore: generate

* wip

* chore: squash migrations

* chore: separate retention into new service

* feat: regularly clean up

* chore: migrations

* fix: tests

* fix: queries

* fix: ambiguous

* fix: refs

* fix: ambiguous id

* fix: remove update from

* fix: soft delete

* fix: cleanup retention scheduler

* fix: has more query

* chore: gen

* fix: query

* fix: table
2024-07-18 09:06:05 -04:00
Gabe Ruttner f4f752b5d2 fix: large step run payloads (#719)
* wip: just needs working transformation

* fix: refactor data into separate query
2024-07-14 10:27:25 -04:00
Gabe Ruttner af705a0aea fix: resolve unresolved failed steps (#700)
* feat: resolve unresolved failed steps

* fix: poll active queues

* chore: comment temp unused
2024-07-12 14:14:29 -04:00
abelanger5 f36e66cd28 feat: configurable data retention period (#693)
* feat: data retention for tenants

* chore: generate and docs

* chore: lint
2024-07-06 14:31:12 +00:00
abelanger5 f2c6bc1f44 feat: tenant partitioning (#649)
* feat: tenant partitioning

* fix: rebalance inactive partitions, split into separate partitioner

* fix: shutdown partitioner scheduler properly

* update config options

* fix: config options linting
2024-06-26 21:06:51 +00:00
abelanger5 7c3ddfca32 feat: api server extensions (#614)
* feat: allow extending the api server

* chore: remove internal packages to pkg

* chore: update db_gen.go

* fix: expose auth

* fix: move logger to pkg

* fix: don't generate gitignore for prisma client

* fix: allow extensions to register their own api spec

* feat: expose pool on server config

* fix: nil pointer exception on empty opts

* fix: run.go file
2024-06-19 09:36:13 -04:00
Gabe Ruttner a37ee57cc1 feat: workflow run cancel (#489)
* feat: workflow run cancel endpoint

* feat: cancel all running button

* feat: select and cancel runs

* chore: error handling

* fix: remove ani

* fix: error toast

* chore: gen

* fix: feedback

* fix: redundant method

* Update frontend/app/src/pages/main/workflow-runs/$run/index.tsx

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>

* fix: response obj

* fix: cancel by job run

* fix: lint

* chore: generate

---------

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>
Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-05-14 17:47:30 -04:00
abelanger5 68a79fe071 fix: handle nil input more gracefully (#486) 2024-05-13 13:07:41 -04:00
abelanger5 9a2b3648a0 fix: get group key run cancellations should cancel job runs, clean up cancellation logic (#469) 2024-05-08 19:36:26 -04:00
abelanger5 b50ed62924 feat: alerting from slack and email (#461)
* feat: alerting. implements slack alerting, email, and refactors tenant settings to make them more manageable

* chore: generate

* chore: generate sqlc after migrate
2024-05-08 10:04:58 -04:00
abelanger5 7543a0c2a5 add jobs which always run on failure (#445)
* (wip) prisma schema

* feat: on-failure steps

* chore: address changes from PR review

* chore: bump migration number
2024-05-06 15:39:22 -04:00
abelanger5 4ce1dd8632 feat: multi-workflow runs listener on a single endpoint
* new api-contract for workflow run events

* feat: initial implementation for new subscribe listener

* fix: sync issues and send workflow runs immediately

* refactor: add context to all engine db queries, fix deadlocking query

* fix: use new ctx for deleting dispatcher and ticker

* add cancellation reasons

* fix: docs linting

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
2024-04-18 20:55:11 -04:00
Gabe Ruttner eb23e6916d feat: improved lists (#383)
* fix: only close sm

* chore: linting

* feat: filter by workflow run status

* feat: radio filter

* feat: filter run by workflow

* fix: rm unused status

* chore: rm comment

* chore: refactor workflow run table

* fix: visibility state

* feat: visibility and reload

* chore: linting

* chore: shared table component

* chore: linting

* chore: clarify nav

* feat: link to workflow dfn

* chore: share workflow run table

* chore: linting

* wip: parent link

* feat: data table card rendering

* feat: expose manual table control

* feat: data table workflow list

* feat: workflow list actions

* feat: worker actions

* fix: toolbar for any option

* fix: rm search

* fix: rm bad filter func

* chore: linting

* fix: link card title

* fix: rm updated at for now

* fix: gen
2024-04-18 13:35:02 -07:00
abelanger5 334ce758ec feat: queued metrics, along with bug fixes for round robin queueing (#340) 2024-04-04 17:45:10 -04:00
abelanger5 092f54c64f refactor: separate api and engine repositories, change ticker logic (#281)
* refactor: separate api and engine repositories, change ticker logic

* fix: nil error blocks

* fix: run migration on load test

* fix: generate db package in load test

* fix: test.yml

* fix: add pnpm to load test

* fix: don't lock CTEs with columns that don't get updated

* fix: update heartbeat for worker every 4 seconds, not 5

* chore: remove dead code

* chore: update python sdk

* chore: add back telemetry attributes
2024-03-21 14:10:34 -04:00
Luca Steeb d91a8d7bc3 fix: handle inefficient assignments (#263) 2024-03-16 17:09:35 +07:00
abelanger5 c66f97c856 fix: deadlocks on workers and tickers (#241)
* chore: add sentry support to engine

* fix: deadlocks on workers and tickers

* refactor: reduce prisma calls in engine

* trigger

* fix: remove some tenant lookups

* feat: dlx and renamed taskqueue -> msgqueue

* refactor: get group key run logic

* fix: retry counts on messages and concurrency edge cases

* fix: rabbitmq integration tests

* feat: add consumer timeouts

---------

Co-authored-by: Luca Steeb <contact@luca-steeb.com>
2024-03-12 00:45:18 -04:00
abelanger5 5617ab82ce feat: requeueing group key runs (#203) 2024-02-27 23:32:06 -05:00
abelanger5 6ea38a99f2 feat: support maxRuns parameter on workers (#195)
* feat: round robin queueing

* feat: configurable max runs per worker

* fix: address PR review

* docs for max runs and group round robin
2024-02-26 00:48:46 -05:00
abelanger5 2d625fec81 feat: round robin queueing (#194) 2024-02-26 00:16:40 -05:00
abelanger5 82d7995343 feat: manual triggers and give clients a hook into step run events (#141)
* feat: pubsub for clients, more qol stuff

* fix: generate sqlc files

* chore: linting and comments
2024-02-02 12:52:34 -05:00
abelanger5 aed11c3958 feat: workflow visualization and qol improvements (#140)
* feat: workflow visualization and qol improvements

* fix: npm build
2024-02-02 01:35:05 -05:00
abelanger5 d63b66a837 feat: concurrency groups (#135)
* first pass at moving controllers around

* feat: concurrency limits for strategy CANCEL_IN_PROGRESS

* fix: linting

* chore: bump python sdk version
2024-01-30 00:00:28 -05:00