Commit Graph

130 Commits

Author SHA1 Message Date
Gabe Ruttner ccfc0c8d60 feat: fe overhaul api changes (#1639)
* api changes

* generate

* pkg

* backmerge main

* revert to main

* revert main
2025-04-30 08:09:34 -07:00
abelanger5 1dcb50c170 hotfix: priority on schedule workflow (#1556)
* hotfix: priority on schedule workflow

* fix: build
2025-04-15 13:10:40 -04:00
abelanger5 c6abd6b9d2 feat: multiple workflow concurrency keys (#1511)
* feat: multiple workflow concurrency keys

* [Python]: Allow multiple workflow-level concurrency keys (#1512)

* chore: generate

* feat: multi concurrency

* chore: version

* feat: example + test

* fix: expand tests

* Feat  ts multiple wf concurrency (#1522)

* feat: multiple concurrency

* release: 1.2.0

* fix: merge

* fix: concurrency defn

* fix: ts multiple concurrency backwards compat (#1531)

* fix

* gen

* chore: lint

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>

---------

Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>

* chore: ver

* chore: gen

* chore: versions

* fix: manually rename migration

* fix: patch ver

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>
Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>
2025-04-14 17:29:17 -04:00
Matt Kaye 80137736af Feat: Priority (#1513)
* feat: initial work wiring up priorities

* fix: add default to default prio in the db

* feat: wire priority through api on wf creation

* feat: extend python test

* feat: priority for scheduled workflows

* feat: wire priority through python api

* feat: more wiring priority through the api

* feat: I think it works?

* feat: e2e test for priority

* it works!

* feat: expand tests for default priorities

* feat: e2e scheduling test

* fix: skip broken test for now

* fix: lint

* feat: add priority columns to cron and schedule ref  tables

* feat: update inserts to include prio

* feat: wire up more apis

* feat: more wiring

* feat: wire up more rest api fields

* chore: cruft

* fix: more wiring

* fix: lint

* chore: gen + wire up priorities

* fix: retries

* fix: try changing fixture scope

* chore: bump version again

* feat: send priority with action payload

* fix: generate script

* Feat  priority ts (#1518)

* feat: initial work wiring up priorities

* fix: add default to default prio in the db

* feat: wire priority through api on wf creation

* feat: extend python test

* feat: priority for scheduled workflows

* feat: wire priority through python api

* feat: more wiring priority through the api

* feat: I think it works?

* feat: e2e test for priority

* it works!

* feat: expand tests for default priorities

* feat: e2e scheduling test

* chore: minor version for priority

* fix: skip broken test for now

* fix: lint

* feat: add priority columns to cron and schedule ref  tables

* feat: update inserts to include prio

* feat: wire up more apis

* feat: more wiring

* feat: wire up more rest api fields

* chore: cruft

* fix: more wiring

* fix: lint

* chore: gen + wire up priorities

* fix: increase timeout

* fix: retries

* fix: try changing fixture scope

* chore: generate

* fix: set schedule priority

* feat: priority

* fix: move priority to wf

* release: 1.2.0

* rm log

* fix: import

* fix: add priority to step

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>

* fix: add dummy runs to priority test to prevent race conditions

* fix: non-breaking field

* fix: gen

* feat: initial pass at docs

* feat: priority in go sdk

* feat: initial work on go example

* fix: doc examples

* fix: proofread

* chore: version

* feat: go sdk

* fix: lint

* fix: declarations and add back RunAsChild

* fix: child workflows

* fix: namespace

* fix: faster child workflows

* fix: sticky

* add back run as child

---------

Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>
Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2025-04-14 16:22:00 -04:00
Matt Kaye 7781200123 Hotfix: Pydantic validation of actionId (#1525)
* fix: pydantic validation of action

* chore: version
2025-04-10 12:25:39 -04:00
Matt Kaye 771b733e10 Feat: Add Parent Task External ID to workflow run details, Fix input type marshalling to null (#1505)
* feat: add parent task external id to api for workflow run

* feat: wire up parent ids

* feat: custom go type for non-null json

* feat: gen

* feat: send action ids back over the api

* fix: lint
2025-04-07 23:40:13 -04:00
abelanger5 b6d077f96d feat: show concurrency queue counts in the UI (#1495)
* feat: show concurrency queue counts in the UI

* fix: parent concurrency queues
2025-04-04 12:22:14 -04:00
Matt Kaye 58d54703b2 Feat: Non-Retryable Exceptions (#1456)
* feat: add boolean flag to proto

* feat: initial wiring up priorities and non-retryables

* fix: query

* fix: cruft comment

* fix: rm priority changes

* feat: python side

* feat: tests for non-retrying workflows

* feat: expand tests

* chore: generate ts

* feat: add name prop to wf

* feat(go-sdk): non retryable error

* feat: start implementing ts

* cleanup: simplify to raising a specific error

* fix: simplify ts

* feat: ts examples

* feat: ver

* feat: docs

* fix: tests + linters

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2025-04-01 15:34:43 -04:00
abelanger5 c54bf9266c feat(v1): tenant limits (#1388)
* feat(v1): tenant limits

* fix: migration

* fix: kill metered cache
2025-03-23 19:03:55 -07:00
abelanger5 00c4bbff09 feat(v1): new gRPC API endpoints (#1367)
* wip: api contracts

* feat: implement put workflow version endpoint

* add support for match existing data, get scaffolding in place for additional triggers

* create additional matches

* feat: durable sleep, user event matching

* update protos

* fix: working poc of user events, durable sleep

* add migration

* fix: migration column

* feat: durable event listener

* fix: skip overrides

* fix: input -> output
2025-03-23 18:58:20 -07:00
abelanger5 1f2096313d feat: v1 engine (#1318) 2025-03-11 14:57:13 -04:00
Gabe Ruttner 158b56c43b feat: add retry count param (#1236) 2025-01-29 07:19:19 -08:00
Gabe Ruttner 0e91542d87 wip: backoff state (#1225)
* wip: backoff state

* fix: retry state and step run start condition

* fix: missing key

* fix: gen

* chore: squash migration

* chore: rm todos

* ops: upgrade proto
2025-01-28 19:16:12 +00:00
Gabe Ruttner 13024c09bd Feat canceled state (#1228)
* feat: add cancel state to event list

* ops: db conns

* feat: add cancelled status to wfr

* feat: mark cancelled workflow runs
2025-01-28 10:31:04 -08:00
Sean Reilly 9e961ac196 Feature add version info (#1154)
* adding a /version endpoint for the engine and a /api/v1/version endpoint for the API

* make the security optional so we don't get redirected for having auth

* lint

* upgrade protoc to the latest available version on brew

* use useQuery and clean up html
2025-01-06 10:50:04 -08:00
abelanger5 e12e700980 feat: CANCEL_NEWEST strategy and make cancel in progress more reliable (#1127) 2024-12-18 01:40:14 +00:00
Sean Reilly cbc2526c0b add a monitoring probe (#1108)
* add a monitoring probe

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-12-17 15:55:50 -05:00
Sean Reilly 279f68c3a4 add pagination for workflows (#1109)
* add some more detail to the list worklflow API endpoint and add pagination and a search function to the frontend that searches the workflow names

* switch

* remove the column filter stuff

* cleanup

* lint and clean up query

* fix linter warning

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-12-12 19:33:43 +00:00
abelanger5 b0c6c7cd46 feat(go-sdk): cron and schedules API, minor fixes (#1083)
* feat(go-sdk): cron and schedules API, minor fixes

* try to improve code block and docs

* revert pre-commit

* fix: generate

* fix: put overflow in right place

* remove branch specs
2024-12-04 21:18:05 +00:00
Gabe Ruttner 23201665a6 Feat: dynamic crons fixes (#1073) 2024-11-26 10:47:30 -05:00
Gabe Ruttner 574eb0b67e feat: dynamic crons (#1000)
* wip: stub schedule page

* wip: stub list

* fix: 2025 bug...

* feat: wip cron list

* feat: addl meta

* feat: expose metadata column

* feat: sort and created at

* cron to recurring

* scheduled: with statuses

* fix: links

* feat: expose schedule ids

* feat: delete run

* fix: remove search

* feat: filterable scheduled

* fix: remove broken features

* chore: lint

* rm metadata for now

* chore: lint

* chore: recurring to cron job

* fix: review comments

* fix: populator

* wip cron changes

* fix: ids are helpful

* fix: populator

* wip

* wip: create crons, stub scheduled

* wip: create schedule

* wip add trigger buttons to all the pages

* wip: reusable trigger form

* fix: hash

* fixes: cron bugs

* fixes: cron sort

* fix: out of order migrations

* fix: add internalRetryCount

* feat: api things survive version transitions

* feat: table things

* feat: delete disabled for non api

* feat: prevent delete non api

* feat: filters

* require cron name for api

* default name

* fix: migrations

* frontend improvements and migrations

* fix: pagination

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-11-21 16:18:24 -05:00
abelanger5 197bdd1f88 feat: exponential backoff (#1062)
* initial migration

* feat: exp backoff, fix linting

* fix utc issue and cleanup
2024-11-21 13:39:02 -05:00
Gabe Ruttner 44addbb47e Feat scheduled improvements (#992)
* wip: stub schedule page

* wip: stub list

* fix: 2025 bug...

* feat: wip cron list

* feat: addl meta

* feat: expose metadata column

* feat: sort and created at

* cron to recurring

* scheduled: with statuses

* fix: links

* feat: expose schedule ids

* feat: delete run

* fix: remove search

* feat: filterable scheduled

* fix: remove broken features

* chore: lint

* rm metadata for now

* chore: lint

* chore: recurring to cron job

* fix: review comments

* fix: populator
2024-11-01 07:16:20 -04:00
Gabe Ruttner 4932e7f863 Feat sdk runtime (#942)
* feat: runtime signature

* feat: add sdk runtime to worker model

* feat: post runtime

* feat: expose sdk version on worker

* feat: go inf

* chore: gen

* chore: migrations and generation

* fix: simpler runtime

* feat: hatchet sdk ver

* fix: rm debug line
2024-10-28 13:47:12 -07:00
Gabe Ruttner 7cd08077d5 feat: improved sdk ack (#931)
* feat: add step run event reasons

* feat: ack

* fix: remove rejected reason

* fix: merge

* fix: correct buffer

* fix: consistent message

* chore: rm todo
2024-10-15 15:52:42 +00:00
Sean Reilly 29721cd1f0 Feat bulk workflows (#940)
Adds support for inserting workflows in bulk via the API and an optional buffered insert on the engine.
2024-10-14 15:35:29 -04:00
abelanger5 b4c861d7a1 patch: release semaphore slots before jobs controller (#927)
* fix: don't need acks on queue checks

* patch: release semaphores early

* proper list on high queue depth

* fix: don't release on started
2024-10-02 11:36:05 -04:00
Gabe Ruttner 7d7e43d4e1 feat: pauseable workflows (#879)
* feat: pause workflow state

* feat: dont run paused workflows

* feat: skipped paused

* implement unpaused behavior for workflow runs

* fix: frontend

* fix: more frontend

* fix: imports

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-09-29 10:58:10 -04:00
abelanger5 925b2654c8 feat: workflow run metrics view (#912)
* feat: add callbacks for workflow run completed

* add tenant id to resolve row

* add finishedBefore, finishedAfter to workflow runs query

* add more callbacks

* feat: tenant ids and loggers in callback

* feat: workflow run metrics frontend

* fix: frontend build
2024-09-27 07:38:15 -04:00
abelanger5 a1a10b4073 feat: dynamic rate limits (#904)
* wip: step run expressions on rate limits

* feat: dynamic rate limits

* chore: v0.47.0

* chore: address changes from PR review

* fix: improved error handling

* address pr review

* better error messages for step run cels, remove debug logs

* fix: hash

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
2024-09-26 22:00:34 +00:00
abelanger5 840e590312 fix: frontend improvements (#905)
* fix: set time range properly on reload

* fix: small button to show queue counts for now
2024-09-24 19:08:41 -04:00
Sean Reilly 5811929928 feat: bulk inserts of events (#887)
* progress commit of bulk inserts

* in_flight: Add changes to metering finish the bulk insert

* remove an attempt to overide enforce limits

* merge in PR fixes

* update docs to add in an additional section in the User guide to describe pushing single events and pushing multiple events

* run lint fix

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-09-23 09:19:39 -07:00
Sean Reilly 15c50f46b5 Partial PR - need to generate SDK - Add endpoint to get the total free worker slots for a worker and the … (#857)
* Add endpoint to get the total free worker slots for a worker and the max runs

* update to use WorkerSempahoreCount instead of checking stepRunId

* modify the query for the new table and change the interface

* bump golangci-lint make changes to name of returned data

* revert the simple example

---------

Co-authored-by: Sean Reilly <sean@hatchet.run>
2024-09-19 10:11:16 -07:00
abelanger5 d23e5d9963 feat: expression-based concurrency keys (#889)
* feat: expression-based concurrency keys

* fix: build

* fix: typos

* fix: gen

* fix: migration

* fix: remove print statements

* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
Gabe Ruttner af9ed49f1e fix: events list view (#878)
* fix: filter by event id

* fix: run count

* feat: filter by id api

* feat: filter by Event Id

* chore: default page is runs

* feat: cancel event runs

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-09-16 16:46:31 +00:00
Gabe Ruttner c64c62f66a feat: improved workflow run details page (#821)
* wip: rip prisma

* wip

* wip

* fix: lint

* wip

* wip

* gen

* wip

* wip

* fix trigger

* hide overview

* revert db changes

* feat: wrap up frontend changes and perf

* chore: generate

* chore: frontend build

* fix: workflow transformer

* fix: avoid race conditions on simultaneous parent completions

* fix: 2025 started

* feat: toast for replay/cancel

* fix: toast

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-09-16 15:39:49 +00:00
Gabe Ruttner 460438ab22 fix: tag (#862) 2024-09-10 09:31:39 -07:00
abelanger5 891514b461 feat: queue v4 (#842)
* wip: v4 of queue

* fix: correct query for updating counts

* tmp: save migration files

* feat: wrap up initial queue

* fix compilation

* fix: reassigns
2024-09-06 16:12:22 -04:00
Gabe Ruttner 526e7ef308 feat: expose priority queue (#814)
* feat: workflow default priority

* feat: write priority on run

* feat: propagate to queue

* chore: squash migrations

* chore: generate
2024-08-26 14:11:28 -04:00
Gabe Ruttner 53be615d5f Enhancement webhook usability (#807)
* feat: secret copier

* feat: improved form

* fix: quotes

* wip: improved flow

* feat: health check logging

* fix: page design

* fix: hard delete, no upsert

* fix: reset modal state

* fix: empty text

* fix: worker state

* fix: update only token

* fix: dont delete name

* fix: logs component

* fix: sort order

* chore: build

* fix: webhook worker cleanup

* chore: squash migrations

* Update api-contracts/openapi/paths/webhook-worker/webhook-worker.yaml

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>

* chore: rename

* fix: wrong query

---------

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>
2024-08-23 10:09:09 -04:00
Gabe Ruttner 9bea55438a Fix webhook healthcheck race (#797)
* fix: race

* fix: partition no rows

* chore: move to workers tab

* feat: redirect empty worker path to all

* chore: add worker type and webhook id

* fix: upsert webhook worker

* fix: update by webhookId

* fix: only stub on create

* feat: url on worker

* chore: migration version

* fix: move

* fix: upsert

* fix: upert

* chore: fix migration

* fix: migrations

* chore: generate
2024-08-21 19:23:24 +00:00
abelanger5 84f7334a06 feat: add windows for metrics and selector (#794)
* feat: add windows for metrics and selector

* better placeholder
2024-08-20 15:29:32 +00:00
abelanger5 dd50515aeb fix: cancelling status on frontend (#779)
* fix: cancelling status on frontend

* fix: slot grid and dedupe on slots
2024-08-13 17:01:26 +00:00
Gabe Ruttner 4ea4712d4d refactor: performance and throughput (#756)
Refactors the queueing logic to be fairly balanced between actions, with each action backed as a separate FIFO queue. Also adds support for priority queueing and custom queues, though those aren't exposed on the API layer yet. Improves throughput to be > 5000 tasks/second on a single queue. 

---------

Co-authored-by: Alexander Belanger <alexander@hatchet.run>
2024-08-12 14:38:47 +00:00
Gabe Ruttner b802f9f45f feat: stream by addl meta (#751)
* feat: prop schedule and run

* wip

* fix: filter wfrid

* feat: hangup

* chore: rm debug log

* chore: func name

* fix: cancelled payload

* fix: load

* fix: cleanup the cahce

* fix: single proto

* fix: key -> val

* chore: case

* chore: rm dead code

* chore: rm dead code

* feat: go and docs

* fix: docs
2024-07-29 19:09:51 +00:00
abelanger5 a245151d91 feat: add workflow kind to workflow versions (#750)
* feat: support workflow kinds

* chore: generate
2024-07-29 12:07:34 -07:00
abelanger5 77193df928 feat: expose additional fields on assigned action (#738) 2024-07-26 09:17:05 -07:00
abelanger5 c0b01f1b9b fix: workflow runs replays and show workflow run input (#744) 2024-07-25 17:35:10 +00:00
Gabe Ruttner ee68786d69 feat: sticky workers (#695)
* wip

* wip: functional query

* feat: expose affinity config

* feat: add weight to proto

* feat: upsert affinity state on worker start

* fix: linting

* feat: add upsert proto

* feat: upsert handler

* feat: revise model

* fix: labels

* feat: functional desired worker

* wip: ui

* feat: add state to step run events

* fix: filter empty keys

* fix: labels as badges

* feat: empty state and descriptive text

* chore: add todo

* chore: whitespace

* chore: cleanup

* chore: cleanup

* chore: fix hash

* chore: squash migrations

* fix: fair worker assignment

* fix: remaining slots on valid desired workers

* wip: sticky

* fix: count slots

* chore: rm log line

* feat: expose sticky config

* wip: sticky dag

* feat: expose desired worker id to trigger

* feat: trigger on desired worker

* feat: typescript docs

* feat: sticky python

* feat: py sticky children

* wip: py affinity

* serverless note

* feat: complete python examples

* linting

* fix: doc link

* chore: rm debug log

* fix: simplify list labels

* fix: typo
2024-07-22 17:20:23 -04:00
abelanger5 a5724fc4f9 feat: replay workflow runs (#732)
* feat: replay workflow runs

* address pr review changes
2024-07-18 08:05:15 -07:00