1ea4dfc5de
feat: deduplicated enqueue ( #735 )
...
* wip
* wip: functional query
* feat: expose affinity config
* feat: add weight to proto
* feat: upsert affinity state on worker start
* fix: linting
* feat: add upsert proto
* feat: upsert handler
* feat: revise model
* fix: labels
* feat: functional desired worker
* wip: ui
* feat: add state to step run events
* fix: filter empty keys
* fix: labels as badges
* feat: empty state and descriptive text
* chore: add todo
* chore: whitespace
* chore: cleanup
* chore: cleanup
* chore: fix hash
* chore: squash migrations
* fix: fair worker assignment
* fix: remaining slots on valid desired workers
* wip: sticky
* fix: count slots
* chore: rm log line
* feat: expose sticky config
* wip: sticky dag
* feat: expose desired worker id to trigger
* feat: trigger on desired worker
* feat: typescript docs
* feat: sticky python
* feat: py sticky children
* wip: py affinity
* serverless note
* feat: complete python examples
* linting
* feat: deduplicated enqueue
* fix: address changes from PR review
* chore: generate
---------
Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com >
2024-07-26 16:47:46 +00:00
77193df928
feat: expose additional fields on assigned action ( #738 )
2024-07-26 09:17:05 -07:00
ee68786d69
feat: sticky workers ( #695 )
...
* wip
* wip: functional query
* feat: expose affinity config
* feat: add weight to proto
* feat: upsert affinity state on worker start
* fix: linting
* feat: add upsert proto
* feat: upsert handler
* feat: revise model
* fix: labels
* feat: functional desired worker
* wip: ui
* feat: add state to step run events
* fix: filter empty keys
* fix: labels as badges
* feat: empty state and descriptive text
* chore: add todo
* chore: whitespace
* chore: cleanup
* chore: cleanup
* chore: fix hash
* chore: squash migrations
* fix: fair worker assignment
* fix: remaining slots on valid desired workers
* wip: sticky
* fix: count slots
* chore: rm log line
* feat: expose sticky config
* wip: sticky dag
* feat: expose desired worker id to trigger
* feat: trigger on desired worker
* feat: typescript docs
* feat: sticky python
* feat: py sticky children
* wip: py affinity
* serverless note
* feat: complete python examples
* linting
* fix: doc link
* chore: rm debug log
* fix: simplify list labels
* fix: typo
2024-07-22 17:20:23 -04:00
ad29edb44f
fix: partitioned semaphore resolver ( #731 )
...
* fix: partition and improve query
* feat: paginate until done
* chore: address comments
* fix: write partitions
2024-07-18 11:06:25 -04:00
a5724fc4f9
feat: replay workflow runs ( #732 )
...
* feat: replay workflow runs
* address pr review changes
2024-07-18 08:05:15 -07:00
b7cec9ec53
feat: soft delete ( #717 )
...
* feat: soft delete workflows and versions
* feat: filter soft deletes wf and wfr
* feat: filter events and step runs
* fix: query
* fix: query
* chore: generate
* wip
* chore: squash migrations
* chore: separate retention into new service
* feat: regularly clean up
* chore: migrations
* fix: tests
* fix: queries
* fix: ambiguous
* fix: refs
* fix: ambiguous id
* fix: remove update from
* fix: soft delete
* fix: cleanup retention scheduler
* fix: has more query
* chore: gen
* fix: query
* fix: table
2024-07-18 09:06:05 -04:00
5d87f380ef
feat: managed worker pools ( #725 )
...
* change api extension spec to register custom populators
* fix: support only bearer auth
* fix: correct authn logic
* fix: indexes on workflow runs, events
* feat: managed worker pools
* chore: lint fix
* hide workers view when not enabled
* support internal api tokens, minor improvements
* fix: actually write internal
* fix breaking changes
* don't allow revoking internal tokens
* fix: linting and remove metrics view
* fix: token
* address review and add feat flags
2024-07-16 13:33:46 +00:00
c935a38708
fix: revert failed assigned grpc stream and write event ( #726 )
...
* fix: revert and write event
* fix: write critical event
* fix: unassign repository method
* fix: remove unneeded event
2024-07-15 18:05:49 -04:00
d866f8167d
fix: add job run lookup data to data query ( #722 )
2024-07-15 09:17:42 -04:00
f4f752b5d2
fix: large step run payloads ( #719 )
...
* wip: just needs working transformation
* fix: refactor data into separate query
2024-07-14 10:27:25 -04:00
fee4c63319
fix: infinite meta loading loop ( #718 )
...
* fix: infinite meta loading loop
* fix: workflow run id parsing
2024-07-12 14:39:01 -04:00
af705a0aea
fix: resolve unresolved failed steps ( #700 )
...
* feat: resolve unresolved failed steps
* fix: poll active queues
* chore: comment temp unused
2024-07-12 14:14:29 -04:00
8f8f3ad287
fix: reduce max throughput of requeue ( #713 )
...
* fix: reduce max throughput of requeue
* fix: reassign query
* fix: move step run timeout to partition model
* fix: partitioning queries and index
* better logs on requeue
* fix: inactive rebalance and get step run for engine query
* fix: correct inactive queries
2024-07-12 14:03:55 -04:00
a96738539e
fix: simplify reassign query ( #712 )
...
* fix: simplify reassign query
* chore: lint and rehash
2024-07-10 19:38:02 -04:00
1e20bf946a
fix: improved assign, reassign, and requeue ( #702 )
...
* fix: improved queries
* fix: 1s timeout
* fix: indexes
* fix: increase timeout to 4s
* fix: migration
* merge in db changes
* chore: squash migrations
* chore: re-hash
* chore: remove comment
* chore: rm unused query
* fix: state
* fix: check valid workers before commit
* fix: query
* chore: gen
2024-07-10 12:45:08 -04:00
7dc274bbd6
fix: remove expired crons ( #697 )
...
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-07-10 11:01:40 -04:00
f36e66cd28
feat: configurable data retention period ( #693 )
...
* feat: data retention for tenants
* chore: generate and docs
* chore: lint
2024-07-06 14:31:12 +00:00
b96cee1615
chore: better logging in dispatcher ( #692 )
2024-07-04 16:21:26 -04:00
62e02495c9
fix(webhooks): remove unused workflows field ( #665 )
2024-07-02 15:24:23 +01:00
8a8a033af6
feat: variable token expiration ( #670 )
...
* feat: variable token expiration
* fix: typo
* fix: long tokens for webhook workers
* fix: format
2024-07-01 15:47:32 +00:00
f2c6bc1f44
feat: tenant partitioning ( #649 )
...
* feat: tenant partitioning
* fix: rebalance inactive partitions, split into separate partitioner
* fix: shutdown partitioner scheduler properly
* update config options
* fix: config options linting
2024-06-26 21:06:51 +00:00
a51681ddc6
fix(webhooks): use PUT for healthcheck ( #644 )
2024-06-26 10:39:11 -04:00
1490d88954
feat: webhook workers ( #542 )
...
Adds serverless support via the concept of webhook workers. Allows any webhook to be registered as a serverless endpoint for executing a step.
2024-06-25 17:06:43 -04:00
697757879f
feat: billing ( #624 )
...
* feat: init lago client
* feat: billable meter
* feat: db persistence
* wip: expose sub
* feat: rename page
* wip: billing section
* wip: lago integration
* feat: separate plan and period
* wip: webhook
* feat: improve empty state
* feat: update limits on plan changes
* feat: can change plans
* feat: change plan loading state
* feat: yearly filter
* feat: billing clarification
* fix: treatment
* feat: filter plans
* feat: prevent non-owner from changing plan
* fix: loading state
* fix: jit portal link
* fix: rm import
* fix: build errors
* fix: default to free
* fix: wrong files
* fix: select or insert customer
* fix: note
* feat: upgrade dependent on payment method state
* fix: dedupe
* chore: remove github-app from core
* chore: port to cloud
* chore: port to cloud
* chore: port to cloud
* chore: port to cloud
* chore: port to cloud
* add new components, repository callbacks
* chore: rm unused packages
* chore: fix generation
* chore: gen
* fix: cloud api references
* debug
* debug
* fix: actually set plans
* chore: rm debug
* fix: build
* feat: callbacks
* fix: add generated code
* chore: group cloud components
* chore: group by feature
* feat: alert change
* feat: confirm
* fix: confirm modal
* fix: ui
* fix: remove arrears
* fix: open in same tab
* fix: wan alert
* fix: call the callback
* fix: callback obj
* fix: disable if no cloud meta
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-06-25 13:57:16 -04:00
b6dcb4e7e9
refactor(random): refactor random string generation ( #633 )
2024-06-24 23:44:03 +01:00
37c0f6549c
feat: add endpoint to REST API for creating an event, add UI for creating an event ( #616 )
...
* feat: add create event to rest api
* fix: ingestor usage
* feat: add create event to the UI
* fix: address PR review
2024-06-19 19:53:27 +00:00
7c3ddfca32
feat: api server extensions ( #614 )
...
* feat: allow extending the api server
* chore: remove internal packages to pkg
* chore: update db_gen.go
* fix: expose auth
* fix: move logger to pkg
* fix: don't generate gitignore for prisma client
* fix: allow extensions to register their own api spec
* feat: expose pool on server config
* fix: nil pointer exception on empty opts
* fix: run.go file
2024-06-19 09:36:13 -04:00
4328207701
fix: add delay to requeue and reassign to allow workers to reconnect ( #598 )
...
* fix: add delay to requeue and reassign to allow workers to reconnect
* address changes from review
2024-06-17 13:32:50 +00:00
5434332115
fix: clarify that empty strings are not valid returns ( #597 )
2024-06-17 09:21:13 -04:00
31b2fa15cf
fix: don't requeue step runs immediately ( #582 )
2024-06-12 09:49:45 -04:00
6145dbc88d
fix: empty message ( #581 )
2024-06-12 06:27:36 -07:00
bbc4e58dd9
feat: limits ( #559 )
...
* feat: workflow run limits
* fix: resource exhausted 429
* feat: event limit
* feat: worker limit
* fix: sensible error
* fix: pb
* feat: expose limits api
* feat: default limits
* feat: add enable alert option
* feat: slack and email alerts
* fix: cron interval
* feat: make metered util
* wip: schedules and crons
* chore: squash migration
* fix: select or insert
* fix: remove unfinished meter
* chore: atlas migration
* fix: template format
* fix: shared ErrResourceExhausted
* feat: cache
* fix: limit can be nil
* fix: clarification
* fix: close meter ticker
* fix: friendly error for child workflows
2024-06-07 10:57:57 -07:00
e09ee98df5
feat: expose retry count ( #524 )
...
* feat: expose retry count
* feat: expose retry count go
* docs: accessing retry count
* fix: import
* fix: tests
* fix: docs formatting
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2024-06-07 15:27:38 +00:00
8b1417b1d8
feat: worker semaphore v2 ( #540 )
...
* wip: semaphore slot table
* wip: acquire and release semaphore
* Fix deadlock/pool exhaustion
* fix: update skip locked cte
* feat: release resolver
* wip: migration
* chore: migrate running steps to semaphore
* feat: add next alert column
* feat: add notification templates
* feat: add poll token ticker
* fix: expiring 7 days
* fix: no expired tokens
* fix: subject string
* fix: message string
* fix: slack format
* feat: add config columns
* feat: expose api
* feat: update alerter opts
* feat: update ui
* chore: ui tweaks
* fix: ui tweaks
* fix: migration with loop
* chore: generate
* chore: remove unused fn
* feat: add worker id index on slot
* chore: rm unused fn
* fix: typo
* feat: optimized query
* chore: generate all
* chore: collapse migration
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-06-04 07:56:29 -07:00
01512a6493
Feat expiring token email and slack alerts ( #546 )
...
* feat: add next alert column
* feat: add notification templates
* feat: add poll token ticker
* fix: expiring 7 days
* fix: no expired tokens
* fix: subject string
* fix: message string
* fix: slack format
* fix: increase interval
2024-05-31 20:38:11 +00:00
db30bb0b4e
fix: reject heartbeats when dispatcher disconnects, step runs ( #545 )
2024-05-31 08:47:35 -04:00
3ef1f802c7
fix: hangup subscriber when workflow run is finished ( #543 )
...
* fix: better handling for subscribe to workflow run events
* fix: remove debug latency
2024-05-30 21:31:11 -04:00
e94d60538c
fix: old Listen endpoint not setting worker to active ( #534 )
...
* fix: remove active checks from sdk
* fix: add back isActive check
2024-05-28 11:37:47 -04:00
31b2764f17
fix: new context with timeout ( #527 )
2024-05-23 14:03:39 +00:00
3f67323eb3
chore: remove debug line ( #516 )
2024-05-21 18:39:17 -04:00
1b607713e3
fix: worker active bug ( #517 )
...
* fix: only update most recent session
* feat: listener connect visibility
* fix: lint warning
2024-05-21 18:38:57 -04:00
eba6ab7912
fix: worker stream state ( #514 )
...
* feat: add active state to worker
* feat: reassign stuck work
2024-05-21 10:29:14 -04:00
7e75c7c07b
fix: throw proper error when return value of function is not a json object ( #509 )
2024-05-17 18:00:32 -04:00
84c3f722cf
fix: InvalidArgument errors for badly formatted data ( #508 )
2024-05-17 18:00:25 -04:00
0dd38e45f4
feat: make step run replays more intuitive ( #507 )
...
* feat: make step run replays more intuitive
* fix: npmrc file for vercel pnpm version diff
* fix: address changes from PR review
2024-05-17 17:32:15 -04:00
ba391594c3
fix: error text on timeout ( #504 )
...
* fix: error text on timeout
* chore: lint
2024-05-16 12:23:47 -04:00
e6617e1381
feat: refresh timeout ( #495 )
...
* feat: show timeout on step run
* feat: refresh timeout
* fix: increment timeout from last time
* feat: add step run event for timeout refresh
* chore: generate
* feat: add refreshing timeouts docs section
* chore: linting
* feat: go client
* chore: generate
* chore: generate
* fix: test context
* chore: generate
* fix: rm txn
* fix: validator
* chore: lint
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-05-16 12:23:36 -04:00
0ffc3e9664
fix: keep alive enforcement policy ( #499 )
...
* fix: keep alive enforcement policy
* fix: half min
2024-05-15 06:31:04 -07:00
a37ee57cc1
feat: workflow run cancel ( #489 )
...
* feat: workflow run cancel endpoint
* feat: cancel all running button
* feat: select and cancel runs
* chore: error handling
* fix: remove ani
* fix: error toast
* chore: gen
* fix: feedback
* fix: redundant method
* Update frontend/app/src/pages/main/workflow-runs/$run/index.tsx
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* fix: response obj
* fix: cancel by job run
* fix: lint
* chore: generate
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-05-14 17:47:30 -04:00
d1835458d8
chore: update versions of protoc,pnpm ( #497 )
...
* chore: update versions of protoc,pnpm
* chore: bump pnpm
2024-05-14 17:47:01 -04:00
abelanger5