abelanger5
7190078b8d
fix: remove fk constraint on crons ( #1543 )
2025-04-14 09:59:27 -04:00
abelanger5
d4e489996c
fix: v1 edge cases on concurrency, go SDK, parent outputs ( #1497 )
...
* fix: v1 edge cases on concurrency, go SDK, parent outputs
* fix: overflow on queue metrics
* revert changes to DAG
* fix: remove prefix on error for Result method
* cleanup schema, fix migrations
* fix panic edge case
2025-04-07 08:19:13 -04:00
abelanger5
b6d077f96d
feat: show concurrency queue counts in the UI ( #1495 )
...
* feat: show concurrency queue counts in the UI
* fix: parent concurrency queues
2025-04-04 12:22:14 -04:00
abelanger5
1bad2de35f
fix: dag statuses should wait for all tasks to be created ( #1428 )
2025-03-27 15:22:01 -07:00
abelanger5
bc790dbdbb
fix: migration ( #1395 )
2025-03-24 12:52:11 -07:00
abelanger5
c54bf9266c
feat(v1): tenant limits ( #1388 )
...
* feat(v1): tenant limits
* fix: migration
* fix: kill metered cache
2025-03-23 19:03:55 -07:00
abelanger5
00c4bbff09
feat(v1): new gRPC API endpoints ( #1367 )
...
* wip: api contracts
* feat: implement put workflow version endpoint
* add support for match existing data, get scaffolding in place for additional triggers
* create additional matches
* feat: durable sleep, user event matching
* update protos
* fix: working poc of user events, durable sleep
* add migration
* fix: migration column
* feat: durable event listener
* fix: skip overrides
* fix: input -> output
2025-03-23 18:58:20 -07:00
abelanger5
aebcf0bb0c
fix: boundary conditions on 1-second rate limiters ( #1379 )
2025-03-20 21:44:08 +00:00
abelanger5
e91047d7b3
feat: add back tenant alerting to v1 ( #1372 )
2025-03-19 17:50:42 -04:00
abelanger5
4164b80f13
fix: race conditions on retries w/out backoff and concurrency keys ( #1368 )
2025-03-19 12:36:36 -04:00
Gabe Ruttner
3670b94fc4
Feat v1 UI tweaks ( #1344 )
...
* fix: drop uncached loader
* feat: upgrade modal
* add beta
* hacky feature flag
* fix: build
* refetch interval
* 5s
* stop flashing on load
* lint
* fix: map
* fix: last redir
* nil check
* small styling and wording things, change default canUpgrade -> true
* switch link to github discussion
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2025-03-15 09:23:32 -04:00
abelanger5
4cbde4405a
fix: more v1 bug bashing ( #1334 )
2025-03-13 17:13:04 -04:00
abelanger5
d676ef30dc
switches the default msgqueue kind to postgres in hatchet-lite ( #1325 )
2025-03-12 16:30:46 -04:00
abelanger5
ac968e94b8
fix: concurrency issues and a few small improvements ( #1324 )
2025-03-12 16:30:34 -04:00
abelanger5
1f2096313d
feat: v1 engine ( #1318 )
2025-03-11 14:57:13 -04:00
Sean Reilly
a8dd33c61f
Feature - configurable logging backend ( #1188 )
...
* allow us to configure different repos
* make the struct contents public
* pass in config values to new log repo
* rename functions - possibly breaking changes so lets discuss
* make the logging backend configurable
* fix tests
* don't allow calls to WithAdditionalConfig
* cleanup
* replace sc with server
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
* rename sc to server
* add a LRU cache for the step run lookup
* lets not use an expirable cache and just use the regular one - we cannot close the go func in exirable
---------
Co-authored-by: abelanger5 <belanger@sas.upenn.edu >
2025-01-17 15:34:10 -08:00
Sean Reilly
9e961ac196
Feature add version info ( #1154 )
...
* adding a /version endpoint for the engine and a /api/v1/version endpoint for the API
* make the security optional so we don't get redirected for having auth
* lint
* upgrade protoc to the latest available version on brew
* use useQuery and clean up html
2025-01-06 10:50:04 -08:00
abelanger5
a9936ef687
fix: set otel insecure flag for all telemetry instantiations ( #999 )
2024-10-30 17:34:36 -04:00
abelanger5
7ece86dfff
fix: start scheduler if old config is used ( #989 )
2024-10-24 10:52:57 -04:00
abelanger5
2cdee59aea
refactor: optimize v0.50.0 release ( #975 )
...
- Simplifies architecture for splitting engine services into different components. The three supported services are now `grpc-api`, `scheduler`, and `controllers`. The `grpc-api` service is the only one which needs to be exposed for workers. The other two can run as unexposed services.
- Fixes a set of bugs and race conditions in the `v2` scheduler
- Adds a `lastActive` time to the `Queue` table and includes a migration which sets this `lastActive` time for the most recent 24 hours of queues. Effectively this means that the max scheduling time in a queue is 24 hours.
- Rewrites the `ListWorkflowsForEvent` query to improve performance and select far fewer rows.
2024-10-23 12:05:16 +00:00
abelanger5
0ec434d62e
feat: allow insecure option for otel collector address ( #971 )
...
* feat: allow insecure option for otel collector address
* cast to lower
2024-10-16 20:16:22 +00:00
abelanger5
67a96d7166
feat(throughput): single process per queue ( #956 )
...
* feat(throughput): single process per queue
* fix data race
* fix: golint and data race on load test
* wrap up initial v2 scheduler
* fix: more debug logs and tighten channel logic/blocking sends
* improved casing on dispatcher and lease manager
* fix: data race on min id
* increase wait on load test, fix data race
* fix: trylock -> lock
* clean up queue when no longer in set
* fix: clean up cache on exit
* ensure cleanup is only called once
* address review comments
2024-10-15 11:05:19 -04:00
Sean Reilly
27736fa30f
bulk insert buffering ( #913 )
...
Adds bulk inserts to event writes, and adds a generic buffer which can be used by future batch implementations.
2024-10-03 16:26:12 -04:00
abelanger5
bfb11cac51
fix: always use retention on queues, optional data/worker ( #916 )
2024-09-27 14:23:14 -04:00
abelanger5
d23e5d9963
feat: expression-based concurrency keys ( #889 )
...
* feat: expression-based concurrency keys
* fix: build
* fix: typos
* fix: gen
* fix: migration
* fix: remove print statements
* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
abelanger5
b5014f6b3d
chore: more visibility and debug lines for queues ( #836 )
...
* chore: more visibility and debug options for queues
* better debug lines on queue repo
* don't log so much in load test
2024-08-29 14:49:24 -04:00
abelanger5
263eaf069b
feat: pass otel through msgqueue ( #802 )
...
* feat: pass otel through msgqueue
* feat: more spans on scheduling
* otel increase batch size
2024-08-28 14:45:02 +00:00
abelanger5
6317f86793
refactor: consolidate partition logic ( #826 )
...
* refactor: consolidate partition logic
* fix: race on scheduler
* fix: move partition uuid to db query
* fix: generate
2024-08-27 15:28:53 -04:00
abelanger5
2b9121d295
fix: long token expiry for k8s quickstart ( #808 )
...
* debug overwrite
* fix: k8s 100 year expiry
2024-08-23 07:43:02 -04:00
abelanger5
93438ce09d
feat: adds a k8s helper for easy k8s installation ( #806 )
...
* feat: k8s helper script for generating env
* chore: bump 1.21 -> 1.22
* feat: support both secret, configmap and generate api token
* fix: better errors
* fix: upsert logic
* use default seed tenant for token generation
2024-08-22 21:24:16 +00:00
Gabe Ruttner
4ea4712d4d
refactor: performance and throughput ( #756 )
...
Refactors the queueing logic to be fairly balanced between actions, with each action backed as a separate FIFO queue. Also adds support for priority queueing and custom queues, though those aren't exposed on the API layer yet. Improves throughput to be > 5000 tasks/second on a single queue.
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-08-12 14:38:47 +00:00
Gabe Ruttner
b4670af138
Fix qos otel config ( #754 )
...
* feat: otel trace id ratio
* feat: rabbitmq qos
* feat: requeue limit
* fix: tests
2024-07-30 18:11:10 -04:00
Gabe Ruttner
b802f9f45f
feat: stream by addl meta ( #751 )
...
* feat: prop schedule and run
* wip
* fix: filter wfrid
* feat: hangup
* chore: rm debug log
* chore: func name
* fix: cancelled payload
* fix: load
* fix: cleanup the cahce
* fix: single proto
* fix: key -> val
* chore: case
* chore: rm dead code
* chore: rm dead code
* feat: go and docs
* fix: docs
2024-07-29 19:09:51 +00:00
Gabe Ruttner
ad29edb44f
fix: partitioned semaphore resolver ( #731 )
...
* fix: partition and improve query
* feat: paginate until done
* chore: address comments
* fix: write partitions
2024-07-18 11:06:25 -04:00
Gabe Ruttner
b7cec9ec53
feat: soft delete ( #717 )
...
* feat: soft delete workflows and versions
* feat: filter soft deletes wf and wfr
* feat: filter events and step runs
* fix: query
* fix: query
* chore: generate
* wip
* chore: squash migrations
* chore: separate retention into new service
* feat: regularly clean up
* chore: migrations
* fix: tests
* fix: queries
* fix: ambiguous
* fix: refs
* fix: ambiguous id
* fix: remove update from
* fix: soft delete
* fix: cleanup retention scheduler
* fix: has more query
* chore: gen
* fix: query
* fix: table
2024-07-18 09:06:05 -04:00
abelanger5
5d87f380ef
feat: managed worker pools ( #725 )
...
* change api extension spec to register custom populators
* fix: support only bearer auth
* fix: correct authn logic
* fix: indexes on workflow runs, events
* feat: managed worker pools
* chore: lint fix
* hide workers view when not enabled
* support internal api tokens, minor improvements
* fix: actually write internal
* fix breaking changes
* don't allow revoking internal tokens
* fix: linting and remove metrics view
* fix: token
* address review and add feat flags
2024-07-16 13:33:46 +00:00
abelanger5
8f8f3ad287
fix: reduce max throughput of requeue ( #713 )
...
* fix: reduce max throughput of requeue
* fix: reassign query
* fix: move step run timeout to partition model
* fix: partitioning queries and index
* better logs on requeue
* fix: inactive rebalance and get step run for engine query
* fix: correct inactive queries
2024-07-12 14:03:55 -04:00
Gabe Ruttner
8a8a033af6
feat: variable token expiration ( #670 )
...
* feat: variable token expiration
* fix: typo
* fix: long tokens for webhook workers
* fix: format
2024-07-01 15:47:32 +00:00
abelanger5
c2debe62d8
fix: add back deprecated service names and fix webhook worker query ( #660 )
2024-06-27 08:01:02 -04:00
abelanger5
f2c6bc1f44
feat: tenant partitioning ( #649 )
...
* feat: tenant partitioning
* fix: rebalance inactive partitions, split into separate partitioner
* fix: shutdown partitioner scheduler properly
* update config options
* fix: config options linting
2024-06-26 21:06:51 +00:00
Gabe Ruttner
a8d42819ea
feat: check security service ( #639 )
...
* feat: check security service
* feat: propegate version
* feat: with ident
* fix: lint
* chore: generate
* fix: change domain
* fix: panic recover
* fix: migrations
* fix: hash
* fix: dont check in tests
2024-06-26 16:26:29 -04:00
abelanger5
d19e299d1e
refactor: make engine runnable with config instead of loader ( #640 )
...
* refactor: make hatchet-engine runnable programmatically
* feat: export teardown name and fn
2024-06-26 08:14:30 -04:00
Luca Steeb
1490d88954
feat: webhook workers ( #542 )
...
Adds serverless support via the concept of webhook workers. Allows any webhook to be registered as a serverless endpoint for executing a step.
2024-06-25 17:06:43 -04:00
abelanger5
5538196169
fix: correct lengths on random.Generate ( #638 )
2024-06-25 15:12:59 -04:00
Gabe Ruttner
697757879f
feat: billing ( #624 )
...
* feat: init lago client
* feat: billable meter
* feat: db persistence
* wip: expose sub
* feat: rename page
* wip: billing section
* wip: lago integration
* feat: separate plan and period
* wip: webhook
* feat: improve empty state
* feat: update limits on plan changes
* feat: can change plans
* feat: change plan loading state
* feat: yearly filter
* feat: billing clarification
* fix: treatment
* feat: filter plans
* feat: prevent non-owner from changing plan
* fix: loading state
* fix: jit portal link
* fix: rm import
* fix: build errors
* fix: default to free
* fix: wrong files
* fix: select or insert customer
* fix: note
* feat: upgrade dependent on payment method state
* fix: dedupe
* chore: remove github-app from core
* chore: port to cloud
* chore: port to cloud
* chore: port to cloud
* chore: port to cloud
* chore: port to cloud
* add new components, repository callbacks
* chore: rm unused packages
* chore: fix generation
* chore: gen
* fix: cloud api references
* debug
* debug
* fix: actually set plans
* chore: rm debug
* fix: build
* feat: callbacks
* fix: add generated code
* chore: group cloud components
* chore: group by feature
* feat: alert change
* feat: confirm
* fix: confirm modal
* fix: ui
* fix: remove arrears
* fix: open in same tab
* fix: wan alert
* fix: call the callback
* fix: callback obj
* fix: disable if no cloud meta
---------
Co-authored-by: Alexander Belanger <alexander@hatchet.run >
2024-06-25 13:57:16 -04:00
abelanger5
771054a401
fix: make rabbitmq connection optional and disable for token generation ( #635 )
2024-06-25 15:53:55 +00:00
Luca Steeb
b6dcb4e7e9
refactor(random): refactor random string generation ( #633 )
2024-06-24 23:44:03 +01:00
abelanger5
7c3ddfca32
feat: api server extensions ( #614 )
...
* feat: allow extending the api server
* chore: remove internal packages to pkg
* chore: update db_gen.go
* fix: expose auth
* fix: move logger to pkg
* fix: don't generate gitignore for prisma client
* fix: allow extensions to register their own api spec
* feat: expose pool on server config
* fix: nil pointer exception on empty opts
* fix: run.go file
2024-06-19 09:36:13 -04:00
Gabe Ruttner
bbc4e58dd9
feat: limits ( #559 )
...
* feat: workflow run limits
* fix: resource exhausted 429
* feat: event limit
* feat: worker limit
* fix: sensible error
* fix: pb
* feat: expose limits api
* feat: default limits
* feat: add enable alert option
* feat: slack and email alerts
* fix: cron interval
* feat: make metered util
* wip: schedules and crons
* chore: squash migration
* fix: select or insert
* fix: remove unfinished meter
* chore: atlas migration
* fix: template format
* fix: shared ErrResourceExhausted
* feat: cache
* fix: limit can be nil
* fix: clarification
* fix: close meter ticker
* fix: friendly error for child workflows
2024-06-07 10:57:57 -07:00
abelanger5
b0b2e26952
feat: hatchet-lite ( #560 )
...
* feat: hatchet-lite mvp
* fix: init shadow db
* fix: install atlas
* fix: correct env
* fix: wait for db ready
* fix: remove name flag
* fix: add hatchet-lite to build
2024-06-06 14:03:53 -04:00
