Commit Graph

35 Commits

Author SHA1 Message Date
abelanger5 2c1f1f4808 test: improve Go testing harness (#1631)
* test: improves testing harness for engine

* update CI test

* fix: race condition in test

* make tests more stable

* cleanup pub and sub buffers

* fix: goleak on rampup test

* feat: matrix tests for engine
2025-04-29 10:55:16 -04:00
abelanger5 1f2096313d feat: v1 engine (#1318) 2025-03-11 14:57:13 -04:00
Sean Reilly a8dd33c61f Feature - configurable logging backend (#1188)
* allow us to configure different repos

* make the struct contents public

* pass in config values to new log repo

* rename functions - possibly breaking changes so lets discuss

* make the logging backend configurable

* fix tests

* don't allow calls to WithAdditionalConfig

* cleanup

* replace sc with server

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>

* rename sc to server

* add a LRU cache for the step run lookup

* lets not use an expirable cache and just use the regular one - we cannot close the go func in exirable

---------

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>
2025-01-17 15:34:10 -08:00
abelanger5 d23e5d9963 feat: expression-based concurrency keys (#889)
* feat: expression-based concurrency keys

* fix: build

* fix: typos

* fix: gen

* fix: migration

* fix: remove print statements

* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
abelanger5 263eaf069b feat: pass otel through msgqueue (#802)
* feat: pass otel through msgqueue

* feat: more spans on scheduling

* otel increase batch size
2024-08-28 14:45:02 +00:00
abelanger5 2b9121d295 fix: long token expiry for k8s quickstart (#808)
* debug overwrite

* fix: k8s 100 year expiry
2024-08-23 07:43:02 -04:00
abelanger5 93438ce09d feat: adds a k8s helper for easy k8s installation (#806)
* feat: k8s helper script for generating env

* chore: bump 1.21 -> 1.22

* feat: support both secret, configmap and generate api token

* fix: better errors

* fix: upsert logic

* use default seed tenant for token generation
2024-08-22 21:24:16 +00:00
abelanger5 5d87f380ef feat: managed worker pools (#725)
* change api extension spec to register custom populators

* fix: support only bearer auth

* fix: correct authn logic

* fix: indexes on workflow runs, events

* feat: managed worker pools

* chore: lint fix

* hide workers view when not enabled

* support internal api tokens, minor improvements

* fix: actually write internal

* fix breaking changes

* don't allow revoking internal tokens

* fix: linting and remove metrics view

* fix: token

* address review and add feat flags
2024-07-16 13:33:46 +00:00
Gabe Ruttner 8a8a033af6 feat: variable token expiration (#670)
* feat: variable token expiration

* fix: typo

* fix: long tokens for webhook workers

* fix: format
2024-07-01 15:47:32 +00:00
abelanger5 f2c6bc1f44 feat: tenant partitioning (#649)
* feat: tenant partitioning

* fix: rebalance inactive partitions, split into separate partitioner

* fix: shutdown partitioner scheduler properly

* update config options

* fix: config options linting
2024-06-26 21:06:51 +00:00
Gabe Ruttner a8d42819ea feat: check security service (#639)
* feat: check security service

* feat: propegate version

* feat: with ident

* fix: lint

* chore: generate

* fix: change domain

* fix: panic recover

* fix: migrations

* fix: hash

* fix: dont check in tests
2024-06-26 16:26:29 -04:00
Luca Steeb 1490d88954 feat: webhook workers (#542)
Adds serverless support via the concept of webhook workers. Allows any webhook to be registered as a serverless endpoint for executing a step.
2024-06-25 17:06:43 -04:00
abelanger5 5538196169 fix: correct lengths on random.Generate (#638) 2024-06-25 15:12:59 -04:00
abelanger5 771054a401 fix: make rabbitmq connection optional and disable for token generation (#635) 2024-06-25 15:53:55 +00:00
Luca Steeb b6dcb4e7e9 refactor(random): refactor random string generation (#633) 2024-06-24 23:44:03 +01:00
abelanger5 7c3ddfca32 feat: api server extensions (#614)
* feat: allow extending the api server

* chore: remove internal packages to pkg

* chore: update db_gen.go

* fix: expose auth

* fix: move logger to pkg

* fix: don't generate gitignore for prisma client

* fix: allow extensions to register their own api spec

* feat: expose pool on server config

* fix: nil pointer exception on empty opts

* fix: run.go file
2024-06-19 09:36:13 -04:00
Luca Steeb d1a4d35830 chore(pre-commit): lint whitespace (#494)
Adds a whitespace linter to the pre-commit hook to ensure consistent formatting.
It also enables linting of other SQL files such as for SQLc queries.
2024-05-16 09:17:01 -04:00
abelanger5 7543a0c2a5 add jobs which always run on failure (#445)
* (wip) prisma schema

* feat: on-failure steps

* chore: address changes from PR review

* chore: bump migration number
2024-05-06 15:39:22 -04:00
abelanger5 4ce1dd8632 feat: multi-workflow runs listener on a single endpoint
* new api-contract for workflow run events

* feat: initial implementation for new subscribe listener

* fix: sync issues and send workflow runs immediately

* refactor: add context to all engine db queries, fix deadlocking query

* fix: use new ctx for deleting dispatcher and ticker

* add cancellation reasons

* fix: docs linting

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
2024-04-18 20:55:11 -04:00
abelanger5 092f54c64f refactor: separate api and engine repositories, change ticker logic (#281)
* refactor: separate api and engine repositories, change ticker logic

* fix: nil error blocks

* fix: run migration on load test

* fix: generate db package in load test

* fix: test.yml

* fix: add pnpm to load test

* fix: don't lock CTEs with columns that don't get updated

* fix: update heartbeat for worker every 4 seconds, not 5

* chore: remove dead code

* chore: update python sdk

* chore: add back telemetry attributes
2024-03-21 14:10:34 -04:00
Luca Steeb d577b5f34c fix(cli): make server config cleanup properly (#267) 2024-03-14 17:46:01 +07:00
Luca Steeb ae4841031b feat(engine): standalone tests and engine teardown (#172) 2024-02-28 00:15:25 +07:00
Luca Steeb ab9f8e6c47 fix(cli): log errors via stderr and always exit (#139) 2024-02-01 00:12:22 +07:00
abelanger5 52ba01bf06 chore: qol improvements (#137) 2024-01-30 00:08:52 -05:00
abelanger5 d63b66a837 feat: concurrency groups (#135)
* first pass at moving controllers around

* feat: concurrency limits for strategy CANCEL_IN_PROGRESS

* fix: linting

* chore: bump python sdk version
2024-01-30 00:00:28 -05:00
abelanger5 14a971bfd7 feat: improve versioning to use checksums (#129)
* feat: improve versioning to use checksums

* fix: remove json ignore for scheduled triggers

* update migration with random strings for checksum
2024-01-26 20:47:15 -05:00
abelanger5 78685d0098 feat(security): multiple encryption options, API tokens, easier setup (#125)
* (wip) encryption

* feat: api tokens

* chore: add api token generation command

* fix: e2e tests

* chore: set timeout for e2e job

* fix: e2e tests, remove client-side certs

* chore: address PR review comments

* fix: token tests

* chore: address review comments and fix tests
2024-01-26 15:38:36 -05:00
Luca Steeb c8f87599c7 chore: ignore or fix linter errors (#116)
* chore: ignore or fix linter errors

* chore: wrap up lint errors

* chore: sqlc generation

---------

Co-authored-by: Alexander Belanger <belanger@sas.upenn.edu>
2024-01-21 22:01:20 -05:00
abelanger5 d51a3d34f3 feat: google sso (#112)
* feat: google sso

* chore: update prompt text

* fix: compiling and lint errors

* chore: regenerate sqlc

* chore: address PR comments
2024-01-21 20:06:00 -05:00
Luca Steeb 3071e3c65e refactor: run golangci-lint --fix (#115) 2024-01-21 20:03:12 +07:00
abelanger5 52fde1e704 feat: dag-style execution (#108)
* feat: dag-style execution

* docs: update to reflect new context

* ensure no cycles

* remove example cycle

* linting

* lint and small fixes

* update deferred rollback

* last rollback handling

* unset max issues

* fix requeue edge case
2024-01-16 11:31:24 -05:00
abelanger5 5937b9fd98 feat(go-sdk): add ability to create workflows from the Go SDK more easily, quickstart improvements (#30)
* feat: add initial docs site

* feat: allow workflows to be defined from go sdk

* fix release action

* chore: remove server dependencies from client

* fix: use correct certificate for server

* chore: add port and bind address to grpc config

* fix: add env for grpc config

* fix: nil pointer when output is null

* chore: support variation in output args

* fix unresolve merge conflict

* fix: quickstart improvements

* temp remove database url

* fix: action id not required for event

* fix: actionid validation for events

* Remove deleted files
2024-01-02 09:02:53 -05:00
abelanger5 4ca785ba61 feat: add scheduling timeout (non-configurable) (#24) 2023-12-27 14:40:24 -05:00
Luca Steeb e857004b79 feat(seed): set a default tenant ID (#21) 2023-12-24 14:59:36 +07:00
abelanger5 14ecac8b81 feat: add hatchet-admin command and improve dockerfiles (#18)
* feat: add `hatchet-admin` command and improve dockerfiles

- Also bumps `prisma-client-go` version
2023-12-21 16:38:17 -05:00