Commit Graph

45 Commits

Author SHA1 Message Date
matt 058968c06b Refactor: Attempt II at removing pgtype.UUID everywhere + convert string UUIDs into uuid.UUID (#2894)
* fix: add type override in sqlc.yaml

* chore: gen sqlc

* chore: big find and replace

* chore: more

* fix: clean up bunch of outdated `.Valid` refs

* refactor: remove `sqlchelpers.uuidFromStr()` in favor of `uuid.MustParse()`

* refactor: remove uuidToStr

* fix: lint

* fix: use pointers for null uuids

* chore: clean up more null pointers

* chore: clean up a bunch more

* fix: couple more

* fix: some types on the api

* fix: incorrectly non-null param

* fix: more nullable params

* fix: more refs

* refactor: start replacing tenant id strings with uuids

* refactor: more tenant id uuid casting

* refactor: fix a bunch more

* refactor: more

* refactor: more

* refactor: is that all of them?!

* fix: panic

* fix: rm scans

* fix: unwind some broken things

* chore: tests

* fix: rebase issues

* fix: more tests

* fix: nil checks

* Refactor: Make all UUIDs into `uuid.UUID` (#2897)

* refactor: remove a bunch more string uuids

* refactor: pointers and lists

* refactor: fix all the refs

* refactor: fix a few more

* fix: config loader

* fix: revert some changes

* fix: tests

* fix: test

* chore: proto

* fix: durable listener

* fix: some more string types

* fix: python health worker sleep

* fix: remove a bunch of `MustParse`s from the various gRPC servers

* fix: rm more uuid.MustParse calls

* fix: rm mustparse from api

* fix: test

* fix: merge issues

* fix: handle a bunch more uses of `MustParse` everywhere

* fix: nil id for worker label

* fix: more casting in the oss

* fix: more id parsing

* fix: stringify jwt opt

* fix: couple more bugs in untyped calls

* fix: more types

* fix: broken test

* refactor: implement `GetKeyUuid`

* chore: regen sqlc

* chore: replace pgtype.UUID again

* fix: bunch more type errors

* fix: panic
2026-02-03 11:02:59 -05:00
Mohammed Nafees 6eba6fa91f Billing changes (#2643)
* make changes for billing

* progress around redesign

* meter callback

* modify limits

* upcoming subscription

* fix lint

* fix payment methods

* fix build

* PR comments

* address PR comments

* update cloud contracts

* fix migration name

* fix json serialization error

* loader and fixed for managed compute

* PR comments

* upgrade Go version

* fix migration name

* fix CI

* fix lint CI

* golangci-lint fix

* dedicated subscription
2026-01-19 12:15:11 +01:00
abelanger5 584ba47044 feat: hatchet cli (#2701)
Creates a new CLI for Hatchet! 

- `hatchet profile [add|remove|list|show|update]` - allows managing multiple profiles, like `local` and `cloud`
- `hatchet server [start|stop]` - manages a Hatchet instance locally via docker
- `hatchet quickstart` - generates quickstart directory, similar to `create-react-app`  
- `hatchet worker dev` - creates a filewatcher for a Hatchet instance and automatically injects profile token from a selected profile. Uses a `hatchet.yaml` file in the current directory 
- `hatchet worker run` - runs a script for triggering a workflow using `hatchet.yaml`
- `hatchet tui` - creates the TUI interface
2026-01-13 13:35:43 -05:00
abelanger5 9f463e92d6 refactor: move v1 packages, remove webhook worker references (#2749)
* chore: move v1 packages, remove webhook worker references

* chore: move msgqueue

* fix: relative paths in sqlc.yaml
2026-01-02 11:42:40 -05:00
abelanger5 f82d3bd071 refactor: consolidate repository methods (#2730)
* refactor: remove v0 paths from codebase

* remove uiVersion references

* refactor: remove v0-exclusive database queries

* remove webhook test

* chore: move api token repository

* chore: move dispatcher repository to v1

* chore: move health repository to v1

* chore: remove event repository

* remove some unused repositories

* chore: move mq implementation to v1

* chore: consolidate rate limit implementations

* chore: move security check to v1 repository

* chore: move slack to v1 repository

* chore: move sns implementation to v1 repository

* clean up step repository

* chore: move tenant invite to v1 repository

* chore: move limits, workers, tenant alerts to v1 repository

* chore: move user, tenant, userSession to v1 repository

* chore: move ticker to v1 repository

* chore: move scheduled workflows to v1 repository

* chore: remove workflows

* fix: remove pointer for limits config file

* propagate cache value to api token

* propagate cache durations
2025-12-31 16:35:46 -05:00
matt b65c6de53f Feat: Hatchet Metrics Monitoring, I (#2699)
* Revert "Revert "Feat: Hatchet Metrics Monitoring, I (#2480)" (#2698)"

This reverts commit b87150767a.

* go mod tidy

---------

Co-authored-by: Mohammed Nafees <hello@mnafees.me>
2025-12-23 20:14:14 +01:00
matt b87150767a Revert "Feat: Hatchet Metrics Monitoring, I (#2480)" (#2698)
This reverts commit fdc075ec6f.
2025-12-22 16:26:14 -05:00
matt fdc075ec6f Feat: Hatchet Metrics Monitoring, I (#2480)
* feat: queries + task methods for oldest running task and oldest task

* feat: worker slot and sdk metrics

* feat: wal metrics

* repository stub

* feat: add meter provider thingy

* pg queries

* fix: add task

* feat: repo methods for worker metrics

* feat: active workers query, fix where clauses

* fix: aliasing

* fix: sql, cleanup

* chore: cast

* feat: olap queries

* feat: olap queries

* feat: finish wiring up olap status update metrics

* chore: lint

* chore: lint

* fix: dupes, other code review comments

* send metrics to OTel collector

* last autovac

* flag

* logging updates

* address PR comments

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
Co-authored-by: Mohammed Nafees <hello@mnafees.me>
2025-12-23 01:04:02 +05:30
Gabe Ruttner c8127a43ed chore: add cypres e2e testing to dashboard (#2676)
* feat: remove all v0 paths

* chore: run knip

* chore: knip, allow removals

* chore: rm old components

* fix: gen

* fix: rm redirect

* fix: rm some v0 stuff

* fix: rm more unused stuff

* fix: rm more

* fix: more unused stuff

* chore: gen

* fix: redirect from root

* fix: revert doc changes

* initial migration

* random fix...

* routes

* all routes

* legal

* fixes

* missed route

* remove webhook workers

* reorder links

* rm deadcode

* rm confusing selfhost label

* imo workers are more important and should be first

* feedback

* feedback

* bad merge

* unneeded invariant

* feedback

* lazy load dev tools

* feedback, bundler hints

* feedback

* last bits

* last couple spots

* stale state

* init cypress

* fix lockfile

* pnpm

* doc gen

* add job to existing workflow

* ignore e2e

* seeded sessions

* tests! glorious tests

* fix ci

* lint

* lint

* Update .github/workflows/app.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-18 09:09:33 -08:00
Mohammed Nafees 524c78a7ac fix seed default tenant slug (#2315) 2025-09-18 15:24:10 +02:00
abelanger5 2c1f1f4808 test: improve Go testing harness (#1631)
* test: improves testing harness for engine

* update CI test

* fix: race condition in test

* make tests more stable

* cleanup pub and sub buffers

* fix: goleak on rampup test

* feat: matrix tests for engine
2025-04-29 10:55:16 -04:00
abelanger5 1f2096313d feat: v1 engine (#1318) 2025-03-11 14:57:13 -04:00
Sean Reilly a8dd33c61f Feature - configurable logging backend (#1188)
* allow us to configure different repos

* make the struct contents public

* pass in config values to new log repo

* rename functions - possibly breaking changes so lets discuss

* make the logging backend configurable

* fix tests

* don't allow calls to WithAdditionalConfig

* cleanup

* replace sc with server

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>

* rename sc to server

* add a LRU cache for the step run lookup

* lets not use an expirable cache and just use the regular one - we cannot close the go func in exirable

---------

Co-authored-by: abelanger5 <belanger@sas.upenn.edu>
2025-01-17 15:34:10 -08:00
abelanger5 d23e5d9963 feat: expression-based concurrency keys (#889)
* feat: expression-based concurrency keys

* fix: build

* fix: typos

* fix: gen

* fix: migration

* fix: remove print statements

* fix: reassignment bugs, retries on closed transport, pr review
2024-09-19 10:32:22 -04:00
abelanger5 263eaf069b feat: pass otel through msgqueue (#802)
* feat: pass otel through msgqueue

* feat: more spans on scheduling

* otel increase batch size
2024-08-28 14:45:02 +00:00
abelanger5 2b9121d295 fix: long token expiry for k8s quickstart (#808)
* debug overwrite

* fix: k8s 100 year expiry
2024-08-23 07:43:02 -04:00
abelanger5 93438ce09d feat: adds a k8s helper for easy k8s installation (#806)
* feat: k8s helper script for generating env

* chore: bump 1.21 -> 1.22

* feat: support both secret, configmap and generate api token

* fix: better errors

* fix: upsert logic

* use default seed tenant for token generation
2024-08-22 21:24:16 +00:00
abelanger5 5d87f380ef feat: managed worker pools (#725)
* change api extension spec to register custom populators

* fix: support only bearer auth

* fix: correct authn logic

* fix: indexes on workflow runs, events

* feat: managed worker pools

* chore: lint fix

* hide workers view when not enabled

* support internal api tokens, minor improvements

* fix: actually write internal

* fix breaking changes

* don't allow revoking internal tokens

* fix: linting and remove metrics view

* fix: token

* address review and add feat flags
2024-07-16 13:33:46 +00:00
Gabe Ruttner 8a8a033af6 feat: variable token expiration (#670)
* feat: variable token expiration

* fix: typo

* fix: long tokens for webhook workers

* fix: format
2024-07-01 15:47:32 +00:00
abelanger5 f2c6bc1f44 feat: tenant partitioning (#649)
* feat: tenant partitioning

* fix: rebalance inactive partitions, split into separate partitioner

* fix: shutdown partitioner scheduler properly

* update config options

* fix: config options linting
2024-06-26 21:06:51 +00:00
Gabe Ruttner a8d42819ea feat: check security service (#639)
* feat: check security service

* feat: propegate version

* feat: with ident

* fix: lint

* chore: generate

* fix: change domain

* fix: panic recover

* fix: migrations

* fix: hash

* fix: dont check in tests
2024-06-26 16:26:29 -04:00
Luca Steeb 1490d88954 feat: webhook workers (#542)
Adds serverless support via the concept of webhook workers. Allows any webhook to be registered as a serverless endpoint for executing a step.
2024-06-25 17:06:43 -04:00
abelanger5 5538196169 fix: correct lengths on random.Generate (#638) 2024-06-25 15:12:59 -04:00
abelanger5 771054a401 fix: make rabbitmq connection optional and disable for token generation (#635) 2024-06-25 15:53:55 +00:00
Luca Steeb b6dcb4e7e9 refactor(random): refactor random string generation (#633) 2024-06-24 23:44:03 +01:00
abelanger5 7c3ddfca32 feat: api server extensions (#614)
* feat: allow extending the api server

* chore: remove internal packages to pkg

* chore: update db_gen.go

* fix: expose auth

* fix: move logger to pkg

* fix: don't generate gitignore for prisma client

* fix: allow extensions to register their own api spec

* feat: expose pool on server config

* fix: nil pointer exception on empty opts

* fix: run.go file
2024-06-19 09:36:13 -04:00
Luca Steeb d1a4d35830 chore(pre-commit): lint whitespace (#494)
Adds a whitespace linter to the pre-commit hook to ensure consistent formatting.
It also enables linting of other SQL files such as for SQLc queries.
2024-05-16 09:17:01 -04:00
abelanger5 7543a0c2a5 add jobs which always run on failure (#445)
* (wip) prisma schema

* feat: on-failure steps

* chore: address changes from PR review

* chore: bump migration number
2024-05-06 15:39:22 -04:00
abelanger5 4ce1dd8632 feat: multi-workflow runs listener on a single endpoint
* new api-contract for workflow run events

* feat: initial implementation for new subscribe listener

* fix: sync issues and send workflow runs immediately

* refactor: add context to all engine db queries, fix deadlocking query

* fix: use new ctx for deleting dispatcher and ticker

* add cancellation reasons

* fix: docs linting

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
2024-04-18 20:55:11 -04:00
abelanger5 092f54c64f refactor: separate api and engine repositories, change ticker logic (#281)
* refactor: separate api and engine repositories, change ticker logic

* fix: nil error blocks

* fix: run migration on load test

* fix: generate db package in load test

* fix: test.yml

* fix: add pnpm to load test

* fix: don't lock CTEs with columns that don't get updated

* fix: update heartbeat for worker every 4 seconds, not 5

* chore: remove dead code

* chore: update python sdk

* chore: add back telemetry attributes
2024-03-21 14:10:34 -04:00
Luca Steeb d577b5f34c fix(cli): make server config cleanup properly (#267) 2024-03-14 17:46:01 +07:00
Luca Steeb ae4841031b feat(engine): standalone tests and engine teardown (#172) 2024-02-28 00:15:25 +07:00
Luca Steeb ab9f8e6c47 fix(cli): log errors via stderr and always exit (#139) 2024-02-01 00:12:22 +07:00
abelanger5 52ba01bf06 chore: qol improvements (#137) 2024-01-30 00:08:52 -05:00
abelanger5 d63b66a837 feat: concurrency groups (#135)
* first pass at moving controllers around

* feat: concurrency limits for strategy CANCEL_IN_PROGRESS

* fix: linting

* chore: bump python sdk version
2024-01-30 00:00:28 -05:00
abelanger5 14a971bfd7 feat: improve versioning to use checksums (#129)
* feat: improve versioning to use checksums

* fix: remove json ignore for scheduled triggers

* update migration with random strings for checksum
2024-01-26 20:47:15 -05:00
abelanger5 78685d0098 feat(security): multiple encryption options, API tokens, easier setup (#125)
* (wip) encryption

* feat: api tokens

* chore: add api token generation command

* fix: e2e tests

* chore: set timeout for e2e job

* fix: e2e tests, remove client-side certs

* chore: address PR review comments

* fix: token tests

* chore: address review comments and fix tests
2024-01-26 15:38:36 -05:00
Luca Steeb c8f87599c7 chore: ignore or fix linter errors (#116)
* chore: ignore or fix linter errors

* chore: wrap up lint errors

* chore: sqlc generation

---------

Co-authored-by: Alexander Belanger <belanger@sas.upenn.edu>
2024-01-21 22:01:20 -05:00
abelanger5 d51a3d34f3 feat: google sso (#112)
* feat: google sso

* chore: update prompt text

* fix: compiling and lint errors

* chore: regenerate sqlc

* chore: address PR comments
2024-01-21 20:06:00 -05:00
Luca Steeb 3071e3c65e refactor: run golangci-lint --fix (#115) 2024-01-21 20:03:12 +07:00
abelanger5 52fde1e704 feat: dag-style execution (#108)
* feat: dag-style execution

* docs: update to reflect new context

* ensure no cycles

* remove example cycle

* linting

* lint and small fixes

* update deferred rollback

* last rollback handling

* unset max issues

* fix requeue edge case
2024-01-16 11:31:24 -05:00
abelanger5 5937b9fd98 feat(go-sdk): add ability to create workflows from the Go SDK more easily, quickstart improvements (#30)
* feat: add initial docs site

* feat: allow workflows to be defined from go sdk

* fix release action

* chore: remove server dependencies from client

* fix: use correct certificate for server

* chore: add port and bind address to grpc config

* fix: add env for grpc config

* fix: nil pointer when output is null

* chore: support variation in output args

* fix unresolve merge conflict

* fix: quickstart improvements

* temp remove database url

* fix: action id not required for event

* fix: actionid validation for events

* Remove deleted files
2024-01-02 09:02:53 -05:00
abelanger5 4ca785ba61 feat: add scheduling timeout (non-configurable) (#24) 2023-12-27 14:40:24 -05:00
Luca Steeb e857004b79 feat(seed): set a default tenant ID (#21) 2023-12-24 14:59:36 +07:00
abelanger5 14ecac8b81 feat: add hatchet-admin command and improve dockerfiles (#18)
* feat: add `hatchet-admin` command and improve dockerfiles

- Also bumps `prisma-client-go` version
2023-12-21 16:38:17 -05:00