Commit Graph

193 Commits

Author SHA1 Message Date
Gabe Ruttner ffd0941dc3 fix: list concurrency strategies queries (#2838)
* fixes

* lint
2026-01-21 13:56:01 -08:00
Mohammed Nafees 419819c36c take exclusive lock (#2837) 2026-01-21 20:30:38 +01:00
Gabe Ruttner 622e2d9db9 updates (#2827) 2026-01-21 09:32:21 -08:00
Mohammed Nafees 097d0f15f7 fix naming of migration (#2819) 2026-01-19 13:41:14 +01:00
Mohammed Nafees 6eba6fa91f Billing changes (#2643)
* make changes for billing

* progress around redesign

* meter callback

* modify limits

* upcoming subscription

* fix lint

* fix payment methods

* fix build

* PR comments

* address PR comments

* update cloud contracts

* fix migration name

* fix json serialization error

* loader and fixed for managed compute

* PR comments

* upgrade Go version

* fix migration name

* fix CI

* fix lint CI

* golangci-lint fix

* dedicated subscription
2026-01-19 12:15:11 +01:00
abelanger5 fbeccaad54 fix: typescript post-quickstart (#2809) 2026-01-16 12:18:30 -05:00
abelanger5 c93176152a fix: minor quickstart issues (#2807) 2026-01-16 11:44:20 -05:00
abelanger5 5dd5430f57 feat(cli): update quickstarts with package manager support, e2e tests for quickstarts (#2801)
- Adds options for Python and Typescript package manager support - uv, pip, poetry, npm, pnpm, yarn, and bun. Also adds e2e tests to ensure that all of these quickstarts successfully can start a worker.
- Replaces the scripts with triggers in the CLI
- Adds the hatchet trigger command
2026-01-16 10:08:26 -05:00
abelanger5 c1f8353770 small fixes/improvements to CLI logic (#2793)
* small fixes/improvements to CLI logic

* fix: quickstart should just show error if dir already exists

* add profile tests
2026-01-15 09:36:50 -05:00
abelanger5 fd94823688 docs: cli docs, install script updates (#2784)
* docs: cli docs, install script updates

* lint

* fix cli issues from user interview
2026-01-14 14:37:10 -05:00
abelanger5 584ba47044 feat: hatchet cli (#2701)
Creates a new CLI for Hatchet! 

- `hatchet profile [add|remove|list|show|update]` - allows managing multiple profiles, like `local` and `cloud`
- `hatchet server [start|stop]` - manages a Hatchet instance locally via docker
- `hatchet quickstart` - generates quickstart directory, similar to `create-react-app`  
- `hatchet worker dev` - creates a filewatcher for a Hatchet instance and automatically injects profile token from a selected profile. Uses a `hatchet.yaml` file in the current directory 
- `hatchet worker run` - runs a script for triggering a workflow using `hatchet.yaml`
- `hatchet tui` - creates the TUI interface
2026-01-13 13:35:43 -05:00
abelanger5 6920ec1b61 revert: UI version removal (#2756)
* revert: UI version removal

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-06 14:16:42 -05:00
abelanger5 9f463e92d6 refactor: move v1 packages, remove webhook worker references (#2749)
* chore: move v1 packages, remove webhook worker references

* chore: move msgqueue

* fix: relative paths in sqlc.yaml
2026-01-02 11:42:40 -05:00
abelanger5 f82d3bd071 refactor: consolidate repository methods (#2730)
* refactor: remove v0 paths from codebase

* remove uiVersion references

* refactor: remove v0-exclusive database queries

* remove webhook test

* chore: move api token repository

* chore: move dispatcher repository to v1

* chore: move health repository to v1

* chore: remove event repository

* remove some unused repositories

* chore: move mq implementation to v1

* chore: consolidate rate limit implementations

* chore: move security check to v1 repository

* chore: move slack to v1 repository

* chore: move sns implementation to v1 repository

* clean up step repository

* chore: move tenant invite to v1 repository

* chore: move limits, workers, tenant alerts to v1 repository

* chore: move user, tenant, userSession to v1 repository

* chore: move ticker to v1 repository

* chore: move scheduled workflows to v1 repository

* chore: remove workflows

* fix: remove pointer for limits config file

* propagate cache value to api token

* propagate cache durations
2025-12-31 16:35:46 -05:00
abelanger5 f57ebf7546 refactor: remove v0-exclusive database queries (#2729)
* refactor: remove v0 paths from codebase

* remove uiVersion references

* refactor: remove v0-exclusive database queries

* remove webhook test
2025-12-31 09:36:12 -05:00
abelanger5 dd9c36c315 refactor: remove v0 paths from codebase (#2728)
* refactor: remove v0 paths from codebase

* remove uiVersion references
2025-12-30 09:57:00 -05:00
matt 22a6e6454a fix: rename migration (#2721) 2025-12-26 10:36:18 -07:00
matt 735742c466 Revert "Revert "chore: run list query optimizations (#2670)" (#2708)" (#2720)
This reverts commit 2f301e55cf.
2025-12-26 10:11:02 -07:00
Mohammed Nafees 58758d35b2 Publish COULD_NOT_SEND_TO_WORKER OLAP event due to worker backlog (#2710)
* could not send to worker OLAP event

* fix lint and PR comments

* submodule GHA

* remove submodule

* no gitsubmodule

* fix migration

* revert sdk workflows

* revert sdk workflows

* revert sdk workflows
2025-12-26 09:35:15 -07:00
matt 2f301e55cf Revert "chore: run list query optimizations (#2670)" (#2708)
This reverts commit 87b57febe8.
2025-12-23 17:10:47 -05:00
matt b65c6de53f Feat: Hatchet Metrics Monitoring, I (#2699)
* Revert "Revert "Feat: Hatchet Metrics Monitoring, I (#2480)" (#2698)"

This reverts commit b87150767a.

* go mod tidy

---------

Co-authored-by: Mohammed Nafees <hello@mnafees.me>
2025-12-23 20:14:14 +01:00
Gabe Ruttner 87b57febe8 chore: run list query optimizations (#2670)
* add missing tenant index

* fix span name

* parallelize

* instrument tenant id attribute

* feedback

* cleanup migrations

* rename migration

* correct version

* cleanup
2025-12-23 08:59:13 -08:00
matt bc7f341d13 Fix: Dynamically-sized chunks on payload read (#2700)
* feat: function for pulling chunk size in bytes

* chore: schema

* feat: queries

* chore: rework function

* feat: impl

* fix: ptr

* fix: ptr

* feat: olap side

* fix: handle nulls
2025-12-22 18:28:04 -05:00
matt b87150767a Revert "Feat: Hatchet Metrics Monitoring, I (#2480)" (#2698)
This reverts commit fdc075ec6f.
2025-12-22 16:26:14 -05:00
matt fdc075ec6f Feat: Hatchet Metrics Monitoring, I (#2480)
* feat: queries + task methods for oldest running task and oldest task

* feat: worker slot and sdk metrics

* feat: wal metrics

* repository stub

* feat: add meter provider thingy

* pg queries

* fix: add task

* feat: repo methods for worker metrics

* feat: active workers query, fix where clauses

* fix: aliasing

* fix: sql, cleanup

* chore: cast

* feat: olap queries

* feat: olap queries

* feat: finish wiring up olap status update metrics

* chore: lint

* chore: lint

* fix: dupes, other code review comments

* send metrics to OTel collector

* last autovac

* flag

* logging updates

* address PR comments

---------

Co-authored-by: gabriel ruttner <gabriel.ruttner@gmail.com>
Co-authored-by: Mohammed Nafees <hello@mnafees.me>
2025-12-23 01:04:02 +05:30
matt a4e7584c18 Fix: Last bits of payload job cleanup (#2690)
* feat: offset cleanup queries

* feat: wire it up

* fix: extend lease interval

* feat: new function for diffing rows

* feat: queries

* fix: signature, start wiring up

* feat: wire up diffing logic at the end

* chore: longer lease

* chore: fix type

* fix: couple bugs

* fix: columns

* fix: dupe

* chore: rm unused function

* fix: err

* fix: reduce memory footprint by a bunch
2025-12-22 12:43:08 -05:00
abelanger5 fe6583fc41 fix: rare cases of duplicate writes causing stuck updates (#2681)
* fix: rare cases of duplicate writes causing stuck updates

* update to go 1.25

* fix: add version to sdk-go.yml
2025-12-18 13:05:01 -05:00
Gabe Ruttner c8127a43ed chore: add cypres e2e testing to dashboard (#2676)
* feat: remove all v0 paths

* chore: run knip

* chore: knip, allow removals

* chore: rm old components

* fix: gen

* fix: rm redirect

* fix: rm some v0 stuff

* fix: rm more unused stuff

* fix: rm more

* fix: more unused stuff

* chore: gen

* fix: redirect from root

* fix: revert doc changes

* initial migration

* random fix...

* routes

* all routes

* legal

* fixes

* missed route

* remove webhook workers

* reorder links

* rm deadcode

* rm confusing selfhost label

* imo workers are more important and should be first

* feedback

* feedback

* bad merge

* unneeded invariant

* feedback

* lazy load dev tools

* feedback, bundler hints

* feedback

* last bits

* last couple spots

* stale state

* init cypress

* fix lockfile

* pnpm

* doc gen

* add job to existing workflow

* ignore e2e

* seeded sessions

* tests! glorious tests

* fix ci

* lint

* lint

* Update .github/workflows/app.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-18 09:09:33 -08:00
matt 2226a3eaa4 Chore: Remove payload WAL + corresponding tables 🥳 (#2645)
* chore: nuke the wal and cutover qi tables

* fix: migration name
2025-12-17 11:27:59 -05:00
matt bf849d415c Fix: Payload List Index Performance (#2669)
* feat: update core func

* feat: migration to update function

* fix: pass batch size through

* fix: limit

* feat: materialized cte

* chore: comment
2025-12-16 12:35:29 -05:00
matt 23db2a4fac Fix: Pagination by bounds (#2654)
* fix: pagination missing rows

* fix: separate functions

* fix: return both bounds from query

* fix: wiring

* fix: func

* fix: order col

* fix: bug

* fix: math is hard

* fix: more math

* fix: math and math and math

* fix: slightly more math

* fix: placeholders 🤦

* fix: where clause

* fix: math!

* fix: schema

* refactor: try with `CEIL`

* fix: mathin up a storm

* fix: I was actually a math major in college, who knew

* fix: copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-15 13:07:51 -05:00
matt 0a947924fa Feat: Parallelize replication from PG -> External (#2637)
* feat: chunking query

* feat: first pass at range chunking

* fix: bug bashing

* fix: function geq

* fix: use maps.Copy

* fix: olap func

* feat: olap side

* refactor: external id

* fix: order by

* feat: wire up env vars

* fix: pass var through

* fix: naming

* fix: append to returnErr properly

* fix: use eg.Go
2025-12-10 17:11:03 -05:00
matt 3ff672ebe4 Fix: Don't reset offset if a new process acquires lease (#2628)
* fix: don't reset offset if a new process acquires the lease

* fix: copy paste

* feat: migration, fix queries

* fix: more queries

* fix: down migration

* fix: comment

* feat: finish wiring up everything else

* fix: placeholder initial type

* fix: zero values everywhere

* fix: param ordering

* fix: handle no rows

* fix: zero values

* fix: limit

* fix: simplify

* fix: better defaults
2025-12-09 19:01:51 -05:00
matt 9e14814acb Feat: OLAP Payload Cutover Job (#2618)
* feat: migration

* feat: queries

* feat: overwrite queries

* fix: bug

* feat: first pass

* fix: more olap job wiring

* fix: signature

* fix: refs to a bunch of funcs

* feat: job

* fix: table names

* fix: span name

* chore: lint

* fix: redundant error check

* fix: naming

* fix: handle nil external id

* fix: order payload partitions descending

* fix: param for limiting which partitions get processed

* fix: olap
2025-12-09 12:33:07 -05:00
Mohammed Nafees 5f7b149b5b Add guide for downgrading versions (#2588)
* add downgrading guide and migrate cmd

* cleanup
2025-12-08 12:18:01 +01:00
matt 7e48ac7d02 Fix: Leasing for payload job (#2609)
* refactor: acquire a lease instead of an advisory lock

* refactor: partition dates

* fix: single query to acquire / extend

* fix: explicit alias

* fix: unwind

* fix: hwere clause

* fix: handle no rows

* fix: lease bug

* fix: rm debug

* fix: comment for clarity

* fix: syntax that doesn't actually matter

* fix: error
2025-12-05 13:55:59 -05:00
matt 18940869ae Feat: Job for payload cutovers to external (#2586)
* feat: initial payload cutover job

* refactor: fix a couple things

* feat: start wiring up writes

* feat: only run job if external store is enabled

* fix: add some notes, add loop

* feat: function for reading out payloads

* fix: date handling, logging

* feat: remove wal and immediate offloads

* feat: advisory lock

* feat: partition swap logic

* fix: rm debug

* fix: add todo

* fix: sql cleanup

* fix: sql cleanup, ii

* chore: nuke a bunch of WAL stuff

* chore: more wal

* feat: trigger for crud opts

* feat: drop trigger + function in swapover

* feat: move autovac to later

* feat: use unlogged table initially

* feat: update migration

* fix: drop trigger

* fix: use insert + on conflict

* fix: types

* refactor: clean up a bit

* fix: panic

* fix: detach partition before dropping

* feat: configurable batch size

* feat: offset tracking in the db

* feat: explicitly lock

* fix: down migration

* fix: bug

* fix: offset handling

* fix: try explicit ordering of the insert

* fix: lock location

* fix: do less stuff after locking

* fix: ordering

* fix: dont drop and recreate if temp table exists

* fix: explicitly track completed status

* fix: table name

* fix: dont use unlogged table

* fix: rm todos

* chore: lint

* feat: configurable delay

* fix: use date as pk instead of varchar

* fix: daily job

* fix: hack check constraint to speed up partition attach

* fix: syntax

* fix: syntax

* fix: drop constraint after attaching

* fix: syntax

* fix: drop triggers properly

* fix: factor out insert logic

* refactor: factor out loop logic

* refactor: factor out job preparation work

* fix: ordering

* fix: run the job more often

* fix: use `WithSingletonMode`

* fix: singleton mode sig

* fix: env var cleanup

* fix: overwrite sig

* fix: re-enable immediate offloads with a flag

* fix: order, offload at logic

* feat: add count query to compare

* fix: row-level triggers, partition time bug

* fix: rm todo

* fix: for true

* fix: handle lock not acquired

* fix: handle error

* fix: comment
2025-12-05 10:54:26 -05:00
abelanger5 d071a1c36b fix: prevent large worker gRPC stream backlogs (#2597)
* fix: prevent large worker backlogs

* add config value

* add doc for troubleshooting
2025-12-03 17:15:43 -05:00
Mohammed Nafees 8842a2a9cf Case on conflict for v1_statuses_olap entry (#2528)
* case on conflict for v1_statuses_olap

* fix sql
2025-11-14 17:18:35 +01:00
matt 7fe9806f5d Feat: Configurable OLAP status update size limits (#2499)
* feat: configurable status updates

* fix: config

* fix: wiring

* feat: export limits from olap

* fix: param drilling
2025-11-06 13:37:40 -05:00
Mohammed Nafees 3a4f225788 [hotfix] Temporarily increase TestLoadCLI average threshold (#2461)
* increase TestLoadCLI average threshold

* update thresholds
2025-10-31 17:43:51 +05:30
Mohammed Nafees ed4c0327ce [hotfix] Meaningful casing for engine liveness and readiness probes (#2465)
* more fixes for engine live and ready probes

* rename

* no need to set it to false

* fix casing health check

* log onlt when not shutting down
2025-10-30 20:24:33 +01:00
Mohammed Nafees b58359d7b3 Do not run cleanup on v1_workflow_concurrency_slot (#2463)
* do not run cleanup on v1_concurrency_slot

* fix health endpoints for engine
2025-10-30 15:34:50 +01:00
abelanger5 e1fdeeaf1c fix: payload performance (#2441)
* change some olap flush settings

* increase timeouts for payload wal

* fix: improve performance of payload wal metrics

* slight updates

* more small tweaks

* undo some olap changes, don't offload some payloads

* remove double reads

* try reducing wal poll limit

* analyze v1_dag

* move partition method
2025-10-23 17:45:49 -04:00
matt c6e154fd03 Feat: OLAP Payloads (#2410)
* feat: olap payloads table

* feat: olap queue messages for payload puts

* feat: wire up writes on task write

* driveby: add + ignore psql-connect

* fix: down migration

* fix: use external id for pk

* fix: insert query

* fix: more external ids

* fix: bit more cleanup

* feat: dags

* fix: the rest of the refs

* fix: placeholder uuid

* fix: write external ids

* feat: wire up messages over the queue

* fix: panic

* Revert "fix: panic"

This reverts commit c0adccf2ea.

* Revert "feat: wire up messages over the queue"

This reverts commit 36f425f3c1.

* fix: rm unused method

* fix: rm more

* fix: rm cruft

* feat: wire up failures

* feat: start wiring up completed events

* fix: more wiring

* fix: finish wiring up completed event payloads

* fix: lint

* feat: start wiring up external ids in the core

* feat: olap pub

* fix: add returning

* fix: wiring

* debug: log lines for pubs

* fix: external id writes

* Revert "debug: log lines for pubs"

This reverts commit fe430840bd.

* fix: rm sample

* debug: rm pub buffer param

* Revert "debug: rm pub buffer param"

This reverts commit b42a5cacbb.

* debug: stuck queries

* debug: more logs

* debug: yet more logs

* fix: rename BulkRetrieve -> Retrieve

* chore: lint

* fix: naming

* fix: conn leak in putpayloads

* fix: revert debug

* Revert "debug: more logs"

This reverts commit 95da7de64f.

* Revert "debug: stuck queries"

This reverts commit 8fda64adc4.

* feat: improve getters, olap getter

* fix: key type

* feat: first pass at pulling olap payloads from the payload store

* fix: start fixing bugs

* fix: start reworking `includePayloads` param

* fix: include payloads wiring

* feat: analyze for payloads

* fix: simplify writes more + write event payloads

* feat: read out event payloads

* feat: env vars for dual writes

* refactor: clean up task prop drilling a bit

* feat: add include payloads params to python for tests

* fix: tx commit

* fix: dual writes

* fix: not null constraint

* fix: one more

* debug: logging

* fix: more debugging, tweak function sig

* fix: function sig

* fix: refs

* debug: more logging

* debug: more logging

* debug: fix condition

* debug: overwrite properly

* fix: revert debug

* fix: rm more drilling

* fix: comments

* fix: partitioning jobs

* chore: ver

* fix: bug, docs

* hack: dummy id and inserted at for payload offloads

* fix: bug

* fix: no need to handle offloads for task event data

* hack: jitter + current ts

* fix: short circuit

* fix: offload payloads in a tx

* fix: uncomment sampling

* fix: don't offload if external store is disabled

* chore: gen sqlc

* fix: migration

* fix: start reworking types

* fix: couple more

* fix: rm unused code

* fix: drill includePayloads down again

* fix: silence annoying error in some cases

* fix: always store payloads

* debug: use workflow run id for input

* fix: improve logging

* debug: logging on retrieve

* debug: task input

* fix: use correct field

* debug: write even null payloads to limit errors

* debug: hide error lines

* fix: quieting more errors

* fix: duplicate example names, remove print lines

* debug: add logging for olap event writes

* hack: immediate event offloads and cutovers

* fix: rm log line

* fix: import

* fix: short circuit events

* fix: duped names
2025-10-20 09:09:49 -04:00
Mohammed Nafees 0ef001c5ce Fix for Hatchet Lite to still properly fallback to using the postgres message queue kind (#2392)
* fix for lite postgres msg kind

* support legacy flags as well
2025-10-16 18:54:25 +02:00
Mohammed Nafees e2b1f1353e Fix OTel span attribute naming convention (#2409)
* rename spans according to convention

* low cardinality
2025-10-16 18:43:40 +02:00
abelanger5 b16be655be feat: stateful polling intervals (#2417)
* initial pass on stateful intervals

* pr review comments + add evict expired idempotency keys

* fix: goroutine leak and name vars better

* fix some cleanup logic
2025-10-15 11:40:22 -04:00
Mohammed Nafees a750ce950d Introduce vars to tune ANALYZE job gocron run intervals (#2407)
* introduce cars to tune ANALYZE job gocron run intervals

* update config doc

* fix assignment
2025-10-10 11:02:10 +02:00
abelanger5 d08bf6aac5 fix: optimize concurrency slot trigger method (#2391)
* fix: optimize concurrency slot trigger method

* fix: migration ordering

---------

Co-authored-by: mrkaye97 <mrkaye97@gmail.com>
2025-10-08 11:50:54 -04:00