Feat: Misc. Python improvements + Streaming Improvements (#1846)

* fix: contextvars explicit copy

* feat: fix a ton of ruff errors

* fix: couple more ruff rules

* fix: ignore unhelpful rule

* fix: exception group in newer Python versions for improved handling

* fix: workflow docs

* feat: context docs

* feat: simple task counter

* feat: config for setting max tasks

* feat: graceful exit once worker exceeds max tasks

* fix: optional

* fix: docs

* fix: events docs + gen

* chore: gen

* fix: one more dangling task

* feat: add xdist in ci

* fix: CI

* fix: xdist fails me once again

* fix: fix + extend some tests

* fix: test cleanup

* fix: exception group

* fix: ugh

* feat: changelog

* Add Ruff linter callout to post

* refactor: clean up runner error handling

* feat: improved errors

* fix: lint

* feat: hacky serde impl

* fix: improve serde + formatting

* fix: logging

* fix: lint

* fix: unexpected errors

* fix: naming, ruff

* fix: rm cruft

* Fix: Attempt to fix namespacing issue in event waits (#1885)

* feat: add xdist in ci

* fix: attempt to fix namespacing issue in event waits

* fix: namespaced worker names

* fix: applied namespace to the wrong thing

* fix: rm hack

* drive by: namespacing improvement

* fix: delay

* fix: changelog

* fix: initial log work

* fix: more logging work

* fix: rm print cruft

* feat: use a queue to send logs

* fix: sentinel value to stop the loop

* fix: use the log sender everywhere

* fix: make streaming blocking, remove more thread pools

* feat: changelog

* fix: linting issues

* fix: broken test

* chore: bunch more generated stuff

* fix: changelog

* fix: one more

* fix: mypy

* chore: gen

* Feat: Streaming Improvements (#1886)

* Fix: Filter list improvements (#1899)

* fix: uuid validation

* fix: improve filter filtering

* fix: inner join

* fix: bug in workflow cached prop

* chore: bump

* fix: lint

* chore: changelog

* fix: separate filter queries

* feat: improve filter filtering

* fix: queries and the like

* feat: add xdist in ci

* feat: streaming test + gen

* feat: add index to stream event

* fix: rm langfuse dep

* fix: lf

* chore: gen

* feat: impl index for stream on context

* feat: tweak protos

* feat: extend test

* feat: send event index through queue

* feat: first pass + debug logging

* debug: fixes

* debug: more possible issues

* feat: generate new stream event protos

* feat: first pass at using an alternate exchange for replaying incoming stream events

* fix: exchange create timing

* fix: rm unused protos

* chore: gen

* feat: python cleanup

* fix: revert rabbit changes

* fix: unwind a bunch of cruft

* fix: optional index

* chore: gen python

* fix: event index nil handling

* feat: improve test

* fix: stream impl in sdk

* fix: make test faster

* chore: gen a ton more stuff

* fix: test

* fix: sorting helper

* fix: bug

* fix: one more ordering bug

* feat: add some tests for buffering logic

* feat: hangup test

* feat: test no buffering if no index sent

* fix: regular mutex

* fix: pr feedback

* fix: conflicts
This commit is contained in:
Matt Kaye
2025-06-25 10:11:01 -04:00
committed by GitHub
parent eb08481483
commit 2f33dd4dbd
157 changed files with 3243 additions and 1369 deletions

View File

@@ -1,5 +1,6 @@
from collections.abc import Mapping
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Mapping
from typing import Any
import requests
from pydantic import BaseModel
@@ -10,13 +11,13 @@ from hatchet_sdk.context.context import Context
from .hatchet_client import hatchet
async def process_image(image_url: str, filters: List[str]) -> Dict[str, Any]:
async def process_image(image_url: str, filters: list[str]) -> dict[str, Any]:
# Do some image processing
return {"url": image_url, "size": 100, "format": "png"}
# > Before (Mergent)
async def process_image_task(request: Any) -> Dict[str, Any]:
async def process_image_task(request: Any) -> dict[str, Any]:
image_url = request.json["image_url"]
filters = request.json["filters"]
try:
@@ -32,12 +33,12 @@ async def process_image_task(request: Any) -> Dict[str, Any]:
# > After (Hatchet)
class ImageProcessInput(BaseModel):
image_url: str
filters: List[str]
filters: list[str]
class ImageProcessOutput(BaseModel):
processed_url: str
metadata: Dict[str, Any]
metadata: dict[str, Any]
@hatchet.task(