bugsink/DESIGN-process-model.md

Snappea

I ditched Celery.

Why? It needs a broker. And the broker is "yet another thing to set up". Moving parts are the enemy! There may still be a future scenario where celery is a choice one can make. I've kept the interface as similar to celery as I could, so I can have it swapped back in later if needed.

Considered alternatives:

• redislite w/ celery this seems "risky", you're basically putting an unsupported part in the (already complicated) machine and hope for the best.

• huey with file or sqlite backend? the documentation seems to steer one away from this.

• "inside" gunicorn: this didn't seem straightforward at all. uvicorn/channels may be better ways forward, but again, the docs are steering one away from this.

For now I'm just running in always-eager mode to facilitate running debugserver. This will probably change at some point.

Considered alternatives for the "waking up"

Currently implemented using inotify. Main drawback: tie-in to Linux.

However, the model of "just writing some unique file" is very simple to understand. And the division of labor regarding these files is also simple (the clients just put files there, the snappeaserver just cleans) And it's a nicely decoupled model too: if either server or client is temporarily gone, the messages are not lost/blocking ("postbox model") And it provides a potential path forward if we ever want to put everything (the messages themselves) in those files.

• signals: abandoned, e.g. https://github.com/python/cpython/issues/118143 for the reason

• SIGSTP / SIGCONT: abandoned, I couldn't get this work reliably with "many signals" either. Process hangs mysteriously.

• busy loop with read-on-the-db: still a good candidate. However: the read on the DB is somewhere between .1ms and 1ms. and appears to be CPU-intensive. This means that the sleep() in the loop becomes something to think about and potentially even configure (boooh!). In particular, you get a trade-off between the latency of picking up the work, and the CPU load of the busy loop, which may depend on the fastness of your platform. This is exactly the kind of thinking that I want to take away from myusers (and myself).

• Zero MQ: still a strong candidate. I like that it's not tying us to Linux. And it's still brokerless. However, half way through reading its manual I decided implementing this myself would still be faster than understanding yet another component. Also: TCP is basically implied, which is another point of questions (firewalls etc).

• Python's multiprocessing.connection.Listener ... introduces pickling. Yuck!

• A FIFO where we write bytes: it doesn't seem to be a good fit for more than a single writer.