mirror/bugsink
1
0
Fork 0
mirror of https://github.com/bugsink/bugsink.git synced 2026-01-05 12:50:17 -06:00
Code Issues Packages Projects Releases Wiki Activity
915 Commits 46 Branches 58 Tags
0.1.13
Commit Graph

52 Commits

Author SHA1 Message Date
Klaas van Schelven
67cfbb58d7 Use 'monofy' package now that it is extracted 2024-09-04 22:54:27 +02:00
Klaas van Schelven
68c556cdca Comment update about WAL & Snappea 2024-08-29 10:23:10 +02:00
Klaas van Schelven
5d6983042a Snappea: workaholic mode 2024-08-28 08:58:35 +02:00
Klaas van Schelven
66bece58c1 snappea: remove a comment that's mostly of historic interest 2024-08-28 08:58:00 +02:00
Klaas van Schelven
46046f894c snappea foreman comment clarifications 2024-08-27 22:17:28 +02:00
Klaas van Schelven
129a8db421 Fix various flake8 errors 2024-08-21 09:31:05 +02:00
Klaas van Schelven
63417d555f Explain why we deal with SIGTERM as we do 
(from memory, in response to the glib remarks in b09cfb21c3, ('as demanded by systemd')
 2024-07-26 16:22:30 +02:00
Klaas van Schelven
d56a8663a7 Remove the periodCounter and the PC registry 
direct consequence of switching to SQL-based counting
 2024-07-16 15:08:05 +02:00
Klaas van Schelven
6767ea593a Fix port number in example nginx conf 2024-07-12 08:39:50 +02:00
Klaas van Schelven
eb23d44962 Enforce a single pc_registry for a single ingesting process 
Using a pid-file that's implied by the ingestion directory.

We do this in `get_pc_registry`, i.e. on the first request. This means failure is
in the first request on the 2nd process.

Why not on startup? Because we don't have a configtest or generic on-startup location
(yet). Making _that_ could be another source of fragility, and getting e.g. the nr
of processes might be non-trivial / config-dependent.
 2024-07-09 13:14:27 +02:00
Klaas van Schelven
c2ec150f52 Cost of connection.close and subsequent reopen documented 2024-07-08 09:53:15 +02:00
Klaas van Schelven
b6cc268333 Remove connection_close 
this was never supposed to have been committed, mistake in c453ca00e5
 2024-07-08 09:41:57 +02:00
Klaas van Schelven
c453ca00e5 Snappea connection_close 2024-07-05 16:28:23 +02:00
Klaas van Schelven
1eb65a7790 Release worker_semaphore when failing to create worker 
exposed when playing around with arbitrary Tasks in a shell; this created
workers I could not run, which would put the foreman in a 'waiting for available threads'
mode.

I briefly looked at the rest of that loop to see whether more exception handling
is necessary, but TBH I don't think we can reasonably recover from e.g. task.delete()
failing (or at least I don't want to think about it now)
 2024-07-05 15:58:15 +02:00
Klaas van Schelven
259069f6e2 Explicit error message for malformed task name 2024-07-05 15:54:55 +02:00
Klaas van Schelven
253380bf2f Foreman: document current understanding of connection.close() 2024-07-05 13:00:08 +02:00
Klaas van Schelven
bf5d221a03 Snappea: fixes on 'atomic' call for Task-getting 
prompted by the work in the previous commit; but somewhat separate from it
 2024-07-04 14:05:04 +02:00
Klaas van Schelven
4daa6c9e09 Close database connections in snappea 2024-07-04 14:04:03 +02:00
Klaas van Schelven
75b620941a Don't (accidentally) load all events into memory on-init 2024-05-23 10:43:28 +02:00
Klaas van Schelven
82d40e3741 Push get_pc_registry into snappea.foreman init 2024-05-23 10:12:24 +02:00
Klaas van Schelven
5af1d2384e Performance logging of Snappea task create/delete 2024-05-22 17:45:28 +02:00
Klaas van Schelven
151af98559 Performance logging: push into development.py (i.e. remove from non-development servers) 2024-05-22 17:06:21 +02:00
Klaas van Schelven
b09cfb21c3 Configure runsnappea to be rebooted every day 
a way to limit memory leaks

also: deal with SIGTERM 'correctly' (i.e. as demanded by systemd)
 2024-05-22 12:44:58 +02:00
Klaas van Schelven
f150c839dc Recommended setup: fix tmpfile troubles in 2 ways 
* recommend to just run in the home dir
* don't use private tmp

The troubles were: when set up using private tmp files, the 2 processes
cannot communicate with each other
 2024-05-22 08:37:52 +02:00
Klaas van Schelven
89dba6e6e5 Fix typo 2024-05-17 15:52:43 +02:00
Klaas van Schelven
5ff2623112 Add checksnappea command 2024-05-17 12:03:40 +02:00
Klaas van Schelven
46220f97ea Snappea: 'ensure' it is running as a singleton 2024-04-27 21:59:20 +02:00
Klaas van Schelven
8d23239526 Be more strict about usage of TransactionTestCase in tests 2024-04-27 20:35:35 +02:00
Klaas van Schelven
2e6be5cbe7 Wrap Task.objects.all() call in a transaction 2024-04-27 11:07:23 +02:00
Klaas van Schelven
52f720bf4f Add missing file 2024-04-24 12:30:25 +02:00
Klaas van Schelven
0855299c97 Robustness of Task -> actual running thread mechanism 2024-04-23 15:28:21 +02:00
Klaas van Schelven
8ca2388c45 Factor out wakeup_server 2024-04-23 15:27:57 +02:00
Klaas van Schelven
1e16f80236 Snappea: minor textual in logging 2024-04-23 13:45:09 +02:00
Klaas van Schelven
ab7b7c467b Snappea: file locations configurable 2024-04-23 13:42:36 +02:00
Klaas van Schelven
9adf971a14 Snappea: settings configurable 2024-04-23 13:32:33 +02:00
Klaas van Schelven
1ef7458619 Snappea: Only do another Task query when the previous result was perhaps limited by the TASK_QS_SIZE 2024-04-23 12:07:03 +02:00
Klaas van Schelven
a7d484f070 Snappea: logging 2024-04-23 11:28:07 +02:00
Klaas van Schelven
08330dd274 Use proper thread-local for wakeup_file uuid part 
Stackoverflow said "In Python, everything is shared, except for function-local
variables (because each function call gets its own set of locals, and threads
are always separate function calls.) "

whatever was meant exactly by that, it's not true for us, because our functions
are called on-import, and the vars-in-closure are shared between threads. i.e.:

>>> from threading import Thread
>>> def closure():
...   l = []
...   def inner():
...     l.append("x")
...     print(len(l))
...   return inner
...
>>> inner = closure()
>>> thread = Thread(target=inner)
>>> thread.start()
1
>>> thread = Thread(target=inner)
>>> thread.start()
2
 2024-04-23 09:42:24 +02:00
Klaas van Schelven
c7d96c362e Bind wakeup_filename to the decorator to avoid flooding the associated dir with wakeup signals 2024-04-22 22:56:39 +02:00
Klaas van Schelven
1fc1da1ceb Implement the registry's 'autodiscover' 
actually: not autodiscover on-bootup, but just try to import the tasks
when you actually need them. This makes it so that we can avoid the whole
of the Django app-loading magical machinery
 2024-04-22 22:37:47 +02:00
Klaas van Schelven
5597e423ea Document roads-not-taken with the wake-up calls 2024-04-22 15:52:34 +02:00
Klaas van Schelven
4fbc283f3e Convert snappea 'wakeup' signal to inotify-based 
see https://github.com/python/cpython/issues/118143
 2024-04-22 15:34:33 +02:00
Klaas van Schelven
46f34370f4 Snappea: Document stuff (and factor a few bits out) 2024-04-19 23:14:19 +02:00
Klaas van Schelven
41a4913299 Implement SNAPPEA_TASK_ALWAYS_EAGER 2024-04-19 21:41:42 +02:00
Klaas van Schelven
cb3b236106 Pea -> Task 2024-04-19 21:29:04 +02:00
Klaas van Schelven
8d2713e09d Logs (slightly) less noisy 2024-04-19 21:24:15 +02:00
Klaas van Schelven
0467e1d316 Snappea: implement graceful shutdown 2024-04-19 21:16:41 +02:00
Klaas van Schelven
3c106521bc snappea WIP commit 2024-04-19 16:21:42 +02:00
Klaas van Schelven
318a26526d Yet another WIP 2024-04-19 14:01:06 +02:00
Klaas van Schelven
14fd545f79 WIP: Example with actual workers and waiting for those 2024-04-19 13:12:01 +02:00