diff --git a/DESIGN-performance.md b/DESIGN-performance.md index 85f5cf8..52beabc 100644 --- a/DESIGN-performance.md +++ b/DESIGN-performance.md @@ -1,5 +1,14 @@ ## Some thoughts on performance +### Thoughts after implementing eviction + +(See also contents of ./performance/stress-with-eviction/) + +* Surprise 1: 50 events/s (60% more than what I found below) +* Surprise 2: in this config, I _am_ getting snappea backlog. + +### Original thoughts + Now that we have playground.bugsink.com, I could get some real data on that system too. I suppose the most "interesting" finding is that the ~30/s events I can handle seem to be entirely limited by the diff --git a/performance/stress-with-eviction/README.md b/performance/stress-with-eviction/README.md new file mode 100644 index 0000000..517cf3e --- /dev/null +++ b/performance/stress-with-eviction/README.md @@ -0,0 +1,21 @@ +Some results of a small stress test (event-size: 50KiB) d.d. 27 Jun 2024 + +15 projects with `10_000` events maximum each. + +Most interesting pictures: `df` and `snappea_queue_size` + +What we can see: there is a backlog this time. + +C. 8:00 UTC: Start of stress test +C. 9:00 UTC: Per project max event is being hit; disk usage increase slows down. +C. 15:00 UTC: I turned off the stress test; from that point the backlog (and hence the disk usage of undigested events) was reduced. +C. 16:00 UTC: Backlog reduced to 0 + +Post-test (next day) db size is 5518716928 (5.2GiB) + +Open question: what is causing the increase of disk usage between 9:00 and 16:00 (i.e. between max-hit and test-done). Some options: + +* Logs +* DB inefficiencies (no vacuum yet) +* Issues / other items in the DB. + Issues seems unlikely TBH: there is some increase in the number of issues (we have random event types, with a long-tail distribution, so new issues may always arise, but after the test there are only 6611 total issues) diff --git a/performance/stress-with-eviction/df-day.png b/performance/stress-with-eviction/df-day.png new file mode 100644 index 0000000..5cd0c82 Binary files /dev/null and b/performance/stress-with-eviction/df-day.png differ diff --git a/performance/stress-with-eviction/event_ingest_count-day.png b/performance/stress-with-eviction/event_ingest_count-day.png new file mode 100644 index 0000000..a517317 Binary files /dev/null and b/performance/stress-with-eviction/event_ingest_count-day.png differ diff --git a/performance/stress-with-eviction/response_time_api_avg-day.png b/performance/stress-with-eviction/response_time_api_avg-day.png new file mode 100644 index 0000000..5ff1983 Binary files /dev/null and b/performance/stress-with-eviction/response_time_api_avg-day.png differ diff --git a/performance/stress-with-eviction/response_time_api_max-day.png b/performance/stress-with-eviction/response_time_api_max-day.png new file mode 100644 index 0000000..5a24bf3 Binary files /dev/null and b/performance/stress-with-eviction/response_time_api_max-day.png differ diff --git a/performance/stress-with-eviction/snappea_queue_size-day.png b/performance/stress-with-eviction/snappea_queue_size-day.png new file mode 100644 index 0000000..a28ab66 Binary files /dev/null and b/performance/stress-with-eviction/snappea_queue_size-day.png differ diff --git a/performance/stress-with-eviction/total_requests-day.png b/performance/stress-with-eviction/total_requests-day.png new file mode 100644 index 0000000..36b8008 Binary files /dev/null and b/performance/stress-with-eviction/total_requests-day.png differ