53049d5779
Shard batch writes, use batch UDP syscalls
...
The idea is to drain the socket and do a single RocksDB WAL
write/fsync for all the write requests we have found.
The read requests are immediately executed. The reasoning here is
that currently write requests are _a lot_ slower than the read
requests because fsyncing takes ~500us on fsf1. In the future this
might change.
Since we're at it, we also use batch UDP syscalls in the CDC.
Fixes #119 .
2023-12-07 14:29:07 +00:00
3eae5bbf9b
Use an EMA for the in-flight CDC txns as well
2023-12-07 10:27:32 +00:00
38f3d54ecd
Wait forever, rather than having timeouts
...
The goal here is to not have constant wakeups due to timeout. Do
not attempt to clean things up nicely before termination -- just
terminate instead. We can setup a proper termination system in
the future, I first want to see if this makes a difference.
Also, change xmon to use pipes for communication, so that it can
wait without timers as well.
Also, `write` directly for logging, so that we know the logs will
make it to the file after the logging call returns (since we now
do not have the chance to flush them afterwards).
2023-12-07 10:11:19 +00:00
af46ab2173
Bump CDC shard response timeout
2023-11-29 15:00:08 +00:00
a52efe217b
Tune CDC logging more
2023-11-29 14:40:33 +00:00
e4c01e8728
Metrics + logging
2023-11-29 14:32:37 +00:00
bd278ff6f6
Better metrics for shard responses in CDC
2023-11-29 13:52:44 +00:00
4453083aa7
Correctly record request id when picking up transactions after restart
2023-11-29 11:08:07 +00:00
a367858684
Drop entire CF at once, rather than one-by-one
...
A dry run of the production upgrade using a backup revealed that
dropping them one-by-one would take ages, since before we kept every
single CDC request.
2023-11-29 11:08:07 +00:00
7537bbc6cf
Remove useless line
2023-11-29 11:08:07 +00:00
fac014a864
Self-PR review, part 2
2023-11-29 11:08:07 +00:00
ba9424e224
Remove unordered_set
...
Almost certainly irrelevant, but it was bugging me
2023-11-29 11:08:07 +00:00
2eab012d76
Fix bug in poll check code
2023-11-29 11:08:07 +00:00
c94ece50cf
Integer sanitizer stuff
2023-11-29 11:08:07 +00:00
59abb24a8e
Add ceiling on max update size
...
We don't want it to grow without bound, but we want to maximize
throughput (we'd like for fsync to not be a factor).
2023-11-29 11:08:07 +00:00
476009381a
Remove maximum enqueued requests limit
...
We already drop in-flight requests that we're already processing,
so I don't think this matters very much currently.
2023-11-29 11:08:07 +00:00
c5562c7ca3
Parallelize CDC by directory
...
Fixes #66 .
2023-11-29 11:08:07 +00:00
340e7f2f37
Harmonize addr-passing, add shuckle beacon and test it in kmod
2023-11-14 13:49:36 +00:00
2ad278adaa
Add ubuntu image to build, use jemalloc in release build
...
I want to use the introspection capabilities of jemalloc, and it
should also be much faster. Preserve alpine build for go build,
it's also really useful to test inside the kmod.
2023-11-13 15:44:55 +00:00
ad3c969772
Push full RocksDB stats to grafana
2023-11-09 16:48:51 +00:00
f70c484883
Dump RocksDB full statistics to file
2023-11-09 14:12:54 +00:00
057be91613
rocksDBStats -> rocksDBMetrics
2023-11-09 13:38:32 +00:00
c5979a9d90
Expose some RocksDB stats
2023-11-09 13:23:49 +00:00
03e9510255
Align xmon's app instances and systemd services
2023-11-08 14:36:58 +00:00
afc4e78a62
Reduce default CDC queue size
2023-11-05 22:38:57 +00:00
71556ce933
Switch to restech EggsFS rota
2023-11-03 14:23:44 +00:00
64d400fcfe
Insert shard/cdc metrics at more regular intervals
2023-11-03 13:49:38 +00:00
654c0d4db4
Report CDC queue size in grafana
2023-11-03 13:49:32 +00:00
9e21969637
Slightly tighter error checks
2023-10-11 13:40:46 +01:00
6726fff0fe
Better "innocuous error" handling in CDC
2023-10-04 18:12:15 +01:00
440a78510e
Add concrete quiet windows to C++ alerts
...
This together with the previous commits fixes #72 .
2023-10-02 23:06:40 +00:00
59237ed673
Limit number of open RocksDB files
...
We got to the point where we had ~4k open SST files per shard, which
meant that we eat up all the available FDs.
2023-09-30 11:08:35 +00:00
2679ee7c80
Retry RocksDB transactions if appropriate
2023-09-30 10:44:40 +00:00
1d4c4abafd
Correctly check that RocksDB txn succeeded
...
This was caught anyway by the fact that we check that the log index
is what we expect. Would have been very nasty otherwise.
The right thing to do is to check for `Status::TryAgain()` and
retry. `Status::Busy()` should never happen because we never
run transactions concurrently so far.
2023-09-30 09:51:26 +00:00
02838e228f
Correct xmon app types
2023-09-28 11:53:12 +00:00
762f047772
Add fsr17 and fsr18 to deployment
2023-09-19 12:56:34 +00:00
77ac15af8d
Allow to choose xmon env in C++ apps
2023-09-18 11:56:44 +00:00
7ec477ca9f
CDC.cpp: minor bugfix with using optional after reset()
...
harmless in release builds, since optional in questino is POD and destructor is a noop
2023-08-09 10:45:43 +00:00
32e2a011ee
More grafana fixes
2023-08-08 09:28:07 +00:00
467fcffefb
A few metrics fixes
2023-08-08 09:21:35 +01:00
e2246afc53
More tweaks to event loops
2023-08-08 09:21:35 +01:00
5117ddd16e
Add shard/CDC metrics
2023-08-08 09:21:35 +01:00
1922cf3c30
Factor out common looping patterns
2023-08-08 09:21:35 +01:00
93b212c665
Alert while initializing shard DB
2023-08-07 10:16:00 +00:00
b370118e90
Rate limit binnable xmon requests
...
This involved clearly separating non-clearable and clearable alerts,
which simplifies the design and I think satisfies all our needs.
2023-08-05 23:41:10 +01:00
9ef3162882
Add error count to inspect how things failed
2023-08-03 06:53:35 +00:00
acf4f129f6
Fix CDC txn status tracking
...
This might be a resolution to #38 , although I'm not sure yet.
2023-08-02 13:49:38 +00:00
63e2db0889
Cap maximum number of CDC requests
...
No point letting huge queues build -- especially now that we
deduplicate client requests.
2023-08-01 21:17:23 +01:00
fe2ce7aa17
See comments
2023-08-01 21:17:23 +01:00
a5eb12a262
Do not alert/log on innocuous shard error in CDC
2023-08-01 13:41:18 +00:00
Francesco Mazzoli