Miroslav Crnic
0b3348b458
SharedRocksDB: (ref) paths in constructor
2024-05-17 14:23:25 +00:00
Miroslav Crnic
ab337068ad
httplib: use poll instead of select
2024-05-17 14:23:01 +00:00
Miroslav Crnic
f5e17dace5
cdc: add LogsDB
...
* cdc: pack req/resp into log entries and apply
* shard: drop support for unused incomming packet drop
* cdc: add logsdb
2024-05-14 12:50:17 +01:00
Miroslav Crnic
91d462ab0e
UDPSocketPair: (fix) dont look at unused fd-s
2024-05-14 08:51:29 +00:00
Miroslav Crnic
aa8925adf9
shard: fix stats
2024-05-04 09:17:45 +01:00
Miroslav Crnic
8a0ea10cde
core: UDPSocketPair and use IpPort AddrsInfo everywhere
...
* core: UDPSocketPair and use IpPort AddrsInfo everywhere
* Refactor UDPSocketPair a bit
* ci: kmod always delete img before create
* shuckle: fix scripts/json marshal
---------
Co-authored-by: Francesco Mazzoli <francesco.mazzoli@xtxmarkets.com >
2024-05-03 11:32:07 +01:00
Francesco Mazzoli
ded0787c18
Fix no-return function
2024-05-01 08:53:38 +00:00
Francesco Mazzoli
cd8e52f8f7
Remove assertions in ShardDB
...
We got a crash because of it (presumably can happen if defrag
conflicts with migrate or something like that)
2024-05-01 08:13:19 +00:00
Miroslav Crnic
bbe201964d
shard: bump quiet time for metric inserter alert to 5 min
2024-04-30 17:54:13 +01:00
Francesco Mazzoli
40ed10b2c6
Bump quiet period for metrics alerts
...
VictoraMetrics is often acting up
2024-04-29 11:39:54 +00:00
Francesco Mazzoli
d3be7bf53a
Remove old-style register block service request
2024-04-22 19:20:04 +00:00
Francesco Mazzoli
f109e3542b
Have eggsblocks to refresh decommissioned block services
...
So that we can reliably ignore stale block services in GC (done in
a future commit). To enable this and future-proof this kind of
mechanism (e.g. having `eggsblocks` to mark something as D itself)
I added a new way to register the block service that lets you mask
which flags you're checking. I'll remove the old way once we've
rolled out everywhere.
2024-04-22 18:47:54 +00:00
Miroslav Crnic
6007192a31
bincode: generate STATIC_SIZE for req/resp
2024-04-22 13:49:49 +01:00
Miroslav Crnic
43f69b1f7e
shuckle: support ClearShardInfoReq/Resp
2024-04-16 10:25:24 +01:00
Miroslav Crnic
a579b41dfc
shuckle: support for MoveLeaderReq
2024-04-15 14:24:15 +01:00
Miroslav Crnic
c1cea71f55
shard: dont serve reads if not leader
2024-04-15 14:23:55 +01:00
Miroslav Crnic
5fad8546bd
shard: increase block fetch interval on leader to 2min
2024-04-10 13:51:00 +01:00
Francesco Mazzoli
51cda3a98b
Clear data in shuckle block service fetch loop for symmetry
2024-04-10 10:39:46 +00:00
Francesco Mazzoli
20e7635d75
Clear data when request fails in Shuckle.cpp
2024-04-10 10:39:30 +00:00
Francesco Mazzoli
d8267f18c6
Add more fsck functionality
2024-04-09 17:57:00 +00:00
Miroslav Crnic
2de5c6b5dc
shard: delay block service update on leader
2024-04-09 17:59:59 +01:00
Francesco Mazzoli
eb766f2fb5
Do not attempt to cross-shard unlink file if the file is a directory
2024-04-09 11:43:03 +00:00
Francesco Mazzoli
e42c548777
Make SwapSpans idempotent
2024-04-09 07:53:10 +01:00
Francesco Mazzoli
13bf9a005a
Run fsck at the end of tests
2024-04-09 07:53:10 +01:00
Francesco Mazzoli
f10c7e0744
Add eggscli functionality to "defrag" files
...
Fixes #50 .
2024-04-09 07:53:10 +01:00
Francesco Mazzoli
4dd929a798
Implement swap spans
2024-04-09 07:53:10 +01:00
Miroslav Crnic
409b126e4b
cdc: use SharedRocksDB
2024-04-05 23:22:39 +01:00
Miroslav Crnic
fcb8ab79f8
shard: always run with logsdb, disable separate ci
2024-04-05 21:50:33 +01:00
Miroslav Crnic
0a6e4be683
shard: disable double flush and improve kmod vm
2024-04-05 17:34:42 +01:00
Francesco Mazzoli
d44b331739
Remove one-off fixup command in db tools
2024-04-05 12:55:35 +00:00
Francesco Mazzoli
498eb0feda
Add single-use utility to fixup some bad RocksDB values in shard 0
2024-04-04 15:55:11 +00:00
Miroslav Crnic
9f1dbf06d0
eggsshuckle: detect leader change attempt and return error
2024-04-04 13:39:08 +01:00
Miroslav Crnic
de17eee24f
core: fix incorrect return in connectHost
2024-04-03 15:08:48 +01:00
Miroslav Crnic
30ee029f7e
shuckle: make requests interruptable and pass timeout to all operations
...
This means that they'll be interrupted at shutdown, rather than holding everything up when shuckle is overloaded.
We also detect idle connection or slow transmitting data.
2024-04-02 18:15:29 +01:00
Francesco Mazzoli
b38dcc550b
fsck fixes/logs
2024-03-28 10:04:15 +00:00
Miroslav Crnic
7e394d2d5e
SharDBTools: add missing include
2024-03-28 09:29:05 +00:00
Francesco Mazzoli
68c4c03750
Add command to run some checks directly in RocksDB database
2024-03-27 18:45:14 +00:00
Miroslav Crnic
16fb2b3797
shard: auto drop all data in LogsDB if not in use
2024-03-27 17:09:51 +00:00
Miroslav Crnic
8035e003f0
shard: re-enable starting with logsdb
2024-03-27 13:16:32 +00:00
Francesco Mazzoli
8de52aa6c9
Simplify eggsdbtools command parsing/execution
2024-03-27 12:18:16 +00:00
Miroslav Crnic
65e5b6e7ac
cdc: shuckle-stats
2024-03-26 09:40:44 +00:00
Miroslav Crnic
f773a88bd9
shard: enable writes
2024-03-26 08:39:17 +00:00
Miroslav Crnic
3c5318e7dc
shard: read-only and no logsdb
2024-03-26 05:47:47 +00:00
Francesco Mazzoli
7a5fc9f8a9
Allow to disable shuckle stat inserting
2024-03-25 16:08:54 +00:00
Miroslav Crnic
aebcce4017
logsdb: fix assert for last relased going backwards
2024-03-25 10:31:58 +00:00
Miroslav Crnic
7df0a5da89
shard: cli options now match migration phases for LogsDB, and support manual failover
2024-03-20 15:34:55 +00:00
Saulius Grusnys
fd9079febf
Rate limited shuckle endpoint to decom blockservices
2024-03-20 15:16:00 +00:00
Francesco Mazzoli
1cf299bfac
Use atomics where appropriate
2024-03-20 13:21:18 +00:00
Francesco Mazzoli
f85714dbba
Use pthread_self() to get pthread thread id
2024-03-20 13:11:14 +00:00
Francesco Mazzoli
3a6e498664
Make some Loop methods static
2024-03-20 13:00:18 +00:00
