Miroslav Crnic
75dfd723c0
shuckle: fix ClearCdcInfoReq name
2024-09-17 10:05:46 +00:00
Miroslav Crnic
b2ea95091a
shuckle: support cdc replica moving across hosts
2024-09-16 17:31:47 +01:00
Miroslav Crnic
59fc480e85
shuckle: remove unused requests
2024-09-16 15:21:06 +01:00
Miroslav Crnic
2dec9ec117
cdc: register location
2024-09-12 14:27:55 +01:00
Miroslav Crnic
f8c4096dc9
shard: register location
2024-09-12 14:27:27 +01:00
Miroslav Crnic
8ac93a4c54
shuckle: add location for all services
2024-09-11 16:59:19 +01:00
Miroslav Crnic
76695ec22b
shard: check enough current block services
2024-09-02 16:49:31 +00:00
Miroslav Crnic
35fe242fe3
eggsshard: check shuckle gives services from different fd-s
2024-09-02 16:05:59 +01:00
Miroslav Crnic
9cd425d7f3
eggsblocks/kmod: add file_id to FetchBlockWithCrcReq
2024-08-22 14:11:01 +01:00
Miroslav Crnic
49bd2e6a2a
eggsblocks: conversion as a separate request
2024-08-21 15:39:11 +01:00
Miroslav Crnic
73622ce637
eggsblocks: write/read from new block format with crc after page
2024-08-20 14:55:45 +01:00
Miroslav Crnic
204cf5bb8f
eggsdbtools: output log entries from shard/cdc
2024-08-19 13:12:33 +00:00
Miroslav Crnic
cf40e318ec
shuckle: support BlockServicesWithFlagChangeReq
2024-07-24 10:08:01 +01:00
Miroslav Crnic
a41a4b7482
shuckle: drop BlockServiceInfoWithoutFlagsLastChanged
2024-07-23 15:40:44 +01:00
Miroslav Crnic
1b258dc422
all: migrate to new BlockServiceInfo
2024-07-23 13:53:07 +01:00
Miroslav Crnic
49723653f8
shuckle: BlockServiceInfo backward compatibility
...
* shuckle: rename BlockServiceInfo to BlockServiceInfoWithoutFlagsLastChanged
* shuckle: handle AllBlockServices
2024-07-23 13:10:57 +01:00
Miroslav Crnic
e2bfb15c5f
blockservice: add BlockFetchWithCrc
2024-07-12 14:24:37 +01:00
Miroslav Crnic
3195d39d9d
stats: fully remove everywhere
2024-07-09 15:22:10 +00:00
Miroslav Crnic
c7b6a1cbeb
stats: stop producing them
2024-07-09 15:57:01 +01:00
Miroslav Crnic
f3b7ef4d94
eggsgc: destroy decommissioned blocks through shuckle
2024-07-02 09:52:20 +00:00
Miroslav Crnic
6cf8a01272
core: UDPSocketPair send in a loop to avoid sendmmsg UIO_MAXIOV limit
2024-06-19 17:23:10 +01:00
Miroslav Crnic
78baed62a5
cdc: request checkpoints from shard and push through log
2024-06-13 16:24:22 +01:00
Miroslav Crnic
9e13d6b56e
shard: support checkpointed responses
2024-06-13 15:39:37 +01:00
Miroslav Crnic
2cd15fc0be
core: various protocol changes
2024-06-13 09:13:11 +01:00
Miroslav Crnic
6eaed4ff0e
core: remove Stopped (unused)
2024-06-12 15:10:58 +00:00
Miroslav Crnic
170f2fbc61
logsdb: add stats and expose in shard/cdc
2024-06-10 16:24:49 +01:00
Miroslav Crnic
1f145c030e
shard/cdc: support snapshoting
2024-05-23 10:17:59 +01:00
Miroslav Crnic
1446e4d0d2
cdc: force log cleanup after crash
...
Transactional db that CDC uses has a slightly
annoying property that it flushes WAL on transaction
start. As a result release point can get moved and
log records persisted even if we crash.
We want to remove them automatically for now.
2024-05-22 16:58:18 +00:00
Miroslav Crnic
f11b675807
shuckle: add cdc replicas to page
2024-05-22 11:57:34 +00:00
Miroslav Crnic
b524748210
cdc: use DEFAULT_UDP_MTU for serialization of entries
2024-05-21 14:23:27 +00:00
Francesco Mazzoli
6faa917c18
Add endpoint and cli util to resurrect files
...
Only works in the same shard, for now.
2024-05-20 12:06:15 +00:00
Miroslav Crnic
0b3348b458
SharedRocksDB: (ref) paths in constructor
2024-05-17 14:23:25 +00:00
Miroslav Crnic
ab337068ad
httplib: use poll instead of select
2024-05-17 14:23:01 +00:00
Miroslav Crnic
f5e17dace5
cdc: add LogsDB
...
* cdc: pack req/resp into log entries and apply
* shard: drop support for unused incomming packet drop
* cdc: add logsdb
2024-05-14 12:50:17 +01:00
Miroslav Crnic
91d462ab0e
UDPSocketPair: (fix) dont look at unused fd-s
2024-05-14 08:51:29 +00:00
Miroslav Crnic
8a0ea10cde
core: UDPSocketPair and use IpPort AddrsInfo everywhere
...
* core: UDPSocketPair and use IpPort AddrsInfo everywhere
* Refactor UDPSocketPair a bit
* ci: kmod always delete img before create
* shuckle: fix scripts/json marshal
---------
Co-authored-by: Francesco Mazzoli <francesco.mazzoli@xtxmarkets.com >
2024-05-03 11:32:07 +01:00
Francesco Mazzoli
cd8e52f8f7
Remove assertions in ShardDB
...
We got a crash because of it (presumably can happen if defrag
conflicts with migrate or something like that)
2024-05-01 08:13:19 +00:00
Francesco Mazzoli
d3be7bf53a
Remove old-style register block service request
2024-04-22 19:20:04 +00:00
Francesco Mazzoli
f109e3542b
Have eggsblocks to refresh decommissioned block services
...
So that we can reliably ignore stale block services in GC (done in
a future commit). To enable this and future-proof this kind of
mechanism (e.g. having `eggsblocks` to mark something as D itself)
I added a new way to register the block service that lets you mask
which flags you're checking. I'll remove the old way once we've
rolled out everywhere.
2024-04-22 18:47:54 +00:00
Miroslav Crnic
6007192a31
bincode: generate STATIC_SIZE for req/resp
2024-04-22 13:49:49 +01:00
Miroslav Crnic
43f69b1f7e
shuckle: support ClearShardInfoReq/Resp
2024-04-16 10:25:24 +01:00
Miroslav Crnic
a579b41dfc
shuckle: support for MoveLeaderReq
2024-04-15 14:24:15 +01:00
Francesco Mazzoli
20e7635d75
Clear data when request fails in Shuckle.cpp
2024-04-10 10:39:30 +00:00
Francesco Mazzoli
e42c548777
Make SwapSpans idempotent
2024-04-09 07:53:10 +01:00
Francesco Mazzoli
4dd929a798
Implement swap spans
2024-04-09 07:53:10 +01:00
Miroslav Crnic
409b126e4b
cdc: use SharedRocksDB
2024-04-05 23:22:39 +01:00
Miroslav Crnic
0a6e4be683
shard: disable double flush and improve kmod vm
2024-04-05 17:34:42 +01:00
Miroslav Crnic
de17eee24f
core: fix incorrect return in connectHost
2024-04-03 15:08:48 +01:00
Miroslav Crnic
30ee029f7e
shuckle: make requests interruptable and pass timeout to all operations
...
This means that they'll be interrupted at shutdown, rather than holding everything up when shuckle is overloaded.
We also detect idle connection or slow transmitting data.
2024-04-02 18:15:29 +01:00
Francesco Mazzoli
68c4c03750
Add command to run some checks directly in RocksDB database
2024-03-27 18:45:14 +00:00
