49bd2e6a2a
eggsblocks: conversion as a separate request
2024-08-21 15:39:11 +01:00
73622ce637
eggsblocks: write/read from new block format with crc after page
2024-08-20 14:55:45 +01:00
204cf5bb8f
eggsdbtools: output log entries from shard/cdc
2024-08-19 13:12:33 +00:00
cf40e318ec
shuckle: support BlockServicesWithFlagChangeReq
2024-07-24 10:08:01 +01:00
a41a4b7482
shuckle: drop BlockServiceInfoWithoutFlagsLastChanged
2024-07-23 15:40:44 +01:00
1b258dc422
all: migrate to new BlockServiceInfo
2024-07-23 13:53:07 +01:00
49723653f8
shuckle: BlockServiceInfo backward compatibility
...
* shuckle: rename BlockServiceInfo to BlockServiceInfoWithoutFlagsLastChanged
* shuckle: handle AllBlockServices
2024-07-23 13:10:57 +01:00
e2bfb15c5f
blockservice: add BlockFetchWithCrc
2024-07-12 14:24:37 +01:00
3195d39d9d
stats: fully remove everywhere
2024-07-09 15:22:10 +00:00
c7b6a1cbeb
stats: stop producing them
2024-07-09 15:57:01 +01:00
f3b7ef4d94
eggsgc: destroy decommissioned blocks through shuckle
2024-07-02 09:52:20 +00:00
6cf8a01272
core: UDPSocketPair send in a loop to avoid sendmmsg UIO_MAXIOV limit
2024-06-19 17:23:10 +01:00
78baed62a5
cdc: request checkpoints from shard and push through log
2024-06-13 16:24:22 +01:00
9e13d6b56e
shard: support checkpointed responses
2024-06-13 15:39:37 +01:00
2cd15fc0be
core: various protocol changes
2024-06-13 09:13:11 +01:00
6eaed4ff0e
core: remove Stopped (unused)
2024-06-12 15:10:58 +00:00
170f2fbc61
logsdb: add stats and expose in shard/cdc
2024-06-10 16:24:49 +01:00
1f145c030e
shard/cdc: support snapshoting
2024-05-23 10:17:59 +01:00
1446e4d0d2
cdc: force log cleanup after crash
...
Transactional db that CDC uses has a slightly
annoying property that it flushes WAL on transaction
start. As a result release point can get moved and
log records persisted even if we crash.
We want to remove them automatically for now.
2024-05-22 16:58:18 +00:00
f11b675807
shuckle: add cdc replicas to page
2024-05-22 11:57:34 +00:00
b524748210
cdc: use DEFAULT_UDP_MTU for serialization of entries
2024-05-21 14:23:27 +00:00
6faa917c18
Add endpoint and cli util to resurrect files
...
Only works in the same shard, for now.
2024-05-20 12:06:15 +00:00
0b3348b458
SharedRocksDB: (ref) paths in constructor
2024-05-17 14:23:25 +00:00
ab337068ad
httplib: use poll instead of select
2024-05-17 14:23:01 +00:00
f5e17dace5
cdc: add LogsDB
...
* cdc: pack req/resp into log entries and apply
* shard: drop support for unused incomming packet drop
* cdc: add logsdb
2024-05-14 12:50:17 +01:00
91d462ab0e
UDPSocketPair: (fix) dont look at unused fd-s
2024-05-14 08:51:29 +00:00
8a0ea10cde
core: UDPSocketPair and use IpPort AddrsInfo everywhere
...
* core: UDPSocketPair and use IpPort AddrsInfo everywhere
* Refactor UDPSocketPair a bit
* ci: kmod always delete img before create
* shuckle: fix scripts/json marshal
---------
Co-authored-by: Francesco Mazzoli <francesco.mazzoli@xtxmarkets.com >
2024-05-03 11:32:07 +01:00
cd8e52f8f7
Remove assertions in ShardDB
...
We got a crash because of it (presumably can happen if defrag
conflicts with migrate or something like that)
2024-05-01 08:13:19 +00:00
d3be7bf53a
Remove old-style register block service request
2024-04-22 19:20:04 +00:00
f109e3542b
Have eggsblocks to refresh decommissioned block services
...
So that we can reliably ignore stale block services in GC (done in
a future commit). To enable this and future-proof this kind of
mechanism (e.g. having `eggsblocks` to mark something as D itself)
I added a new way to register the block service that lets you mask
which flags you're checking. I'll remove the old way once we've
rolled out everywhere.
2024-04-22 18:47:54 +00:00
6007192a31
bincode: generate STATIC_SIZE for req/resp
2024-04-22 13:49:49 +01:00
43f69b1f7e
shuckle: support ClearShardInfoReq/Resp
2024-04-16 10:25:24 +01:00
a579b41dfc
shuckle: support for MoveLeaderReq
2024-04-15 14:24:15 +01:00
20e7635d75
Clear data when request fails in Shuckle.cpp
2024-04-10 10:39:30 +00:00
e42c548777
Make SwapSpans idempotent
2024-04-09 07:53:10 +01:00
4dd929a798
Implement swap spans
2024-04-09 07:53:10 +01:00
409b126e4b
cdc: use SharedRocksDB
2024-04-05 23:22:39 +01:00
0a6e4be683
shard: disable double flush and improve kmod vm
2024-04-05 17:34:42 +01:00
de17eee24f
core: fix incorrect return in connectHost
2024-04-03 15:08:48 +01:00
30ee029f7e
shuckle: make requests interruptable and pass timeout to all operations
...
This means that they'll be interrupted at shutdown, rather than holding everything up when shuckle is overloaded.
We also detect idle connection or slow transmitting data.
2024-04-02 18:15:29 +01:00
68c4c03750
Add command to run some checks directly in RocksDB database
2024-03-27 18:45:14 +00:00
aebcce4017
logsdb: fix assert for last relased going backwards
2024-03-25 10:31:58 +00:00
7df0a5da89
shard: cli options now match migration phases for LogsDB, and support manual failover
2024-03-20 15:34:55 +00:00
fd9079febf
Rate limited shuckle endpoint to decom blockservices
2024-03-20 15:16:00 +00:00
1cf299bfac
Use atomics where appropriate
2024-03-20 13:21:18 +00:00
f85714dbba
Use pthread_self() to get pthread thread id
2024-03-20 13:11:14 +00:00
3a6e498664
Make some Loop methods static
2024-03-20 13:00:18 +00:00
9bc7e209e4
Safer ShuckleSock
2024-03-20 11:33:39 +00:00
66fe0a2621
Correct pthread_timedjoin_np handling
2024-03-20 11:13:26 +00:00
8f1ba6361b
Resist interruptions when joining threads
2024-03-20 10:32:42 +00:00
Miroslav Crnic