mirror/ternfs-XTXMarkets
1
0
Fork 0
mirror of https://github.com/XTXMarkets/ternfs.git synced 2025-12-19 01:30:13 -06:00
Code Issues Packages Projects Releases Wiki Activity

README.md

Browse Source
This commit is contained in:
Francesco Mazzoli
2025-09-04 13:02:46 +00:00
parent 05799dc1ed
commit d7fc8ec14d
1 changed files with 10 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
README.md
View File

@@ -56,7 +56,7 @@ TODO decorate list below with links drilling down on specific concepts.
* file to blocks mapping * file to blocks mapping
* block service to file mapping * block service to file mapping
* UDP bincode req/resp * UDP bincode req/resp
* state persisted through RocksDB, currently single physical instance per logical instance, will move to a Paxos (or similar) cluster per logical instance (see #56) * state persisted through RocksDB with 5-node distributed consensus through LogsDB
* communicates with shuckle to fetch block services, register itself, insert statistics * communicates with shuckle to fetch block services, register itself, insert statistics
* **CDC** * **CDC**
* 1 logical instance * 1 logical instance
@@ -70,14 +70,14 @@ TODO decorate list below with links drilling down on specific concepts.
* very little state required * very little state required
* information about which transactions are currently being executed and which are queued (currently transactions are executed serially) * information about which transactions are currently being executed and which are queued (currently transactions are executed serially)
* directory -> parent directory mapping to perform "no loops" checks * directory -> parent directory mapping to perform "no loops" checks
* state persisted through RocksDB, currently single physical instance, will move to Paxos or similar just like shards (see #56) * state persisted through RocksDB with 5-node distributed consensus through LogsDB
* communicates with the shards to perform the cross-directory actions * communicates with the shards to perform the cross-directory actions
* communicates with shuckle to register itself, fetch shards, insert statistics * communicates with shuckle to register itself, fetch shards, insert statistics
* **block service** * **block service**
* up to 1 million logical instances * up to 1 million logical instances
* 1 logical instance = 1 disk * 1 logical instance = 1 disk
* 1 physical instance handles ~100 logical instances (since there are ~100 disks per server) * 1 physical instance handles all the disks in the server it's running on
* `ternblocks`, Go binary (for now, will be rewritten in C++ eventually) * `ternblocks`, Go binary (for now, might become C++ in the future if performance requires it)
* stores "blocks", which are blobs of data which contain file contents * stores "blocks", which are blobs of data which contain file contents
* each file is split in many blocks stored on many block services (so that if up to 4 block services fail we can always recover files) * each file is split in many blocks stored on many block services (so that if up to 4 block services fail we can always recover files)
* single instances are not redundant, the redundancy is handled by spreading files over many instances so that we can recover their contents * single instances are not redundant, the redundancy is handled by spreading files over many instances so that we can recover their contents
@@ -98,14 +98,17 @@ TODO decorate list below with links drilling down on specific concepts.
* **FUSE** * **FUSE**
* `ternfuse`, Go FUSE implementation of a TernFS client * `ternfuse`, Go FUSE implementation of a TernFS client
* much slower but should be almost fully functional (there are some limitations concerning when a file gets flushed) * much slower but should be almost fully functional (there are some limitations concerning when a file gets flushed)
* **daemons**, these also talk to all of the servers * **S3**
* `terns3`, Go implementation of the S3 API
* minimal example intended as a start point for a more serious implementation
* **daemons**, these also talk to all of the servers, and all live in `terngc`
* **GC** * **GC**
* `terngc`, Go binary
* permanently deletes expired snapshots (i.e. deleted but not yet purged data) * permanently deletes expired snapshots (i.e. deleted but not yet purged data)
* cleans up all blocks for permanently deleted files * cleans up all blocks for permanently deleted files
* **scrubber** * **scrubber**
* TODO see #32
* goes around detecting and repairing bitrot * goes around detecting and repairing bitrot
* **migrator**
* evacuates failed disks
* **additional tools** * **additional tools**
* `ternrun`, a tool to quickly spin up a TernFS instance * `ternrun`, a tool to quickly spin up a TernFS instance
* `terntests`, runs some integration tests * `terntests`, runs some integration tests