mirror/ternfs-XTXMarkets
1
0
Fork 0
mirror of https://github.com/XTXMarkets/ternfs.git synced 2026-01-06 02:49:45 -06:00
Code Issues Packages Projects Releases Wiki Activity
278 Commits 6 Branches 0 Tags
cd86e632e2bda9d42c809ce0fa1adb4bf9f3ff39
Commit Graph

278 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Francesco Mazzoli
cd86e632e2 Implement RS recovery, although it won't really be used now... 
...since it only relies on block service flags, and we don't
set them right now.
 2023-06-03 17:27:54 +00:00
Francesco Mazzoli
efb92be31a Use bitmaps for RS recover API 
The previous array was sort of silly, and particularly silly now
from the kernel.
 2023-06-01 14:41:21 +00:00
Francesco Mazzoli
974500633a Implement stripe prefetching 2023-06-01 13:01:33 +00:00
Francesco Mazzoli
d1494e3c84 Simplify ownership of fetch requests... 
...preparing for prefetching. I like this scheme more anyway, I'll
probably switch to something like this for block writing too.
 2023-06-01 10:10:25 +00:00
Francesco Mazzoli
273fa40dda Fix USR2 logging teardown 2023-06-01 10:10:25 +00:00
Saulius Grusnys
e64aea102a Switch logging to the ecninfra logger based on sirupsen/logrus. (#3) 2023-05-31 14:20:55 +01:00
Francesco Mazzoli
55074b16b4 Implement fs stat 
10.97.12.10:10001       29P  208T   29P   1% /home/restechprod/eggs/mnt
 2023-05-29 18:49:50 +00:00
Francesco Mazzoli
d48b100ab1 Implement symlinks 2023-05-29 17:32:15 +00:00
Francesco Mazzoli
ee63265d1f Do not use localhost in tests 
For some reason the DNS server gets hammered with localhost queries,
it's not clear why DNS requests would be needed at all to resolve
localhost (maybe something musl related) but switching to the IP
should help.
 2023-05-29 17:32:15 +00:00
Francesco Mazzoli
499bada153 Other silly eggsktools utility 2023-05-29 17:32:15 +00:00
Saulius Grusnys
f3a3b289bb Add basic xmon connection to shuckle. (#1) 
Add basic xmon connection to shuckle.
 2023-05-29 11:26:48 +01:00
Francesco Mazzoli
7e25c1fd95 Do not crash if we can't find blocks 2023-05-29 09:52:01 +00:00
Francesco Mazzoli
a12a938c40 syslogify logs 2023-05-29 09:52:01 +00:00
Francesco Mazzoli
5428b33922 View transient files in shuckle 2023-05-28 22:56:11 +00:00
Francesco Mazzoli
b58de80342 Fix old edges GC 2023-05-28 22:14:39 +00:00
Francesco Mazzoli
45471eded4 eggsktools readfile QOL 2023-05-28 22:12:49 +00:00
Francesco Mazzoli
79fbf4f029 Add more detailed block tracing 
This reveals the `fsync` on the block writing side is the only thing
preventing a block to be written at the full 10Gbit/s. There are
other things that slow down the end-to-end writing time (like setting
up the span), but actually the pipe itself is as fast as it should,
which is nice.
 2023-05-28 11:30:26 +00:00
Francesco Mazzoli
3a1e70d769 Write block service responses in one go 2023-05-28 10:47:24 +00:00
Francesco Mazzoli
05d6553390 Do not exit module before probes are gone 2023-05-28 10:46:52 +00:00
Francesco Mazzoli
cc165539c5 Use __print_symbolic, custom string functions don't work in traces 2023-05-27 23:13:10 +00:00
Francesco Mazzoli
4526b85dec Small tweak 2023-05-27 22:44:52 +00:00
Francesco Mazzoli
845e86e952 Trace eggsfs errors 2023-05-27 20:31:39 +00:00
Francesco Mazzoli
6bf025aa66 Add creation time on readdir 
The creation time/revalidation business is currently probably a bit
broken anyway, the right thing to do is probably just to retry when
we get `EGGSFS_ERR_MISMATCHING_CREATION_TIME`.
 2023-05-27 20:30:39 +00:00
Francesco Mazzoli
951628ecfd Add str functions for kmod kinds 2023-05-27 20:30:16 +00:00
Francesco Mazzoli
d65b3ed12d Remove useless defines 2023-05-27 09:01:19 +00:00
Francesco Mazzoli
a4bc32a18f Span drop improvements 
We could get into situations where async droppings were scheduled
at every read.
 2023-05-26 17:22:43 +00:00
Francesco Mazzoli
9112cac1b0 si_mem_available() returns the _pages_ available, not bytes 2023-05-26 17:22:43 +00:00
Francesco Mazzoli
02d6c4828c More little bug fixes in error situations. 2023-05-26 17:22:43 +00:00
Francesco Mazzoli
2aaec3bb15 Do not crash > 10 iterations of eggsfs_get_span 
See comment on why this can happen. Also, now I add the spans to
the LRU after every metadata request, which should significantly
reduce the section in which we can loop in this fashion.
 2023-05-26 17:22:43 +00:00
Francesco Mazzoli
03497d948c Fix bad error reporting in eggsfs_file_read_iter 2023-05-26 17:22:43 +00:00
Francesco Mazzoli
f95d177c34 WIP commit... 
...which I mistakely left in and I'm too lazy to fix.
 2023-05-26 17:22:30 +00:00
Francesco Mazzoli
fc0ae851c8 Typo 2023-05-26 10:14:32 +00:00
Francesco Mazzoli
dd9c4ce7a9 Correct GB/s measure in eggscli when writing to many block services 2023-05-26 10:06:03 +00:00
Francesco Mazzoli
f98f0f3e95 Move some utility around, allow to deploy kmod easily 2023-05-26 10:05:25 +00:00
Francesco Mazzoli
1458759534 Allow to enable shard/cdc debugging at runtime using USR2 2023-05-26 10:03:59 +00:00
Francesco Mazzoli
1b83a50419 Ensure files actually are on different failure domains... 
...in a very lazy way which will probably do for now.
 2023-05-25 15:14:02 +00:00
Francesco Mazzoli
1a701f22a2 Actually reclaim spans 2023-05-25 10:22:31 +00:00
Francesco Mazzoli
52e50a1888 Simplify span cache locking further 
With seqcount with the current locking strategy we can get into
subtle nasty situations. For now, just be "dumb" with a rw semaphore.

For example, consider the case where we get the span in the
seqcount protected section in get_span. Between getting the
span and acquiring it the reclaimer might free it.

Ideally we'd do something similar to what Paweł planned (seqcount
and RCU protection) but let's start with something simple which
seems to work.

Also, shard the LRU by 128, to reduce contention, and fix a couple
of unrelated bugs.
 2023-05-25 08:34:40 +00:00
Francesco Mazzoli
0bc1994fe0 Fetch all spans metadata at once 
Will make prefecthing easier.
 2023-05-24 09:56:49 +00:00
Francesco Mazzoli
63e7112b2c Remove blocksimple terminology 2023-05-24 08:53:14 +00:00
Francesco Mazzoli
47c1022896 Bugfixes 2023-05-23 22:10:11 +00:00
Francesco Mazzoli
8f32ecc1b6 Cached spans reclamation 
Right now this is very crude (global spinlock), but reasonably simple.
We can improve if needed.
 2023-05-23 16:57:29 +00:00
Francesco Mazzoli
a61fce55f8 Simple write test for block service 
We can only write 3.2Gbit/s right now, so that's definitely something
to improve.
 2023-05-22 13:08:22 +00:00
Francesco Mazzoli
9b4975ae8e Fix bincode_tests string handling 2023-05-22 10:13:00 +00:00
Francesco Mazzoli
ac25c0aa3d Limit to 64 CPP expansions 
Seems a more reasonable number than 256 for some reason.
 2023-05-22 08:13:22 +00:00
Francesco Mazzoli
3cccb451d6 Remove unused define 2023-05-22 08:11:40 +00:00
Francesco Mazzoli
1eab8ee6cf Add versions to some RocksDB values 
Only the ones where it is needed -- in some cases we can just
modify the keys (e.g. metadata stuff).

Also, come up with a sort of horrifying but more robust way
to specify the RocksDB values with the C preprocessor.
 2023-05-22 08:03:01 +00:00
Francesco Mazzoli
b292333baf Fix bad unaligned access of AVX vectors 2023-05-19 14:06:22 +00:00
Francesco Mazzoli
972bb55356 A few tweaks 
Notably, make heavy debug output a sysctl flag.
 2023-05-19 10:57:26 +00:00
Francesco Mazzoli
7335f89bf0 wip 2023-05-19 08:23:34 +00:00