Sebastian Jeltsch
1ea889eb8a
Update Rust dependencies.
2025-05-27 22:09:49 +02:00
Sebastian Jeltsch
5152cf9000
Replace vendored refinery with a more proper fork stripping all the unused bits and bobs.
2025-05-27 22:01:06 +02:00
Sebastian Jeltsch
01feb22e44
serde_rusqlite finally got an update. Remove vendored version.
2025-05-27 12:15:52 +02:00
Sebastian Jeltsch
2b9d65074f
Best effort attach other *.db files found in <traildepot>/data.
...
All the admin operations do not properly qualify tables, views... . To
be generally useful, this is also missigng schema-handling for multiple
dbs.
We could just rely on migration files fully qualifying, however this
wouldn't work well with db renames. We also don't want to store
migration state for "test.db" in "main.db".
2025-05-22 15:28:30 +02:00
Sebastian Jeltsch
522e0e13bf
Add some early basics to deal with multiple attached databases.
2025-05-22 14:17:02 +02:00
Sebastian Jeltsch
a40fc92b9e
Update Rust dependencies.
2025-05-17 11:08:31 +02:00
Sebastian Jeltsch
48fd0b214b
Propagate execute_batch errors in first statement.
2025-05-11 13:00:34 +02:00
Sebastian Jeltsch
6b58a6b531
Fix error handling of Connection::execute_batch.
2025-05-06 10:22:41 +02:00
Sebastian Jeltsch
69b4f69d7c
Periodically yield back to the JS event loop while waiting for a DB write lock to allow it to make progress even while the lock is held.
2025-05-05 16:14:32 +02:00
Sebastian Jeltsch
c06b3ad1ef
Close sqlite connections locally rather than on the worker loops.
2025-05-05 10:30:08 +02:00
Sebastian Jeltsch
5f1b5614b4
Add an experimental API to get a write-lock to the underlying rusqlite::Connection.
2025-05-03 12:10:19 +02:00
Sebastian Jeltsch
c4a4dfd575
Update rusqlite to v0.35, which has more strict query validation \o/. Also update other Rust and JS deps.
2025-05-03 11:42:59 +02:00
Sebastian Jeltsch
122dd1ebb0
Allow applying recorded transaction as migration or plain transaction. Add tests.
2025-05-02 21:31:34 +02:00
Sebastian Jeltsch
e897169aac
Move JS runtime into a separate crate.
2025-04-29 13:47:10 +02:00
Sebastian Jeltsch
9ca34f46f3
Further tune SQLite execution model.
...
Previously reader and writer channels were operating entirely
independently, which yields the lowest overhead for pure read and pure
write benchmarks.
In mixed workloads like prod, however, this is not ideal since we're
running into SQLite's own file-locking, which has more overhead than
in-process locking.
With this change, we use an RwLock to reduce congestion while still
allowing parallel reads.
Together with a low-overhead kanal channel, we see little to no
regression for pure read and pure write benchmarks, while seeing about a
3x improvement for mixed workloads.
We also move ACL checks for Create/Update queries to the writer SQLite
connection, to reduce thrashing.
2025-04-28 14:11:44 +02:00
Sebastian Jeltsch
706870c50a
Minor: use const reference internally for SQLite read queue operations.
2025-04-27 12:07:58 +02:00
Sebastian Jeltsch
fe2e8cfda4
Clean up Sqlite connection setup, no-parallel reads for in-memory dbs, and add a placeholder storage for persisting apalis queues.
2025-04-23 11:56:36 +02:00
Sebastian Jeltsch
d6b199587d
Update to Rust edition 2024.
2025-04-20 07:36:26 +02:00
Sebastian Jeltsch
e0ad4a898c
Extend trailbase-sqlites execution model to allow for parallel reads. This reduces the latency long-tail for slow reads.
...
Note that this complicates the APIs, since it pushes the responsibility
of declaring a query a read or write to the user to then be scheduled
appropriately.
Add `.(read_|)query_row_f` APIs similar to rusqlites
`conn.query_row` accepting a `|row| -> Result<T>` to reduce the use of
`Row` and `Rows`.
Make benchmarks more isolated by not sharing a DB across runs
accumulating writes.
2025-04-15 12:15:40 +02:00
Sebastian Jeltsch
eecab32d39
Minor: consistent tokio and env_logger dependencies.
2025-04-14 09:25:44 +02:00
Sebastian Jeltsch
9e040d34f6
Make chat-room benchmark more realistic: hammer server in parallel.
2025-04-13 12:28:20 +02:00
Sebastian Jeltsch
914a9b3f03
Make query mix benchmark's slow queries slower and group benchmarks with throughput as requested.
2025-04-13 11:12:08 +02:00
Sebastian Jeltsch
844ccfa80c
Add a mixed query benchmark to trailbase-sqlite and make all benchmarks concurrently hammer the connection.
2025-04-12 15:23:17 +02:00
Sebastian Jeltsch
faa1712601
Add read benchmarks to trailbase-sqlite.
2025-04-12 13:09:22 +02:00
Sebastian Jeltsch
d0d60857eb
Add a shared and TL/pool SQLite connection implementation to the benchmarks as a mere reference.
...
These implementations are dangerous since despite being behind an async
API, they block on I/O and can thus clog up the tokio runtime. Which is
fine if SQLite is all you do but really bad if other, independent work
gets stalled.
2025-04-12 11:56:47 +02:00
Sebastian Jeltsch
458adcf9ec
Add a benchmark scaffold to sqlite.
2025-04-12 10:35:55 +02:00
Sebastian Jeltsch
c0c7681f4a
Restructure crates: make the sqlite execution model more standalone, i.e. move more extension stuff back to extension crate and schema stuff to a new schema crate.
2025-04-08 12:25:10 +02:00
Sebastian Jeltsch
503d7cae9b
Pass connection constructor to prepare for broader concurrency models.
2025-04-08 10:05:56 +02:00
Sebastian Jeltsch
f7cce2dc44
Fix logging: deterministically and separately initialize log and tracing.
2025-04-04 13:23:26 +02:00
Sebastian Jeltsch
31d7dc4158
Use statement cache in more places.
2025-04-03 21:59:23 +02:00
Sebastian Jeltsch
edfbd4c4f6
Manage file deletions through an auxiliary table: "_file_deletions" and triggers.
...
This allows deleting files eagerly w/o a transaction (i.e. pre-op query to get
old values), allows for defered cleanups, and retries.
2025-04-02 00:15:05 +02:00
Sebastian Jeltsch
d9525cd754
Minor: remove log dep from library crates.
2025-03-31 11:26:03 +02:00
Sebastian Jeltsch
3fe9dc8494
Update UUID sqlite extensions, fix example blog and clean it up.
2025-03-24 15:34:01 +01:00
Sebastian Jeltsch
2f07a86030
Update rusqlite: v0.33 -> v0.34.
2025-03-10 14:10:18 +01:00
Sebastian Jeltsch
c63f3f4089
Cleanup: remove legacy query_one_row utility.
2025-03-10 11:09:31 +01:00
Sebastian Jeltsch
68824465a0
Publish updated crates: trailbase(|-sqlite|-extension).
2025-03-09 12:08:46 +01:00
Sebastian Jeltsch
5705b99b6c
Audit and remove remaining unwraps.
2025-03-09 00:04:09 +01:00
Sebastian Jeltsch
527974284f
Update Rust and JS deps.
2025-02-27 00:59:20 +01:00
Sebastian Jeltsch
c06ceee31b
Control expansion of FKs in RecordApi.(read|list) via query parameters.
2025-02-13 13:47:10 +01:00
Sebastian Jeltsch
fd318cf328
Expand/inline foreign records in RecordApi list results.
2025-02-13 13:47:10 +01:00
Sebastian Jeltsch
757f0c370d
Expand/inline foreign records in RecordApi read.
2025-02-13 13:47:03 +01:00
Sebastian Jeltsch
eec17e1343
Update Rust dependencies.
2025-02-12 22:37:55 +01:00
Sebastian Jeltsch
031e2a5032
Update Rust and JS deps.
2025-02-02 12:26:02 +01:00
Sebastian Jeltsch
31e8421451
Updating and cleaning up some Rust deps.
2025-01-26 21:29:44 +01:00
Sebastian Jeltsch
55c26afb0f
Prepare release v0.5.1 and publish new crates.
2025-01-20 15:27:13 +01:00
Sebastian Jeltsch
5144dabc68
Major version update: SQLite 3.46.1 to 3.48.0.
2025-01-19 17:39:30 +01:00
Sebastian Jeltsch
2fb294f17c
Breaking: nest records in output of RecordAPI.list to contain cursor now and potentially more in the future.
...
Also update all the client libraries to accept the new format.
2025-01-19 15:40:09 +01:00
Sebastian Jeltsch
1828ebad5a
Optimization: reduce allocations, cloning and re-encoding.
2025-01-13 22:40:27 +01:00
Sebastian Jeltsch
30f295e6fd
Add "realtime" subscriptions for a specific record, i.e. updates and deletion.
2025-01-13 22:38:14 +01:00
Sebastian Jeltsch
c9199edd22
Prepare trailbase-sqlite for publishing.
2024-12-30 11:52:02 +01:00
