Since Dolt v1.78.5, Dolt has truncated the journal file to the latest successfully loaded record when it successfully loads a database. This is the correct behavior when the CLI is operating as the exclusive writer to the database. However, Dolt also has a mode where it can load the database in read-only mode. Due to a bug, Dolt was also truncating the journal file when it was operating in this mode. The end result was the running something like `dolt sql -r csv -q ...` against a Dolt database that was running a sql-server and was currently accepting writes could incorrectly truncate the journal file. The Dolt sql-server process would go ahead writing into the journal at its previously extended offset and the space between the truncation and the next write offset would be 0 filled by the operating system. Attempting to load the database later or accessing the chunks located at that corrupted portion of the journal file would result in checksum errors or failure to load messages.
This change correctly updates Dolt to only Truncate the journal file when we are loading it in read-write mode.
Previously, early in the process life-cycle after dolt was run, dolt would immediately check if a file created in `os.TempDir()` was able to be `os.Rename`d into a subdirectory of the dolt process's data directory, by default `$PWD/.dolt`. If the rename failed, it would configure Dolt to use `.dolt/temptf` as the movable temp file directory instead.
This meant that for many `dolt` invocations, Dolt would do some local filesystem writes, including potentially `Mkdir(.dolt)` before it it was fully loaded. With the advent of things like dolt accessing the running server through sql-server.info and `dolt --host ... sql` this behavior was not ideal. It creates races with concurrently run `dolt` processes that try to use the `.dolt` directory, and it creates odd behavior when running something like `dolt sql` in a non-Dolt directory but on a filesystem mount where this rename from `os.TempDir()` does not work.
This PR changes the check and the configuration of a potential `.dolt/temptf` directory to be delayed until the first temporary movable file is actually requested by Dolt. At that point we know that we have a need for a renamable temp file and it is OK to go ahead and create `.dolt/temptf` if we need it.
This PR changes the error handling around some edge cases like permissions errors on the calling process creating `temptf` within `.dolt`. In particular, some errors which were previously early and fatal are now delayed until use site and may end up being persistent but non-fatal to the process, depending on the operation.
Online GC could write new tale files to oldgen, and the manifest write was not crash safe. This could result in data loss depending on filesystem settings if there was an inopportune crash.
Aaron Son