This change enables the `dolt admin conjoin` command to work with archive
files (.darc) as targets, in addition to regular table files.
Key changes:
- Enhanced findTableSpec() in store.go to detect archive files by attempting
to open them through the persister
- Modified ConjoinAll() in file_table_persister.go to detect when all sources
are archive chunk sources and branch to archive-specific logic
- Added conjoinArchives() method that leverages the clean conjoinAll() method
from archive_writer.go
- Added comprehensive test coverage in admin-conjoin.bats
The implementation follows the principle of using the well-factored archive
conjoin logic rather than exposing archive internals through the messy table
file conjoin approach.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added precise chunk count verification and comprehensive iteration testing:
- Replaced approximate assertions with exact chunk count verification
- Added iteration testing to verify each chunk ref is visited exactly once
- Verify duplicate chunks appear the correct number of times during iteration
- Ensure total iteration count equals chunk count for complete validation
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added TestArchiveConjoinAllComprehensive that tests complex scenarios:
- 10 initial archives with mixed compression and chunk sizes (10-250 bytes)
- Shared chunks duplicated across archives with mixed Snappy/zStd compression
- Nested conjoin operations (first conjoin 10 archives, then conjoin result with 3 more)
- Single validation loop testing all expected chunks and hashes
- Proper buffer sizing to handle large amounts of data
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhanced createTestArchiveWithHashes to support mixed compression types
and consolidated duplicate archive creation code. Fixed variable shadowing
issue where 'chunks' parameter was shadowing the chunks package name.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Refactor createTestArchive to delegate to createTestArchiveWithHashes
- Remove ~35 lines of duplicate archive creation code
- Maintain same functionality with cleaner test helper hierarchy
- All tests continue to pass
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add reconstructHashFromPrefixAndSuffix helper function in archive_reader.go
- Update archive_writer.go to use the helper function
- Remove duplicate hash reconstruction logic while maintaining functionality
- All tests continue to pass
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Update conjoinAll to allow duplicate chunks instead of erroring
- Each chunk gets its own index entry pointing to its actual data location
- Enhanced TestArchiveConjoinAllDuplicateChunk with more comprehensive test cases
- Validate that duplicate chunks appear twice in the index as expected
- Maintain performance by writing entire data blocks from each archive
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add TestArchiveConjoinAllDuplicateChunk test that expects successful deduplication
- Create createTestArchiveWithHashes helper for precise hash control
- Create createArchiveWithDuplicates helper for mixed duplicate/unique scenarios
- Test verifies combined archive contains all expected chunks after deduplication
- Currently fails as expected since conjoinAll errors on duplicates
- Test is correctly structured for future implementation of duplicate handling
The helper functions will enable easy expansion to 10+ archives with complex duplicate patterns.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Change conjoinAll return type from (archiveReader, error) to just error
- Replace indexFinalizeFlushArchive with indexFinalize for in-memory completion
- Remove file persistence logic to make method suitable for unit testing
- Update test to create archive reader from in-memory bytes using provided pattern
- Add documentation clarifying the method completes archive writing in memory only
This makes conjoinAll more appropriate for unit tests that don't need disk materialization.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add full implementation of conjoinAll method that merges multiple archive readers
- Includes context parameter, data span sorting, and memory-efficient io.Copy
- Uses indexFinalizeFlushArchive for proper archive finalization and file persistence
- Fix createTestArchive to properly compress chunk data before writing byte spans
- Remove redundant readerAtAdapter type in favor of existing readerAtWithStatsBridge
- All tests pass including new TestArchiveConjoinAll validation
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added conjoinAll method to archiveWriter that takes multiple archiveReader instances
and combines them into a single archive. The method currently returns a stub
implementation with proper validation and error handling.
Added comprehensive test that validates the expected behavior:
- Creates two test archives with different chunk prefixes
- Tests that combined reader contains all chunks from both archives
- Verifies data integrity is maintained through the conjunction process
- Includes helper function to reduce test code duplication
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Neil Macneale IV