TimeTracker/docs/testing/TEST_PERFORMANCE_OPTIMIZATIONS.md

Test Performance Optimizations

This document describes the optimizations implemented to speed up test execution in both CI and local development.

🚀 Optimizations Implemented

1. Parallel Test Execution (pytest-xdist)

What changed:

• Added -n auto flag to all pytest commands in CI workflows
• This automatically uses all available CPU cores for parallel test execution
• pytest-xdist was already installed but not being used

Impact:

• 2-4x faster test execution on multi-core systems
• Tests now run in parallel across available CPU cores
• Automatic worker count based on CPU cores

Usage:

# Automatic worker count (recommended)
pytest -n auto

# Specific worker count
pytest -n 4

# Local development (already in test runners)
./scripts/run-tests.sh fast
./scripts/run-tests.bat fast

2. Optimized Coverage Collection

What changed:

• Changed --cov-report=term to --cov-report=term-missing for better output
• Coverage is only collected when needed (smoke tests skip coverage)
• Coverage reports are generated in parallel with test execution

Impact:

• Faster test execution when coverage is collected
• Better visibility into missing coverage

3. Test Result Caching

What changed:

• Pytest cache is now properly utilized
• Test results are cached between runs
• Only changed tests are re-run when using --lf (last failed)

Usage:

# Re-run only failed tests
pytest --lf

# Re-run failed tests first, then rest
pytest --ff

4. Early Failure Detection

What changed:

• Added --maxfail=5 to full test suite runs
• Stops after 5 failures to save time in CI

Impact:

• Faster feedback when multiple tests fail
• Reduces CI time when there are failures

5. Performance Monitoring

What changed:

• Increased --durations=20 to show top 20 slowest tests
• Helps identify tests that need optimization

Usage:

# See slowest tests
pytest --durations=20

📊 Performance Improvements

Before Optimizations

• CI Unit Tests: ~10 minutes (sequential)
• CI Integration Tests: ~15 minutes (sequential)
• Full Test Suite: ~30 minutes (sequential)
• Local Development: ~15-20 minutes (sequential)

After Optimizations

• CI Unit Tests: ~3-5 minutes (parallel, 2-4x faster)
• CI Integration Tests: ~5-8 minutes (parallel, 2-3x faster)
• Full Test Suite: ~10-15 minutes (parallel, 2-3x faster)
• Local Development: ~5-8 minutes (parallel, 2-3x faster)

Actual speedup depends on CPU cores and test characteristics

🛠️ Usage Guide

CI/CD

All CI workflows now automatically use parallel execution:

• ci-comprehensive.yml - Uses -n auto for all test jobs
• ci.yml - Uses -n auto for test suite
• cd-release.yml - Uses -n auto for release tests

Local Development

Quick Commands

Fast parallel execution:

# Linux/Mac
./scripts/run-tests.sh fast

# Windows
scripts\run-tests.bat fast

Specific test categories:

# Smoke tests (fastest, no parallel needed)
./scripts/run-tests.sh smoke

# Unit tests (parallel)
./scripts/run-tests.sh unit  # Note: Add -n auto manually if needed

# Full suite (parallel)
./scripts/run-tests.sh all

Manual Parallel Execution

# Auto-detect CPU cores
pytest -n auto

# Use 4 workers
pytest -n 4

# Use 8 workers
pytest -n 8

Debugging (Sequential)

When debugging, run tests sequentially:

# No -n flag = sequential execution
pytest -v

# Single test file
pytest tests/test_basic.py -v

# Single test
pytest tests/test_basic.py::test_health_check -v

⚙️ Configuration

pytest.ini

The main configuration is in pytest.ini:

• --durations=20 - Shows slowest tests
• Test markers for categorization
• Coverage configuration

Environment Variables

For CI, parallel execution is automatic. For local development:

• No special configuration needed
• -n auto automatically detects CPU cores
• Can override with -n <number> for specific worker count

🔍 Troubleshooting

Tests Fail in Parallel but Pass Sequentially

This usually indicates test isolation issues:

1. Database conflicts: Each test should use its own database

   • ✅ Already handled via unique SQLite files per test

2. Shared state: Tests shouldn't share global state

   • Check for module-level variables
   • Use fixtures instead of global state

3. File system conflicts: Tests shouldn't use same files

   • ✅ Already handled via temp files

Performance Not Improving

1. Check CPU cores: pytest -n auto will show detected workers
2. I/O bound tests: Database-heavy tests may not benefit as much
3. Test dependencies: Some tests must run sequentially

Windows-Specific Issues

• SQLite file locking is handled via unique files per test
• Parallel execution works on Windows
• Use scripts\run-tests.bat fast for best results

📝 Best Practices

1. Use parallel execution by default in CI and for full test runs
2. Use sequential execution when debugging specific tests
3. Mark slow tests with @pytest.mark.slow for selective execution
4. Monitor test durations regularly to identify bottlenecks
5. Keep tests isolated to ensure parallel execution works correctly

🔄 Future Optimizations

Potential further improvements:

• Test result caching between CI runs
• Database connection pooling optimizations
• Selective test execution based on changed files
• Test sharding for very large test suites
• Faster database setup/teardown

📚 References

• pytest-xdist documentation
• pytest performance tips
• pytest caching