TimeTracker/docs/ENHANCED_DATABASE_STARTUP.md

Enhanced Database Startup Procedure

This document describes the improved database initialization and startup procedure for TimeTracker that ensures all tables are correctly created with proper schema and handles migrations automatically.

Overview

The enhanced startup procedure addresses several issues found in the previous implementation:

1. Missing Analytics Table: The analytics functionality exists but there was no dedicated analytics table
2. Schema Verification: Previous scripts didn't verify that all required columns exist in existing tables
3. Migration Handling: Analytics setting migration wasn't integrated into the main startup process
4. Table Recreation Logic: Previous logic might drop and recreate tables unnecessarily

New Components

1. Enhanced Database Initialization Script (init-database-enhanced.py)

This script provides comprehensive database setup:

• Schema Definition: Defines the complete expected database schema including all tables, columns, and relationships
• Smart Table Creation: Creates tables only if they don't exist, or adds missing columns if they do exist
• Index Management: Creates performance indexes for all tables
• Trigger Setup: Sets up automatic timestamp updates for updated_at columns
• Data Initialization: Inserts default admin user, project, and settings
• Schema Verification: Verifies that all required tables and columns exist after initialization

Tables Created

• users - User accounts and authentication
• projects - Project definitions and client information
• time_entries - Time tracking records
• tasks - Task management within projects
• settings - Application configuration and company branding
• invoices - Invoice management
• invoice_items - Individual invoice line items

Key Features

• Non-destructive: Never drops existing tables or data
• Migration-friendly: Automatically adds missing columns to existing tables
• Comprehensive: Handles all aspects of database setup in one script
• Error-tolerant: Continues operation even if some optional features fail

2. Enhanced Startup Script (start-enhanced.py)

Simplified startup process that:

• Uses the enhanced database initialization script
• Provides better error handling and logging
• Displays network information for debugging
• Ensures proper startup sequence

3. Database Verification Script (verify-database.py)

Independent verification tool that:

• Checks all tables exist with correct schema
• Verifies indexes and foreign keys
• Reports any missing or incorrect elements
• Can be run independently to diagnose issues

Startup Sequence

1. Container Start: Docker container starts with the enhanced startup script
2. Database Wait: Waits for PostgreSQL database to be ready
3. Enhanced Initialization: Runs the comprehensive database setup script
4. Schema Verification: Verifies all tables and columns are correct
5. Application Start: Launches the Flask application with Gunicorn

Database Schema

Core Tables

users

id SERIAL PRIMARY KEY,
username VARCHAR(80) UNIQUE NOT NULL,
role VARCHAR(20) DEFAULT 'user' NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
last_login TIMESTAMP,
is_active BOOLEAN DEFAULT true NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL

projects

id SERIAL PRIMARY KEY,
name VARCHAR(200) NOT NULL,
client VARCHAR(200) NOT NULL,
description TEXT,
billable BOOLEAN DEFAULT true NOT NULL,
hourly_rate NUMERIC(9, 2),
billing_ref VARCHAR(100),
status VARCHAR(20) DEFAULT 'active' NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP

time_entries

id SERIAL PRIMARY KEY,
user_id INTEGER REFERENCES users(id) ON DELETE CASCADE,
project_id INTEGER REFERENCES projects(id) ON DELETE CASCADE,
task_id INTEGER REFERENCES tasks(id) ON DELETE SET NULL,
start_time TIMESTAMP NOT NULL,
end_time TIMESTAMP,
duration_seconds INTEGER,
notes TEXT,
tags VARCHAR(500),
source VARCHAR(20) DEFAULT 'manual' NOT NULL,
billable BOOLEAN DEFAULT true NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP

tasks

id SERIAL PRIMARY KEY,
project_id INTEGER REFERENCES projects(id) ON DELETE CASCADE NOT NULL,
name VARCHAR(200) NOT NULL,
description TEXT,
status VARCHAR(20) DEFAULT 'pending' NOT NULL,
priority VARCHAR(20) DEFAULT 'medium' NOT NULL,
assigned_to INTEGER REFERENCES users(id),
created_by INTEGER REFERENCES users(id) NOT NULL,
due_date DATE,
estimated_hours NUMERIC(5,2),
actual_hours NUMERIC(5,2),
started_at TIMESTAMP,
completed_at TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL

settings

id SERIAL PRIMARY KEY,
timezone VARCHAR(50) DEFAULT 'Europe/Rome' NOT NULL,
currency VARCHAR(3) DEFAULT 'EUR' NOT NULL,
rounding_minutes INTEGER DEFAULT 1 NOT NULL,
single_active_timer BOOLEAN DEFAULT true NOT NULL,
allow_self_register BOOLEAN DEFAULT true NOT NULL,
idle_timeout_minutes INTEGER DEFAULT 30 NOT NULL,
backup_retention_days INTEGER DEFAULT 30 NOT NULL,
backup_time VARCHAR(5) DEFAULT '02:00' NOT NULL,
export_delimiter VARCHAR(1) DEFAULT ',' NOT NULL,
allow_analytics BOOLEAN DEFAULT true NOT NULL,
-- Company branding fields...
company_name VARCHAR(200) DEFAULT 'Your Company Name' NOT NULL,
company_address TEXT DEFAULT 'Your Company Address' NOT NULL,
company_email VARCHAR(200) DEFAULT 'info@yourcompany.com' NOT NULL,
company_phone VARCHAR(50) DEFAULT '+1 (555) 123-4567' NOT NULL,
company_website VARCHAR(200) DEFAULT 'www.yourcompany.com' NOT NULL,
company_logo_filename VARCHAR(255) DEFAULT '' NOT NULL,
company_tax_id VARCHAR(100) DEFAULT '' NOT NULL,
company_bank_info TEXT DEFAULT '' NOT NULL,
-- Invoice defaults...
invoice_prefix VARCHAR(10) DEFAULT 'INV' NOT NULL,
invoice_start_number INTEGER DEFAULT 1000 NOT NULL,
invoice_terms TEXT DEFAULT 'Payment is due within 30 days of invoice date.' NOT NULL,
invoice_notes TEXT DEFAULT 'Thank you for your business!' NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL

invoices

id SERIAL PRIMARY KEY,
invoice_number VARCHAR(50) UNIQUE NOT NULL,
project_id INTEGER REFERENCES projects(id) ON DELETE CASCADE,
client_name VARCHAR(200) NOT NULL,
client_email VARCHAR(200),
client_address TEXT,
issue_date DATE NOT NULL,
due_date DATE NOT NULL,
status VARCHAR(20) DEFAULT 'draft' NOT NULL,
subtotal NUMERIC(10, 2) NOT NULL DEFAULT 0,
tax_rate NUMERIC(5, 2) NOT NULL DEFAULT 0,
tax_amount NUMERIC(10, 2) NOT NULL DEFAULT 0,
total_amount NUMERIC(10, 2) NOT NULL DEFAULT 0,
notes TEXT,
terms TEXT,
created_by INTEGER REFERENCES users(id) ON DELETE CASCADE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP

invoice_items

id SERIAL PRIMARY KEY,
invoice_id INTEGER REFERENCES invoices(id) ON DELETE CASCADE,
description VARCHAR(500) NOT NULL,
quantity NUMERIC(10, 2) NOT NULL DEFAULT 1,
unit_price NUMERIC(10, 2) NOT NULL,
total_amount NUMERIC(10, 2) NOT NULL,
time_entry_ids VARCHAR(500),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP

Performance Features

Indexes

• User authentication: username, role
• Project queries: client, status
• Time tracking: user_id, project_id, task_id, start_time, billable
• Task management: project_id, status, assigned_to, due_date
• Invoice management: project_id, status, issue_date

Triggers

• Automatic updated_at timestamp updates for all tables that have this column
• Ensures data consistency without application-level code

Migration Support

The enhanced initialization script automatically handles:

• New Tables: Creates any missing tables
• New Columns: Adds missing columns to existing tables
• Schema Updates: Ensures existing databases are updated to current schema
• Data Preservation: Never drops existing data

Verification

Startup Verification

• Automatic verification after initialization
• Reports any issues found
• Ensures application only starts with correct database schema

Manual Verification

# Run verification script independently
python docker/verify-database.py

# Check specific aspects
python docker/test-schema.py

Error Handling

The enhanced startup procedure includes comprehensive error handling:

• Database Connection: Waits for database to be ready with retry logic
• Schema Issues: Reports specific problems and continues where possible
• Non-critical Failures: Continues operation even if some optional features fail
• Detailed Logging: Provides clear information about what succeeded and what failed

Troubleshooting

Common Issues

1. Missing Tables: Run verify-database.py to identify missing tables
2. Schema Mismatches: The enhanced script automatically adds missing columns
3. Permission Issues: Ensure database user has CREATE, ALTER, and INSERT privileges
4. Connection Problems: Check DATABASE_URL environment variable and network connectivity

Debug Commands

# Check database schema
python docker/verify-database.py

# Test database connection
python docker/test-db.py

# View startup logs
docker logs <container_name>

Benefits

1. Reliability: Ensures consistent database schema across all deployments
2. Maintainability: Single script handles all database setup
3. Migration Safety: Never loses existing data during updates
4. Performance: Proper indexes and triggers for optimal operation
5. Debugging: Comprehensive verification and error reporting

Future Enhancements

• Version Tracking: Database schema version tracking for future migrations
• Rollback Support: Ability to rollback schema changes if needed
• Performance Monitoring: Database performance metrics collection
• Automated Testing: Integration with CI/CD pipeline for schema validation