Files
TimeTracker/app/integrations/github.py
T
Dries Peeters 1836cb3c2d chore(typing): resolve mypy errors and harden type checking
Drives ``mypy app/`` from 567 errors in 208 files to 0 errors across the
376 source files checked by ``./scripts/run-ci-local.sh code-quality``.

Configuration & dependencies
- pyproject.toml: enable implicit_optional (Flask-style ``x: str = None``
  defaults), silence truthy-function/truthy-bool (legitimate import-guard
  checks like ``KanbanColumn``), and disable warn_return_any (SQLAlchemy
  1.x ``Query`` API returns Any pervasively). Add module overrides for
  ``app.models.*``, repositories, base CRUD service, and known
  ``joinedload`` / ``Query.paginate`` callers where mypy cannot model the
  Flask-SQLAlchemy runtime API without a plugin.
- requirements-test.txt: pin ``types-requests``, ``types-bleach``,
  ``types-Markdown``, ``types-python-dateutil`` so mypy stops complaining
  about missing stubs.

Latent bugs fixed while driving mypy to zero
- app/utils/logger.py, app/utils/datetime_utils.py: drop imports of
  symbols that don't exist (``get_performance_metrics``,
  ``from_app_timezone``, ``to_app_timezone``) — these would have raised
  at import time on first use.
- app/services/currency_service.py: ``from typing import Decimal`` was a
  bug (typing has no Decimal); switch to ``decimal.Decimal`` and rename
  the ``D`` alias.
- app/utils/env_validation.py, app/utils/role_migration.py: ``Dict[str,
  any]`` → ``Dict[str, Any]`` (built-in ``any`` is not a type).
- app/utils/email.py: introduce ``send_template_email`` and update the
  three callers (``client_approval_service``,
  ``client_notification_service``, ``workflow_engine``) that were
  passing ``to=``/``template=``/etc. to ``send_email`` whose signature
  doesn't accept them — calls would have raised TypeError at runtime.
- app/services/permission_service.py: rewrite ``grant_permission`` /
  ``revoke_permission`` to use the actual ``Role`` ↔ ``Permission``
  many-to-many relationship; the old code referenced non-existent
  ``Permission.role_id`` / ``Permission.granted`` columns.
- app/services/gps_tracking_service.py: pass the required ``title`` and
  ``expense_date`` fields when creating mileage ``Expense`` rows.
- app/services/workflow_engine.py: ``_perform_action`` now forwards the
  ``rule`` argument to ``_action_log_time``, and ``_action_webhook``
  short-circuits when ``url`` is missing.
- app/services/time_tracking_service.py: validate ``start_time`` /
  ``end_time`` before comparing them.
- app/services/export_service.py: build CSV in a ``StringIO`` then wrap
  the bytes in ``BytesIO`` — ``csv.writer`` requires text I/O.
- app/integrations/peppol_smp.py: avoid attribute access on ``None`` in
  the SMP ``href`` fallback.
- app/integrations/{github,gitlab,slack}.py: coerce query-string params
  to strings so ``requests.get(params=...)`` matches the typed signature
  (and is what the HTTP layer expects anyway).
- app/integrations/{xero,quickbooks}.py: guard ``get_access_token()``
  returning ``None`` before calling private ``_api_request`` helpers.

Annotation-only changes
- Add ``Dict[str, Any]`` / ``list`` / ``Optional[...]`` annotations to
  service dict-literals that mypy could not infer from heterogeneous
  values (``ai_suggestion_service``, ``ai_categorization_service``,
  ``custom_report_service``, ``unpaid_hours_service``,
  ``integration_service``, ``invoice_service``, ``backup_service``,
  ``inventory_report_service``, ``analytics_service``, etc.).
- ``app/utils/event_bus.py``: ``emit_event`` accepts ``str |
  WebhookEvent`` and normalizes to ``str`` so all call-sites type-check.
- ``app/utils/api_responses.py``: introduce ``ApiResponse`` alias for
  ``Response | tuple[Response, int] | tuple[str, int]``.
- ``app/utils/budget_forecasting.py``: forecasting helpers return
  ``Optional[Dict]`` (they already returned ``None`` when the project
  was missing).
- ``app/utils/pdf_generator_reportlab.py``: ``_normalize_color`` is
  ``Optional[str]``.
- ``app/utils/pdfa3.py``: remove invalid ``force_version=None`` retry
  call.
- Narrow ``type: ignore`` markers on optional-dependency fallbacks
  (``redis``, ``bleach``, ``markdown``, ``babel``,
  ``powerpoint_export``) and on the documented ``requests.Session``
  / ``RotatingFileHandler`` typeshed limitations.
2026-05-13 10:32:06 +02:00

589 lines
27 KiB
Python

"""
GitHub integration connector.
"""
import logging
import os
from datetime import datetime, timedelta
from typing import Any, Dict, Optional
import requests
from app.integrations.base import BaseConnector
logger = logging.getLogger(__name__)
class GitHubConnector(BaseConnector):
"""GitHub integration connector."""
display_name = "GitHub"
description = "Sync issues and track time from GitHub"
icon = "github"
@property
def provider_name(self) -> str:
return "github"
def get_authorization_url(self, redirect_uri: str, state: str = None) -> str:
"""Get GitHub OAuth authorization URL."""
from app.models import Settings
settings = Settings.get_settings()
creds = settings.get_integration_credentials("github")
client_id = creds.get("client_id") or os.getenv("GITHUB_CLIENT_ID")
if not client_id:
raise ValueError("GITHUB_CLIENT_ID not configured")
scopes = ["repo", "issues:read", "issues:write", "user:email"]
auth_url = "https://github.com/login/oauth/authorize"
params = {"client_id": client_id, "redirect_uri": redirect_uri, "scope": " ".join(scopes), "state": state or ""}
query_string = "&".join([f"{k}={v}" for k, v in params.items()])
return f"{auth_url}?{query_string}"
def exchange_code_for_tokens(self, code: str, redirect_uri: str) -> Dict[str, Any]:
"""Exchange authorization code for tokens."""
from app.models import Settings
settings = Settings.get_settings()
creds = settings.get_integration_credentials("github")
client_id = creds.get("client_id") or os.getenv("GITHUB_CLIENT_ID")
client_secret = creds.get("client_secret") or os.getenv("GITHUB_CLIENT_SECRET")
if not client_id or not client_secret:
raise ValueError("GitHub OAuth credentials not configured")
token_url = "https://github.com/login/oauth/access_token"
response = requests.post(
token_url,
data={"client_id": client_id, "client_secret": client_secret, "code": code, "redirect_uri": redirect_uri},
headers={"Accept": "application/json"},
)
response.raise_for_status()
data = response.json()
if "error" in data:
raise ValueError(f"GitHub OAuth error: {data.get('error_description', data.get('error'))}")
# GitHub tokens don't expire by default, but can be configured
expires_at = None
if "expires_in" in data:
expires_at = datetime.utcnow() + timedelta(seconds=data["expires_in"])
# Get user info
access_token = data.get("access_token")
user_info = {}
if access_token:
try:
user_response = requests.get(
"https://api.github.com/user",
headers={"Authorization": f"token {access_token}", "Accept": "application/vnd.github.v3+json"},
)
if user_response.status_code == 200:
user_info = user_response.json()
except Exception as e:
logger.debug("GitHub user fetch failed: %s", e)
return {
"access_token": access_token,
"refresh_token": data.get("refresh_token"), # GitHub doesn't provide refresh tokens by default
"expires_at": expires_at,
"token_type": data.get("token_type", "Bearer"),
"scope": data.get("scope"),
"extra_data": {
"user_login": user_info.get("login"),
"user_name": user_info.get("name"),
"user_email": user_info.get("email"),
},
}
def refresh_access_token(self) -> Dict[str, Any]:
"""Refresh access token (GitHub tokens typically don't expire)."""
# GitHub tokens don't expire by default
# If using GitHub Apps, refresh would be handled differently
if not self.credentials or not self.credentials.access_token:
raise ValueError("No access token available")
# For now, just return the existing token
# In production, implement proper refresh if using GitHub Apps
return {
"access_token": self.credentials.access_token,
"refresh_token": self.credentials.refresh_token,
"expires_at": self.credentials.expires_at,
}
def test_connection(self) -> Dict[str, Any]:
"""Test connection to GitHub."""
token = self.get_access_token()
if not token:
return {"success": False, "message": "No access token available"}
api_url = "https://api.github.com/user"
try:
response = requests.get(
api_url, headers={"Authorization": f"token {token}", "Accept": "application/vnd.github.v3+json"}
)
if response.status_code == 200:
user_data = response.json()
return {"success": True, "message": f"Connected as {user_data.get('login', 'Unknown')}"}
else:
return {"success": False, "message": f"API returned status {response.status_code}"}
except Exception as e:
return {"success": False, "message": f"Connection error: {str(e)}"}
def sync_data(self, sync_type: str = "full") -> Dict[str, Any]:
"""Sync issues from GitHub repositories and create tasks."""
import logging
from datetime import datetime, timedelta
from app import db
from app.models import Project, Task
from app.utils.integration_sync_context import (
ensure_project_integration_fields,
find_project_by_integration_ref,
find_task_by_integration_ref,
require_sync_context,
set_task_integration_ref,
)
logger = logging.getLogger(__name__)
token = self.get_access_token()
if not token:
return {"success": False, "message": "No access token available. Please reconnect the integration."}
try:
actor_id, client_id = require_sync_context(self.integration)
except ValueError as e:
return {"success": False, "message": str(e)}
# Get repositories from config
repos_str = self.integration.config.get("repositories", "")
if not repos_str:
# Get user's repositories
try:
repos_response = requests.get(
"https://api.github.com/user/repos",
headers={"Authorization": f"token {token}", "Accept": "application/vnd.github.v3+json"},
timeout=30,
)
if repos_response.status_code == 200:
repos = repos_response.json()
repos_list = [f"{r['owner']['login']}/{r['name']}" for r in repos[:10]] # Limit to 10 repos
elif repos_response.status_code == 401:
return {
"success": False,
"message": "GitHub authentication failed. Please reconnect the integration.",
}
else:
error_msg = (
f"Could not fetch repositories: {repos_response.status_code} - {repos_response.text[:200]}"
)
logger.error(error_msg)
return {"success": False, "message": error_msg}
except requests.exceptions.Timeout:
return {"success": False, "message": "GitHub API request timed out. Please try again."}
except requests.exceptions.ConnectionError as e:
return {"success": False, "message": f"Failed to connect to GitHub API: {str(e)}"}
except Exception as e:
logger.error(f"Error fetching repositories: {e}", exc_info=True)
return {"success": False, "message": f"Error fetching repositories: {str(e)}"}
else:
repos_list = [r.strip() for r in repos_str.split(",") if r.strip()]
if not repos_list:
return {"success": False, "message": "No repositories configured or found"}
synced_count = 0
errors = []
try:
for repo in repos_list:
try:
if "/" not in repo:
errors.append(f"Invalid repository format: {repo} (expected owner/repo)")
continue
owner, repo_name = repo.split("/", 1)
# Find or create project (client + custom_fields integration marker)
project = find_project_by_integration_ref(client_id, "github", repo)
if not project:
project = Project.query.filter_by(client_id=client_id, name=repo).first()
if not project:
try:
project = Project(
name=repo,
client_id=client_id,
description=f"GitHub repository: {repo}",
status="active",
)
db.session.add(project)
db.session.flush()
except Exception as e:
errors.append(f"Error creating project for {repo}: {str(e)}")
logger.error(f"Error creating project for {repo}: {e}", exc_info=True)
continue
ensure_project_integration_fields(
project,
source="github",
ref=repo,
display_name=repo,
description=f"GitHub repository: {repo}",
)
# Fetch issues
try:
issues_response = requests.get(
f"https://api.github.com/repos/{repo}/issues",
headers={"Authorization": f"token {token}", "Accept": "application/vnd.github.v3+json"},
params={"state": "open", "per_page": "100"},
timeout=30,
)
if issues_response.status_code == 404:
errors.append(f"Repository {repo} not found or access denied")
continue
elif issues_response.status_code == 401:
errors.append(f"Authentication failed for repository {repo}")
continue
elif issues_response.status_code != 200:
error_text = issues_response.text[:200] if issues_response.text else ""
errors.append(
f"Error fetching issues for {repo}: {issues_response.status_code} - {error_text}"
)
continue
issues = issues_response.json()
except requests.exceptions.Timeout:
errors.append(f"Timeout fetching issues for {repo}")
continue
except requests.exceptions.ConnectionError as e:
errors.append(f"Connection error for {repo}: {str(e)}")
continue
except Exception as e:
errors.append(f"Error fetching issues for {repo}: {str(e)}")
logger.error(f"Error fetching issues for {repo}: {e}", exc_info=True)
continue
for issue in issues:
try:
if issue.get("pull_request"):
continue
issue_number = issue.get("number")
issue_title = (issue.get("title") or "").strip() or "Issue"
issue_title = issue_title[:180]
if not issue_number:
continue
issue_ref = f"{repo}#{issue_number}"
body = (issue.get("body") or "").strip()
url = issue.get("html_url") or ""
if url:
body = f"{body}\n\nGitHub: {url}" if body else f"GitHub: {url}"
gh_state = (issue.get("state") or "").lower()
task_status = "done" if gh_state == "closed" else "todo"
task = find_task_by_integration_ref(project.id, issue_ref, source="github")
if not task:
try:
task_name = f"#{issue_number}: {issue_title}"[:200]
task = Task(
project_id=project.id,
name=task_name,
description=body or None,
status=task_status,
created_by=actor_id,
)
db.session.add(task)
db.session.flush()
except Exception as e:
errors.append(f"Error creating task for issue #{issue_number} in {repo}: {str(e)}")
logger.error(
f"Error creating task for issue #{issue_number} in {repo}: {e}", exc_info=True
)
continue
else:
task.name = f"#{issue_number}: {issue_title}"[:200]
task.description = body or None
task.status = task_status
set_task_integration_ref(
task,
source="github",
ref=issue_ref,
extra={
"issue_number": issue_number,
"issue_id": issue.get("id"),
"url": url,
"repo": repo,
},
)
synced_count += 1
except Exception as e:
errors.append(f"Error syncing issue #{issue.get('number', 'unknown')} in {repo}: {str(e)}")
logger.error(
f"Error syncing issue #{issue.get('number', 'unknown')} in {repo}: {e}", exc_info=True
)
except ValueError as e:
errors.append(f"Invalid repository format: {repo} - {str(e)}")
except Exception as e:
errors.append(f"Error syncing repository {repo}: {str(e)}")
logger.error(f"Error syncing repository {repo}: {e}", exc_info=True)
try:
db.session.commit()
except Exception as e:
db.session.rollback()
error_msg = f"Database error during sync: {str(e)}"
errors.append(error_msg)
logger.error(error_msg, exc_info=True)
return {"success": False, "message": error_msg, "synced_items": synced_count, "errors": errors}
if errors:
return {
"success": True,
"message": f"Sync completed with {len(errors)} error(s). Synced {synced_count} issues.",
"synced_items": synced_count,
"errors": errors,
}
return {
"success": True,
"message": f"Sync completed. Synced {synced_count} issues.",
"synced_items": synced_count,
"errors": errors,
}
except Exception as e:
logger.error(f"GitHub sync failed: {e}", exc_info=True)
try:
db.session.rollback()
except Exception as rollback_err:
logger.debug("Rollback after GitHub sync failure: %s", rollback_err)
return {
"success": False,
"message": f"Sync failed: {str(e)}",
"errors": errors,
"synced_items": synced_count,
}
def handle_webhook(
self, payload: Dict[str, Any], headers: Dict[str, str], raw_body: Optional[bytes] = None
) -> Dict[str, Any]:
"""Handle incoming webhook from GitHub."""
import hashlib
import hmac
import logging
logger = logging.getLogger(__name__)
try:
# Verify webhook signature if secret is configured
signature = headers.get("X-Hub-Signature-256", "")
if signature:
# Get webhook secret from integration config
webhook_secret = self.integration.config.get("webhook_secret") if self.integration else None
if webhook_secret:
# GitHub sends signature as "sha256=<hash>"
if not signature.startswith("sha256="):
logger.warning("GitHub webhook signature format invalid (expected sha256= prefix)")
return {"success": False, "message": "Invalid webhook signature format"}
signature_hash = signature[7:] # Remove "sha256=" prefix
# GitHub signs the raw request body bytes, not the parsed JSON
# This is critical for signature verification to work correctly
if raw_body is None:
# Fallback: try to reconstruct from payload (not ideal but better than nothing)
import json
raw_body = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
logger.warning(
"GitHub webhook: Using reconstructed payload for signature verification (raw body not available)"
)
# Compute expected signature using raw body bytes
expected_signature = hmac.new(webhook_secret.encode("utf-8"), raw_body, hashlib.sha256).hexdigest()
# Use constant-time comparison to prevent timing attacks
if not hmac.compare_digest(signature_hash, expected_signature):
logger.warning("GitHub webhook signature verification failed")
return {"success": False, "message": "Webhook signature verification failed"}
logger.debug("GitHub webhook signature verified successfully")
else:
# Signature provided but no secret configured - reject for security
logger.warning("GitHub webhook signature provided but no secret configured - rejecting webhook")
return {"success": False, "message": "Webhook secret not configured"}
else:
# No signature: always reject (configure secret on GitHub + matching webhook_secret here)
webhook_secret = self.integration.config.get("webhook_secret") if self.integration else None
if webhook_secret:
logger.warning("GitHub webhook secret configured but no signature provided - rejecting webhook")
return {"success": False, "message": "Webhook signature required but not provided"}
logger.warning(
"GitHub webhook rejected: missing X-Hub-Signature-256. "
"Set a secret on the GitHub webhook and store it in integration config as webhook_secret."
)
return {
"success": False,
"message": "Webhook signature required; configure webhook_secret on GitHub and in TimeTracker.",
}
# Process webhook event
action = payload.get("action")
event_type = headers.get("X-GitHub-Event", "")
if event_type == "issues":
issue = payload.get("issue", {})
issue_number = issue.get("number")
repo = payload.get("repository", {}).get("full_name", "")
return {
"success": True,
"message": f"Webhook received for issue #{issue_number} in {repo}",
"event_type": f"{event_type}.{action}",
}
elif event_type == "pull_request":
pr = payload.get("pull_request", {})
pr_number = pr.get("number")
repo = payload.get("repository", {}).get("full_name", "")
return {
"success": True,
"message": f"Webhook received for PR #{pr_number} in {repo}",
"event_type": f"{event_type}.{action}",
}
return {"success": True, "message": f"Webhook processed: {event_type}"}
except ValueError as e:
# Handle validation errors
logger.error(f"GitHub webhook validation error: {e}")
return {"success": False, "message": f"Webhook validation error: {str(e)}"}
except Exception as e:
# Handle all other errors
logger.error(f"GitHub webhook processing error: {e}", exc_info=True)
return {"success": False, "message": f"Error processing webhook: {str(e)}"}
def get_config_schema(self) -> Dict[str, Any]:
"""Get configuration schema."""
return {
"fields": [
{
"name": "repositories",
"label": "Repositories",
"type": "text",
"required": False,
"placeholder": "owner/repo1, owner/repo2",
"help": "Comma-separated list of repositories to sync (e.g., 'octocat/Hello-World, owner/repo'). Leave empty to sync all accessible repositories.",
"description": "Which GitHub repositories to sync",
},
{
"name": "sync_direction",
"type": "select",
"label": "Sync Direction",
"options": [
{"value": "github_to_timetracker", "label": "GitHub → TimeTracker (Import only)"},
{"value": "timetracker_to_github", "label": "TimeTracker → GitHub (Export only)"},
{"value": "bidirectional", "label": "Bidirectional (Two-way sync)"},
],
"default": "github_to_timetracker",
"description": "Choose how data flows between GitHub and TimeTracker",
},
{
"name": "sync_items",
"type": "array",
"label": "Items to Sync",
"options": [
{"value": "issues", "label": "Issues"},
{"value": "pull_requests", "label": "Pull Requests"},
{"value": "projects", "label": "Projects (Repositories)"},
],
"default": ["issues"],
"description": "Select which items to synchronize",
},
{
"name": "issue_states",
"type": "array",
"label": "Issue States to Sync",
"options": [
{"value": "open", "label": "Open Issues"},
{"value": "closed", "label": "Closed Issues"},
{"value": "all", "label": "All Issues"},
],
"default": ["open"],
"description": "Which issue states to include in sync",
},
{
"name": "auto_sync",
"type": "boolean",
"label": "Auto Sync",
"default": False,
"description": "Automatically sync when webhooks are received from GitHub",
},
{
"name": "sync_interval",
"type": "select",
"label": "Sync Schedule",
"options": [
{"value": "manual", "label": "Manual only"},
{"value": "hourly", "label": "Every hour"},
{"value": "daily", "label": "Daily"},
{"value": "weekly", "label": "Weekly"},
],
"default": "manual",
"description": "How often to automatically sync data",
},
{
"name": "create_projects",
"type": "boolean",
"label": "Create Projects",
"default": True,
"description": "Automatically create projects in TimeTracker from GitHub repositories",
},
{
"name": "webhook_secret",
"label": "Webhook Secret",
"type": "password",
"required": False,
"placeholder": "Enter webhook secret from GitHub",
"help": "Secret token for verifying webhook signatures. Configure this in your GitHub repository webhook settings.",
"description": "Security token for webhook verification",
},
],
"required": [],
"sections": [
{
"title": "Repository Settings",
"description": "Configure which repositories to sync",
"fields": ["repositories", "create_projects"],
},
{
"title": "Sync Settings",
"description": "Configure what and how to sync",
"fields": ["sync_direction", "sync_items", "issue_states", "auto_sync", "sync_interval"],
},
{
"title": "Webhook Settings",
"description": "Configure webhook security",
"fields": ["webhook_secret"],
},
],
"sync_settings": {
"enabled": True,
"auto_sync": False,
"sync_interval": "manual",
"sync_direction": "github_to_timetracker",
"sync_items": ["issues"],
},
}