mirror of
https://github.com/Freika/dawarich.git
synced 2026-01-01 02:39:44 -06:00
* fix: move foreman to global gems to fix startup crash (#1971) * Update exporting code to stream points data to file in batches to red… (#1980) * Update exporting code to stream points data to file in batches to reduce memory usage * Update changelog * Update changelog * Feature/maplibre frontend (#1953) * Add a plan to use MapLibre GL JS for the frontend map rendering, replacing Leaflet * Implement phase 1 * Phases 1-3 + part of 4 * Fix e2e tests * Phase 6 * Implement fog of war * Phase 7 * Next step: fix specs, phase 7 done * Use our own map tiles * Extract v2 map logic to separate manager classes * Update settings panel on v2 map * Update v2 e2e tests structure * Reimplement location search in maps v2 * Update speed routes * Implement visits and places creation in v2 * Fix last failing test * Implement visits merging * Fix a routes e2e test and simplify the routes layer styling. * Extract js to modules from maps_v2_controller.js * Implement area creation * Fix spec problem * Fix some e2e tests * Implement live mode in v2 map * Update icons and panel * Extract some styles * Remove unused file * Start adding dark theme to popups on MapLibre maps * Make popups respect dark theme * Move v2 maps to maplibre namespace * Update v2 references to maplibre * Put place, area and visit info into side panel * Update API to use safe settings config method * Fix specs * Fix method name to config in SafeSettings and update usages accordingly * Add missing public files * Add handling for real time points * Fix remembering enabled/disabled layers of the v2 map * Fix lots of e2e tests * Add settings to select map version * Use maps/v2 as main path for MapLibre maps * Update routing * Update live mode * Update maplibre controller * Update changelog * Remove some console.log statements * Pull only necessary data for map v2 points * Feature/raw data archive (#2009) * 0.36.2 (#2007) * fix: move foreman to global gems to fix startup crash (#1971) * Update exporting code to stream points data to file in batches to red… (#1980) * Update exporting code to stream points data to file in batches to reduce memory usage * Update changelog * Update changelog * Feature/maplibre frontend (#1953) * Add a plan to use MapLibre GL JS for the frontend map rendering, replacing Leaflet * Implement phase 1 * Phases 1-3 + part of 4 * Fix e2e tests * Phase 6 * Implement fog of war * Phase 7 * Next step: fix specs, phase 7 done * Use our own map tiles * Extract v2 map logic to separate manager classes * Update settings panel on v2 map * Update v2 e2e tests structure * Reimplement location search in maps v2 * Update speed routes * Implement visits and places creation in v2 * Fix last failing test * Implement visits merging * Fix a routes e2e test and simplify the routes layer styling. * Extract js to modules from maps_v2_controller.js * Implement area creation * Fix spec problem * Fix some e2e tests * Implement live mode in v2 map * Update icons and panel * Extract some styles * Remove unused file * Start adding dark theme to popups on MapLibre maps * Make popups respect dark theme * Move v2 maps to maplibre namespace * Update v2 references to maplibre * Put place, area and visit info into side panel * Update API to use safe settings config method * Fix specs * Fix method name to config in SafeSettings and update usages accordingly * Add missing public files * Add handling for real time points * Fix remembering enabled/disabled layers of the v2 map * Fix lots of e2e tests * Add settings to select map version * Use maps/v2 as main path for MapLibre maps * Update routing * Update live mode * Update maplibre controller * Update changelog * Remove some console.log statements --------- Co-authored-by: Robin Tuszik <mail@robin.gg> * Remove esbuild scripts from package.json * Remove sideEffects field from package.json * Raw data archivation * Add tests * Fix tests * Fix tests * Update ExceptionReporter * Add schedule to run raw data archival job monthly * Change file structure for raw data archival feature * Update changelog and version for raw data archival feature --------- Co-authored-by: Robin Tuszik <mail@robin.gg> * Set raw_data to an empty hash instead of nil when archiving * Fix storage configuration and file extraction * Consider MIN_MINUTES_SPENT_IN_CITY during stats calculation (#2018) * Consider MIN_MINUTES_SPENT_IN_CITY during stats calculation * Remove raw data from visited cities api endpoint * Use user timezone to show dates on maps (#2020) * Fix/pre epoch time (#2019) * Use user timezone to show dates on maps * Limit timestamps to valid range to prevent database errors when users enter pre-epoch dates. * Limit timestamps to valid range to prevent database errors when users enter pre-epoch dates. * Fix tests failing due to new index on stats table * Fix failing specs * Update redis client configuration to support unix socket connection * Update changelog * Fix kml kmz import issues (#2023) * Fix kml kmz import issues * Refactor KML importer to improve readability and maintainability * Implement moving points in map v2 and fix route rendering logic to ma… (#2027) * Implement moving points in map v2 and fix route rendering logic to match map v1. * Fix route spec * fix(maplibre): update date format to ISO 8601 (#2029) * Add verification step to raw data archival process (#2028) * Add verification step to raw data archival process * Add actual verification of raw data archives after creation, and only clear raw_data for verified archives. * Fix failing specs * Eliminate zip-bomb risk * Fix potential memory leak in js * Return .keep files * Use Toast instead of alert for notifications * Add help section to navbar dropdown * Update changelog * Remove raw_data_archival_job * Ensure file is being closed properly after reading in Archivable concern * Add composite index to stats table if not exists * Update changelog * Update entrypoint to always sync static assets (not only new ones) * Add family layer to MapLibre maps (#2055) * Add family layer to MapLibre maps * Update migration * Don't show family toggle if feature is disabled * Update changelog * Return changelog * Update changelog * Update tailwind file * Bump sentry-rails from 6.0.0 to 6.1.0 (#1945) Bumps [sentry-rails](https://github.com/getsentry/sentry-ruby) from 6.0.0 to 6.1.0. - [Release notes](https://github.com/getsentry/sentry-ruby/releases) - [Changelog](https://github.com/getsentry/sentry-ruby/blob/master/CHANGELOG.md) - [Commits](https://github.com/getsentry/sentry-ruby/compare/6.0.0...6.1.0) --- updated-dependencies: - dependency-name: sentry-rails dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump turbo-rails from 2.0.17 to 2.0.20 (#1944) Bumps [turbo-rails](https://github.com/hotwired/turbo-rails) from 2.0.17 to 2.0.20. - [Release notes](https://github.com/hotwired/turbo-rails/releases) - [Commits](https://github.com/hotwired/turbo-rails/compare/v2.0.17...v2.0.20) --- updated-dependencies: - dependency-name: turbo-rails dependency-version: 2.0.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Evgenii Burmakin <Freika@users.noreply.github.com> * Bump webmock from 3.25.1 to 3.26.1 (#1943) Bumps [webmock](https://github.com/bblimke/webmock) from 3.25.1 to 3.26.1. - [Release notes](https://github.com/bblimke/webmock/releases) - [Changelog](https://github.com/bblimke/webmock/blob/master/CHANGELOG.md) - [Commits](https://github.com/bblimke/webmock/compare/v3.25.1...v3.26.1) --- updated-dependencies: - dependency-name: webmock dependency-version: 3.26.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Evgenii Burmakin <Freika@users.noreply.github.com> * Bump brakeman from 7.1.0 to 7.1.1 (#1942) Bumps [brakeman](https://github.com/presidentbeef/brakeman) from 7.1.0 to 7.1.1. - [Release notes](https://github.com/presidentbeef/brakeman/releases) - [Changelog](https://github.com/presidentbeef/brakeman/blob/main/CHANGES.md) - [Commits](https://github.com/presidentbeef/brakeman/compare/v7.1.0...v7.1.1) --- updated-dependencies: - dependency-name: brakeman dependency-version: 7.1.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump redis from 5.4.0 to 5.4.1 (#1941) Bumps [redis](https://github.com/redis/redis-rb) from 5.4.0 to 5.4.1. - [Changelog](https://github.com/redis/redis-rb/blob/master/CHANGELOG.md) - [Commits](https://github.com/redis/redis-rb/compare/v5.4.0...v5.4.1) --- updated-dependencies: - dependency-name: redis dependency-version: 5.4.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Put import deletion into background job (#2045) * Put import deletion into background job * Update changelog * fix null type error and update heatmap styling (#2037) * fix: use constant weight for maplibre heatmap layer * fix null type, update heatmap styling * improve heatmap styling * fix typo * Fix stats calculation to recursively reduce H3 resolution when too ma… (#2065) * Fix stats calculation to recursively reduce H3 resolution when too many hexagons are generated * Update CHANGELOG.md * Validate trip start and end dates (#2066) * Validate trip start and end dates * Update changelog * Update migration to clean up duplicate stats before adding unique index * Fix fog of war radius setting being ignored and applying settings causing errors (#2068) * Update changelog * Add Rack::Deflater middleware to config/application.rb to enable gzip compression for responses. * Add composite index to points on user_id and timestamp * Deduplicte points based on timestamp brought to unix time * Fix/stats cache invalidation (#2072) * Fix family layer toggle in Map v2 settings for non-selfhosted env * Invalidate cache * Remove comments * Remove comment * Add new indicies to improve performance and remove unused ones to opt… (#2078) * Add new indicies to improve performance and remove unused ones to optimize database. * Remove comments * Update map search suggestions panel styling * Add yearly digest (#2073) * Add yearly digest * Rename YearlyDigests to Users::Digests * Minor changes * Update yearly digest layout and styles * Add flags and chart to email * Update colors * Fix layout of stats in yearly digest view * Remove cron job for yearly digest scheduling * Update CHANGELOG.md * Update digest email setting handling * Allow sharing digest for 1 week or 1 month * Change Digests Distance to Bigint * Fix settings page * Update changelog * Add RailsPulse (#2079) * Add RailsPulse * Add RailsPulse monitoring tool with basic HTTP authentication * Bring points_count to integer * Update migration and version * Update rubocop issues --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Robin Tuszik <mail@robin.gg> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
E2E Tests
End-to-end tests for Dawarich using Playwright.
Running Tests
# Run all tests
npx playwright test
# Run V1 map tests (Leaflet-based)
npx playwright test e2e/map/
# Run V2 map tests (MapLibre-based)
npx playwright test e2e/v2/map/
# Run specific test file
npx playwright test e2e/v2/map/settings.spec.js
# Run tests in headed mode (watch browser)
npx playwright test --headed
# Run tests in debug mode
npx playwright test --debug
# Run tests sequentially (avoid parallel issues)
npx playwright test --workers=1
# Run only non-destructive tests (safe for production data)
npx playwright test --grep-invert @destructive
# Run only destructive tests (use with caution!)
npx playwright test --grep @destructive
Test Structure
e2e/
├── setup/ # Test setup and authentication
├── helpers/ # Shared helper functions
├── map/ # V1 Map tests (Leaflet) - 81 tests
├── v2/ # V2 Map tests (MapLibre) - 52 tests
│ ├── helpers/ # V2-specific helpers
│ ├── map/ # V2 core map tests
│ │ └── layers/ # V2 layer-specific tests
│ └── realtime/ # V2 real-time features
└── temp/ # Playwright artifacts (screenshots, videos)
V1 Map Tests (Leaflet-based) - 81 tests
Map Tests
map-controls.spec.js- Basic map controls, zoom, tile layers (5 tests)map-layers.spec.js- Layer toggles: Routes, Heatmap, Fog, etc. (8 tests)map-points.spec.js- Point interactions and deletion (4 tests, 1 destructive)map-visits.spec.js- Confirmed visit interactions and management (5 tests, 3 destructive)map-suggested-visits.spec.js- Suggested visit interactions (6 tests, 3 destructive)map-add-visit.spec.js- Add visit control and form (8 tests)map-selection-tool.spec.js- Selection tool functionality (4 tests)map-calendar-panel.spec.js- Calendar panel navigation (9 tests)map-side-panel.spec.js- Side panel (visits drawer) functionality (13 tests)*map-bulk-delete.spec.js- Bulk point deletion (12 tests, all destructive)map-places-creation.spec.js- Creating new places on map (9 tests, 2 destructive)map-places-layers.spec.js- Places layer visibility and filtering (10 tests)
* Some side panel tests may be skipped if demo data doesn't contain visits
V2 Map Tests (MapLibre-based) - 52 tests
Organized by feature domain:
Core Map Tests
v2/map/core.spec.js- Map initialization, lifecycle, loading states (8 tests)v2/map/navigation.spec.js- Zoom controls, date picker navigation (4 tests)v2/map/interactions.spec.js- Point clicks, hover effects, popups (2 tests)v2/map/settings.spec.js- Settings panel, layer toggles, persistence (10 tests)v2/map/performance.spec.js- Load time benchmarks, efficiency (2 tests)
Layer Tests
v2/map/layers/points.spec.js- Points display, GeoJSON data (3 tests)v2/map/layers/routes.spec.js- Routes geometry, styling, ordering (8 tests)v2/map/layers/heatmap.spec.js- Heatmap creation, toggle, persistence (3 tests)v2/map/layers/visits.spec.js- Visits layer toggle and display (2 tests)v2/map/layers/photos.spec.js- Photos layer toggle and display (2 tests)v2/map/layers/areas.spec.js- Areas layer toggle and display (2 tests)v2/map/layers/advanced.spec.js- Fog of war, scratch map (3 tests)
Real-time Features
v2/realtime/family.spec.js- Family tracking, ActionCable (2 tests, skipped)
V2 Test Organization Benefits
- ✅ Feature-based hierarchy - Clear organization by domain
- ✅ Zero duplication - All settings tests consolidated
- ✅ Easy to navigate - Obvious file naming
- ✅ Better maintainability - One feature = one file
Test Tags
Tests are tagged to enable selective execution:
- @destructive (22 tests in V1) - Tests that delete or modify data:
- Bulk delete operations (12 tests)
- Point deletion (1 test)
- Visit modification/deletion (3 tests)
- Suggested visit actions (3 tests)
- Place creation (3 tests)
Usage:
# Safe for staging/production - run only non-destructive tests
npx playwright test --grep-invert @destructive
# Use with caution - run only destructive tests
npx playwright test --grep @destructive
# Run specific destructive test file
npx playwright test e2e/map/map-bulk-delete.spec.js
Helper Functions
V1 Map Helpers (helpers/map.js)
waitForMap(page)- Wait for Leaflet map initializationenableLayer(page, layerName)- Enable a map layer by nameclickConfirmedVisit(page)- Click first confirmed visit circleclickSuggestedVisit(page)- Click first suggested visit circlegetMapZoom(page)- Get current map zoom level
V2 Map Helpers (v2/helpers/setup.js)
navigateToMapsV2(page)- Navigate to MapLibre mapnavigateToMapsV2WithDate(page, startDate, endDate)- Navigate with date rangewaitForMapLibre(page)- Wait for MapLibre initializationwaitForLoadingComplete(page)- Wait for data loadinghasMapInstance(page)- Check if map is initializedgetMapZoom(page)- Get current zoom levelgetMapCenter(page)- Get map center coordinateshasLayer(page, layerId)- Check if layer existsgetLayerVisibility(page, layerId)- Get layer visibility stategetPointsSourceData(page)- Get points source datagetRoutesSourceData(page)- Get routes source dataclickMapAt(page, x, y)- Click at specific coordinateshasPopup(page)- Check if popup is visible
Navigation Helpers (helpers/navigation.js)
closeOnboardingModal(page)- Close getting started modalnavigateToDate(page, startDate, endDate)- Navigate to specific date rangenavigateToMap(page)- Navigate to V1 map with setup
Selection Helpers (helpers/selection.js)
drawSelectionRectangle(page, options)- Draw selection on mapenableSelectionMode(page)- Enable area selection tool
Common Patterns
V1 Basic Test Template (Leaflet)
import { test, expect } from '@playwright/test';
import { navigateToMap } from '../helpers/navigation.js';
import { waitForMap } from '../helpers/map.js';
test('my test', async ({ page }) => {
await navigateToMap(page);
await waitForMap(page);
// Your test logic
});
V2 Basic Test Template (MapLibre)
import { test, expect } from '@playwright/test';
import { closeOnboardingModal } from '../../helpers/navigation.js';
import {
navigateToMapsV2,
waitForMapLibre,
waitForLoadingComplete
} from '../helpers/setup.js';
test.describe('My Feature', () => {
test.beforeEach(async ({ page }) => {
await navigateToMapsV2(page);
await closeOnboardingModal(page);
await waitForMapLibre(page);
await waitForLoadingComplete(page);
});
test('my test', async ({ page }) => {
// Your test logic
});
});
V2 Testing Layer Visibility
import { getLayerVisibility } from '../helpers/setup.js';
// Check if layer is visible
const isVisible = await getLayerVisibility(page, 'points');
expect(isVisible).toBe(true);
// Wait for layer to exist
await page.waitForFunction(() => {
const element = document.querySelector('[data-controller*="maps--maplibre"]');
const app = window.Stimulus || window.Application;
const controller = app?.getControllerForElementAndIdentifier(element, 'maps--maplibre');
return controller?.map?.getLayer('routes') !== undefined;
}, { timeout: 5000 });
V2 Testing Settings Panel
// Open settings
await page.click('button[title="Open map settings"]');
await page.waitForTimeout(400);
// Switch to layers tab
await page.click('button[data-tab="layers"]');
await page.waitForTimeout(300);
// Check toggle state
const toggle = page.locator('label:has-text("Points")').first().locator('input.toggle');
const isChecked = await toggle.isChecked();
Debugging
View Test Artifacts
# Open HTML report
npx playwright show-report
# Screenshots and videos are in:
test-results/
Common Issues
V1 Tests
- Flaky tests: Run with
--workers=1to avoid parallel interference - Timeout errors: Increase timeout in test or use
page.waitForTimeout() - Map not loading: Ensure
waitForMap()is called after navigation
V2 Tests
- Layer not ready: Use
page.waitForFunction()to wait for layer existence - Settings panel timing: Add
waitForTimeout()after opening/closing - Parallel test failures: Some tests pass individually but fail in parallel - run with
--workers=3or--workers=1 - Source data not available: Wait for source to be defined before accessing data
V2 Test Tips
- Always wait for MapLibre to initialize with
waitForMapLibre(page) - Wait for data loading with
waitForLoadingComplete(page) - Add layer existence checks before testing layer properties
- Use proper waits for settings panel animations
- Consider timing when testing layer toggles
CI/CD
Tests run with:
- 1 worker (sequential)
- 2 retries on failure
- Screenshots/videos on failure
- JUnit XML reports
See playwright.config.js for full configuration.
Important Considerations
- We're using Rails 8 with Turbo, which might not cause full page reloads
- V2 map uses MapLibre GL JS with Stimulus controllers
- V2 settings are persisted to localStorage
- V2 layer visibility is based on user settings (no hardcoded defaults)
- Some V2 layers (routes, heatmap) are created dynamically based on data
Test Migration Notes
V2 tests were refactored from phase-based to feature-based organization:
- Before: 9 phase files, 96 tests (many duplicates)
- After: 13 feature files, 52 focused tests (zero duplication)
- Code reduction: 56% (2,314 lines → 1,018 lines)
- Pass rate: 94% (49/52 tests passing, 1 flaky, 2 skipped)
See E2E_REFACTORING_SUCCESS.md for complete migration details.