Testing Strategy Guide
Philosophy
Risk-Based Testing: Priority = Business Impact (1–5) × Probability of failure (1–5).
| Priority |
Decision |
Example |
| ≥ 15 |
MUST test |
RBAC logic, job state machine, audit logger |
| 9–14 |
SHOULD test |
Translation pipeline, TTS routing |
| ≤ 8 |
SKIP (manual sufficient) |
Email template rendering, UI cosmetics |
Write tests for business logic, not for framework behaviour. Never test that FastAPI routes a request — test that YOUR business logic in the handler produces the correct outcome.
Current Coverage (as of 2026-04-29 audit)
| Layer |
Files |
Files with tests |
Risk-weighted coverage |
| Backend |
118 |
8 (7%) |
~3% |
| Frontend |
98 |
~12 (12%) |
— |
| E2E |
— |
3 spec files |
Effectively 0 (most tests skipped) |
Critical gap: All Celery tasks (10 files) and 19 service files have zero test coverage. See full audit at /tmp/audit/test-audit.md.
Test Pyramid
| Level |
Framework |
Location |
Current count |
| Unit (backend) |
pytest + AsyncMock |
backend/tests/unit/ |
8 files, ~338 assertions |
| Unit (frontend) |
Vitest + RTL |
frontend/src/**/__tests__/ |
9 files, ~218 assertions |
| Integration (backend) |
pytest + FastAPI TestClient |
backend/tests/integration/ |
Does not exist yet |
| E2E |
Playwright |
frontend/tests/e2e/ |
3 files, mostly skipped |
Test Commands
| Command |
What it runs |
cd backend && poetry run pytest |
All backend unit tests |
cd backend && poetry run pytest -v tests/unit/test_security.py |
Single test file |
cd frontend && npm run test |
All frontend unit tests (Vitest) |
cd frontend && npm run test:e2e |
Playwright E2E tests |
cd frontend && npm run test:coverage |
Unit tests with coverage report |
docker compose exec backend python -m pytest |
Tests inside Docker (for integration tests) |
What Exists and Is High-Value
| Test file |
Value |
What it tests |
backend/tests/unit/test_security.py |
HIGH |
JWT encode/decode, expiry, type fields, password hashing |
backend/tests/unit/test_vtt.py |
HIGH |
VTT parsing (26 tests) |
backend/tests/unit/test_vtt_retimer.py |
HIGH |
VTT timing logic (27 tests) |
frontend/src/lib/__tests__/auth.test.ts |
HIGH |
JWT in-memory store, refresh flow |
frontend/src/components/Auth/__tests__/RequireAuth.test.tsx |
HIGH |
Auth guard redirect |
Priority Gaps to Fill
The following are MUST-fill based on Priority ≥15:
| Priority |
Module |
Gap |
| 25 |
tasks/ingest_and_ai.py |
Job state machine — zero tests |
| 20 |
core/authz.py |
RBAC permission checks — zero tests |
| 20 |
services/audit_logger.py |
Audit trail correctness — zero tests |
| 20 |
services/glossary_service.py |
Hybrid retrieval — zero tests |
| 16 |
services/language_qc.py |
QC state transitions — zero tests |
| 16 |
tasks/translate_and_synthesize.py |
Translation pipeline — zero tests |
Full test plan at /tmp/audit/test-plan.md.
Anti-Patterns to Avoid
| Anti-pattern |
Why |
Fix |
Hardcoded job IDs like test-job-123 |
Non-existent in test DB |
Use factories to create real test data |
with patch(...) as mock: in every test method |
Setup duplication |
Move to @pytest.fixture(autouse=True) |
MagicMock() on async functions |
Silently returns a mock, not a coroutine |
Use AsyncMock() |
| Testing that a library function was called |
Tests library, not our logic |
Test the business outcome |
E2E tests that are .skip |
They provide no coverage |
Implement auth fixture and un-skip |
Infrastructure Required Before Writing Integration/E2E Tests
| Blocker |
What's needed |
Backend conftest.py |
Shared MockSettings, mock_db, test_user_factory, test_job_factory |
| Celery test mode |
task_always_eager=True fixture for synchronous task execution |
| Playwright auth fixture |
Wire tests/helpers/auth.ts into beforeEach in all spec files |
| Playwright seed fixture |
tests/fixtures/seed.ts to create test jobs, glossary, linguists |
| Mock AI responses |
tests/mocks/gemini-responses/*.json fixtures |
Maintenance
Update triggers: New test file added, coverage target changes, new testing tool added.
Verification: All commands in the Commands table execute without error. Priority gap table matches the current test-audit report.
a3b300b76a