MongoDB — Schema Validator Migrations Can Be Silently Skipped

MongoDB migration systems (like migrate.py pattern) record migrations as "applied" in a migrations collection. If the collMod command inside a migration fails silently (e.g., network hiccup, auth issue, wrong database) or was never executed, the migration is still marked applied — and migrate up returns "No migrations to apply" on subsequent runs. The validator is never updated, and writes are rejected with validation errors.

Key Points

Migrations marked "applied" does NOT mean the collMod ran — the migration tracking record and the actual schema change are independent
Symptom: worker code writes a new enum value to a field (processing_failed, tts_generating), MongoDB rejects with a validation error, service crashes or enters a retry loop
Diagnosis: verify the actual validator with db.getCollectionInfos({name:"collection_name"})[0].options.validator
Fix: run the collMod directly in the MongoDB shell (or via eval) — no need to manipulate migration history
Prevention: always add a post-migration assertion that reads back the validator and confirms the new value is present

Details

The Silent Migration Failure

# migrations/2026-04-29-000000_add_processing_failed_status.py

async def up(db):
    await db.command({
        "collMod": "jobs",
        "validator": {
            "$jsonSchema": {
                "properties": {
                    "status": {
                        "enum": ["created", "processing", "completed", "processing_failed"]
                    }
                }
            }
        }
    })
    # If this fails silently → migration table records "applied" anyway
    # Next run: "No migrations to apply" — but the validator was never updated

Verifying the Current Validator

// MongoDB shell — check what's actually in the validator
db.getCollectionInfos({name: "jobs"})[0].options.validator

// Or via Python
info = await db.command("listCollections", filter={"name": "jobs"})
validator = info["cursor"]["firstBatch"][0]["options"].get("validator")
print(validator)

Look for the enum list under the relevant field. If processing_failed is absent, the migration didn't run.

Direct Fix via collMod

When the migration is already marked applied and migrate up won't rerun it:

// MongoDB shell — run directly
db.runCommand({
    collMod: "jobs",
    validator: {
        "$jsonSchema": {
            "bsonType": "object",
            "properties": {
                "status": {
                    "bsonType": "string",
                    "enum": ["created", "processing", "completed", "processing_failed", "tts_generating"]
                }
            }
        }
    },
    validationLevel: "moderate"
})

validationLevel: "moderate" allows existing documents that don't conform to remain readable — only new inserts/updates are validated.

Or via Python inline eval:

await db.command(
    "collMod", "jobs",
    validator={"$jsonSchema": {...}},
    validationLevel="moderate"
)

Prevention: Post-Migration Assertion

async def up(db):
    await db.command({"collMod": "jobs", "validator": {...}})

    # VERIFY the change actually took effect
    info = (await db.list_collections(filter={"name": "jobs"}).to_list(1))[0]
    validator = info.get("options", {}).get("validator", {})
    schema = validator.get("$jsonSchema", {})
    status_enum = schema.get("properties", {}).get("status", {}).get("enum", [])
    assert "processing_failed" in status_enum, "Migration collMod did not apply!"

Real Incident (2026-04-30)

video-accessibility workers tried to write status: "processing_failed" to a job document. MongoDB $jsonSchema validator rejected it (enum only had created, processing, completed). The migration 2026-04-29-000000_add_processing_failed_status_and_indexes.py was recorded as applied in the migration tracking collection, but the collMod never ran. Worker entered a retry loop (same validation error every attempt). Fix was running collMod directly with --eval in the MongoDB shell.

wiki/concepts/celery-redis-queue-flush-on-deterministic-error — when a deterministic error (like validation rejection) causes Celery retry loops, Redis must also be flushed
wiki/concepts/mongodb-enum-deserialization — related MongoDB enum handling on the read side

Sources

daily/2026-04-30.md — processing_failed status write rejected by MongoDB validator; migration was marked applied but collMod never ran; fixed with direct collMod via shell eval; getCollectionInfos used to verify

5 KiB Raw Blame History