---
title: "MongoDB — Schema Validator Migrations Can Be Silently Skipped"
aliases: [mongodb-schema-validator, mongodb-collmod-migration, mongodb-jsonschema-verification]
tags: [mongodb, migrations, debugging, python, fastapi, gotcha]
sources:
  - "daily/2026-04-30.md"
created: 2026-04-30
updated: 2026-04-30
---

# MongoDB — Schema Validator Migrations Can Be Silently Skipped

MongoDB migration systems (like `migrate.py` pattern) record migrations as "applied" in a migrations collection. If the `collMod` command inside a migration fails silently (e.g., network hiccup, auth issue, wrong database) or was never executed, the migration is still marked applied — and `migrate up` returns "No migrations to apply" on subsequent runs. The validator is never updated, and writes are rejected with validation errors.

## Key Points

- **Migrations marked "applied" does NOT mean the `collMod` ran** — the migration tracking record and the actual schema change are independent
- **Symptom:** worker code writes a new enum value to a field (`processing_failed`, `tts_generating`), MongoDB rejects with a validation error, service crashes or enters a retry loop
- **Diagnosis:** verify the actual validator with `db.getCollectionInfos({name:"collection_name"})[0].options.validator`
- **Fix:** run the `collMod` directly in the MongoDB shell (or via eval) — no need to manipulate migration history
- **Prevention:** always add a post-migration assertion that reads back the validator and confirms the new value is present

## Details

### The Silent Migration Failure

```python
# migrations/2026-04-29-000000_add_processing_failed_status.py

async def up(db):
    await db.command({
        "collMod": "jobs",
        "validator": {
            "$jsonSchema": {
                "properties": {
                    "status": {
                        "enum": ["created", "processing", "completed", "processing_failed"]
                    }
                }
            }
        }
    })
    # If this fails silently → migration table records "applied" anyway
    # Next run: "No migrations to apply" — but the validator was never updated
```

### Verifying the Current Validator

```javascript
// MongoDB shell — check what's actually in the validator
db.getCollectionInfos({name: "jobs"})[0].options.validator

// Or via Python
info = await db.command("listCollections", filter={"name": "jobs"})
validator = info["cursor"]["firstBatch"][0]["options"].get("validator")
print(validator)
```

Look for the `enum` list under the relevant field. If `processing_failed` is absent, the migration didn't run.

### Direct Fix via collMod

When the migration is already marked applied and `migrate up` won't rerun it:

```javascript
// MongoDB shell — run directly
db.runCommand({
    collMod: "jobs",
    validator: {
        "$jsonSchema": {
            "bsonType": "object",
            "properties": {
                "status": {
                    "bsonType": "string",
                    "enum": ["created", "processing", "completed", "processing_failed", "tts_generating"]
                }
            }
        }
    },
    validationLevel: "moderate"
})
```

`validationLevel: "moderate"` allows existing documents that don't conform to remain readable — only new inserts/updates are validated.

Or via Python inline eval:

```python
await db.command(
    "collMod", "jobs",
    validator={"$jsonSchema": {...}},
    validationLevel="moderate"
)
```

### Prevention: Post-Migration Assertion

```python
async def up(db):
    await db.command({"collMod": "jobs", "validator": {...}})

    # VERIFY the change actually took effect
    info = (await db.list_collections(filter={"name": "jobs"}).to_list(1))[0]
    validator = info.get("options", {}).get("validator", {})
    schema = validator.get("$jsonSchema", {})
    status_enum = schema.get("properties", {}).get("status", {}).get("enum", [])
    assert "processing_failed" in status_enum, "Migration collMod did not apply!"
```

### Real Incident (2026-04-30)

`video-accessibility` workers tried to write `status: "processing_failed"` to a job document. MongoDB `$jsonSchema` validator rejected it (enum only had `created`, `processing`, `completed`). The migration `2026-04-29-000000_add_processing_failed_status_and_indexes.py` was recorded as applied in the migration tracking collection, but the `collMod` never ran. Worker entered a retry loop (same validation error every attempt). Fix was running `collMod` directly with `--eval` in the MongoDB shell.

## Related Concepts

- [[wiki/concepts/celery-redis-queue-flush-on-deterministic-error]] — when a deterministic error (like validation rejection) causes Celery retry loops, Redis must also be flushed
- [[wiki/concepts/mongodb-enum-deserialization]] — related MongoDB enum handling on the read side

## Sources

- [[daily/2026-04-30.md]] — `processing_failed` status write rejected by MongoDB validator; migration was marked applied but `collMod` never ran; fixed with direct `collMod` via shell eval; `getCollectionInfos` used to verify
