---
title: "Proxmox — 502 Bad Gateway Does Not Mean a Dead Container"
aliases: [proxmox-502, lxc-502, container-misdiagnosis, proxmox-nginx-502]
tags: [proxmox, lxc, nginx, debugging, homelab, incident, destructive-action]
sources:
  - "daily/2026-04-21.md"
created: 2026-04-21
updated: 2026-04-21
---

# Proxmox — 502 Bad Gateway Does Not Mean a Dead Container

A 502 Bad Gateway error from an LXC container does not indicate that the container or its primary service is dead. 502 means the reverse proxy (often Nginx) received no valid response from the upstream — the upstream could be a misconfigured hostname, a wrong port, or a crashed sub-process. Deleting a container based solely on a 502 response risks destroying a healthy service.

## Key Points

- **502 = upstream unreachable, not container dead** — the container is running; something inside it (Nginx config, service crash, wrong hostname) is causing the 502
- **Always visually confirm which container the user is pointing at** before any destructive action (delete, restore, wipe)
- The specific incident: CT107 (Homarr, working) showed 502 because Nginx inside CT107 was proxying to hostname `homarr` (unresolvable) instead of `127.0.1.1`; CT103 was the unused/old container — the wrong one was deleted
- **After deleting the wrong container and restoring from backup**, the Nginx config inside needed manual fixing — backups restore data but not post-deploy config fixes applied after the backup was made
- Proxy managers (NPM, Caddy) on a different LXC than the service add an indirection layer — 502 from NPM means NPM can't reach the service, not that the service container itself is broken

## Details

### The Incident (2026-04-21)

**Setup:**
- CT107 = Homarr (the real, working container — had been restored from backup previously)
- CT103 = old/unused Homarr container (should have been the one to delete)

**What happened:**
1. CT107 returned 502 via NPM
2. Conclusion: "CT107 is the broken/wrong container" → CT107 deleted
3. Reality: CT107's Nginx config was proxying to hostname `homarr` (unresolvable in the container) instead of `127.0.1.1`; fixing the Nginx config would have resolved the 502
4. CT103 (the unused one) survived the session; CT107 had to be restored from backup

**Root cause:** 502 ≠ container is dead. 502 meant Nginx inside CT107 was misconfigured.

### Diagnosing 502 Before Destructive Actions

```bash
# Step 1: Confirm the container is running
ssh pve "pct status 107"
# Should show: status: running

# Step 2: Enter the container and check the service directly
ssh pve "pct exec 107 -- systemctl status homarr"
# or
ssh pve "pct exec 107 -- curl -s http://127.0.1.1:7575/api/health"

# Step 3: Check what Nginx/reverse proxy is configured to proxy to
ssh pve "pct exec 107 -- cat /etc/nginx/sites-enabled/*"
# Look for: proxy_pass http://homarr:7575  ← "homarr" hostname won't resolve

# Step 4: Check if the hostname resolves inside the container
ssh pve "pct exec 107 -- getent hosts homarr"
# No output = unresolvable hostname = 502

# Fix: change proxy_pass to use 127.0.1.1 or 127.0.0.1
ssh pve "pct exec 107 -- sed -i 's|proxy_pass http://homarr|proxy_pass http://127.0.1.1|g' /etc/nginx/sites-enabled/homarr"
ssh pve "pct exec 107 -- nginx -s reload"
```

### Common Causes of 502 in LXC Containers

| Root Cause | Symptom | Fix |
|-----------|---------|-----|
| Nginx proxying to unresolvable hostname | 502 on all requests | Change `proxy_pass` to use IP |
| Service crashed / not started | 502, service not in `ps aux` | `systemctl restart <service>` |
| Wrong port in proxy config | 502 even with correct hostname | Fix port number in proxy config |
| ARP cache stale | Intermittent 502 | `ip neigh flush dev eth0` |
| Container networking issue | 502 + ping fails from inside | Check LXC network config |

### Pre-Destructive-Action Checklist

Before deleting, wiping, or restoring over any Proxmox container:

1. **`pct status <ID>`** — confirm which VMID you're targeting
2. **Ask user** to point at the container in the Proxmox UI web interface (visual confirmation)
3. **`pct exec <ID> -- <service-check>`** — confirm the service state independently of the proxy layer
4. **Check NPM/Caddy config** — is the 502 from the proxy layer, not the container?
5. Only proceed with destructive action if the container is confirmed unnecessary or truly broken

### Backup + Restore Nuance

When a container is restored from backup, it returns to the state at backup time. Any configuration changes made after the backup — including fixing misconfigured Nginx, adding new env vars, or updating service configs — are lost and must be re-applied manually.

A restored container may therefore show the same 502 that triggered the investigation, because the post-backup Nginx fix was never applied to the backup.

## Related Concepts

- [[wiki/concepts/lxc-arp-cache-api-failures]] — another LXC failure mode that appears as API/connectivity errors but has a non-obvious cause
- [[wiki/concepts/homarr-proxmox-integration]] — Homarr-specific configuration (the service involved in this incident)
- [[wiki/connections/lxc-networking-api-failures]] — broader pattern of LXC failures with misleading symptoms
- [[wiki/homelab/_index]] — Proxmox and LXC infrastructure context

## Sources

- [[daily/2026-04-21.md]] — Critical incident: CT107 (working Homarr) deleted based on 502; root cause was Nginx inside CT107 proxying to hostname `homarr` instead of `127.0.1.1`; CT103 was the unused container; CT107 restored from backup; Nginx config fixed post-restore; lesson: 502 ≠ wrong container, always confirm visually before destructive actions
