---
title: "Connection: LXC Networking Failures — ARP Cache vs SSL as Root Cause"
connects:
  - "concepts/lxc-arp-cache-api-failures"
  - "concepts/nodejs-ssl-system-trust-store"
  - "concepts/homepage-proxmox-widget-quirks"
sources:
  - "daily/2026-04-19.md"
created: 2026-04-19
updated: 2026-04-19
---

# Connection: LXC Networking Failures — ARP Cache vs SSL as Root Cause

## The Connection

When a service inside an LXC container fails to reach another service via API, there are two distinct root causes that produce identical symptoms: stale ARP cache entries (layer 2 network issue) and Node.js refusing self-signed TLS certificates (application layer issue). Both manifest as "API Error" in Homepage widgets and both require fixes that seem unrelated to each other. Solving one doesn't solve the other — both must be addressed.

## Key Insight

**The surface symptom is identical but the causes are orthogonal.** ARP cache failure means packets never leave the container correctly (layer 2 routing). SSL failure means packets arrive at the destination but the TLS handshake fails (application layer). Both look like "the API isn't responding" from inside the container.

This is non-obvious because:
1. External tools (curl from the host, another machine) work fine in both cases — they bypass both the LXC's ARP cache and Node.js's SSL stack
2. The error messages are generic ("API Error", connection refused) — they don't distinguish between "I couldn't send the packet" and "I sent the packet but rejected the response"
3. Fixing one reveals the other: after flushing ARP cache, Homepage showed different errors (SSL-related N/A) instead of generic API errors

## Diagnosis Flow

```
"All widget API calls failing inside LXC"
         │
         ├─→ Does ping <IP> from inside LXC fail?
         │         YES → ARP cache issue
         │         → ip neigh flush dev eth0
         │
         └─→ Does curl https://<IP>:<port> from inside LXC fail?
                   YES → SSL issue
                   → Add NODE_TLS_REJECT_UNAUTHORIZED=0 to systemd unit
                   NO (curl works but widget fails) → Widget config issue
                   → Check widget type field, pnpm start, env var duplicates
```

## Evidence (2026-04-19 Session)

The Homepage debugging session on 2026-04-19 hit both issues sequentially:

1. **ARP cache** was the first issue — LXC 103 had stale MAC entries for LXC 104/105 IPs. All API calls failed. After `arp -d` and `ip neigh flush dev eth0`, some widgets started working.

2. **SSL** was the second issue — Proxmox API at `https://192.168.1.100:8006` uses a self-signed cert. After adding the cert to the Linux trust store (no effect on Node.js), setting `NODE_TLS_REJECT_UNAUTHORIZED=0` in the systemd unit file resolved the Proxmox widget SSL errors.

3. **Widget config** was the third issue — Homepage v0.10.9 proxmox widget requires explicit `type: node|vm|lxc`. Without it, API calls succeed but N/A is displayed.

The session took significantly longer because these three issues were entangled — fixing one revealed the next, making it appear the fixes weren't working.

## Practical Lesson

When debugging API failures from inside an LXC container, always run this sequence before diving into application-layer debugging:

```bash
# 1. Test raw connectivity (rules out ARP)
ping -c 3 <target_IP>

# 2. Test TCP + SSL (rules out SSL)  
curl -k https://<target_IP>:<port>/some/endpoint

# 3. Test SSL strictly (confirms SSL is the issue)
curl https://<target_IP>:<port>/some/endpoint

# Only if both ping and curl work → investigate application layer
```

## Related Concepts

- [[wiki/concepts/lxc-arp-cache-api-failures]] — ARP cache mechanics and fix
- [[wiki/concepts/nodejs-ssl-system-trust-store]] — why Node.js ignores Linux certs and the systemd fix
- [[wiki/concepts/homepage-proxmox-widget-quirks]] — the application where both issues occurred simultaneously
