Vadym/obsidian
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
obsidian/wiki/connections/lxc-networking-api-failures.md
Vadym Samoilenko 1bf6284acd vault backup: 2026-04-19 22:03:39
2026-04-19 22:03:39 +01:00

3.9 KiB
Raw Blame History

title connects sources created updated
Connection: LXC Networking Failures — ARP Cache vs SSL as Root Cause
concepts/lxc-arp-cache-api-failures
concepts/nodejs-ssl-system-trust-store
concepts/homepage-proxmox-widget-quirks
daily/2026-04-19.md
2026-04-19 2026-04-19

Connection: LXC Networking Failures — ARP Cache vs SSL as Root Cause

The Connection

When a service inside an LXC container fails to reach another service via API, there are two distinct root causes that produce identical symptoms: stale ARP cache entries (layer 2 network issue) and Node.js refusing self-signed TLS certificates (application layer issue). Both manifest as "API Error" in Homepage widgets and both require fixes that seem unrelated to each other. Solving one doesn't solve the other — both must be addressed.

Key Insight

The surface symptom is identical but the causes are orthogonal. ARP cache failure means packets never leave the container correctly (layer 2 routing). SSL failure means packets arrive at the destination but the TLS handshake fails (application layer). Both look like "the API isn't responding" from inside the container.

This is non-obvious because:

  1. External tools (curl from the host, another machine) work fine in both cases — they bypass both the LXC's ARP cache and Node.js's SSL stack
  2. The error messages are generic ("API Error", connection refused) — they don't distinguish between "I couldn't send the packet" and "I sent the packet but rejected the response"
  3. Fixing one reveals the other: after flushing ARP cache, Homepage showed different errors (SSL-related N/A) instead of generic API errors

Diagnosis Flow

"All widget API calls failing inside LXC"
         │
         ├─→ Does ping <IP> from inside LXC fail?
         │         YES → ARP cache issue
         │         → ip neigh flush dev eth0
         │
         └─→ Does curl https://<IP>:<port> from inside LXC fail?
                   YES → SSL issue
                   → Add NODE_TLS_REJECT_UNAUTHORIZED=0 to systemd unit
                   NO (curl works but widget fails) → Widget config issue
                   → Check widget type field, pnpm start, env var duplicates

Evidence (2026-04-19 Session)

The Homepage debugging session on 2026-04-19 hit both issues sequentially:

  1. ARP cache was the first issue — LXC 103 had stale MAC entries for LXC 104/105 IPs. All API calls failed. After arp -d and ip neigh flush dev eth0, some widgets started working.

  2. SSL was the second issue — Proxmox API at https://192.168.1.100:8006 uses a self-signed cert. After adding the cert to the Linux trust store (no effect on Node.js), setting NODE_TLS_REJECT_UNAUTHORIZED=0 in the systemd unit file resolved the Proxmox widget SSL errors.

  3. Widget config was the third issue — Homepage v0.10.9 proxmox widget requires explicit type: node|vm|lxc. Without it, API calls succeed but N/A is displayed.

The session took significantly longer because these three issues were entangled — fixing one revealed the next, making it appear the fixes weren't working.

Practical Lesson

When debugging API failures from inside an LXC container, always run this sequence before diving into application-layer debugging:

# 1. Test raw connectivity (rules out ARP)
ping -c 3 <target_IP>

# 2. Test TCP + SSL (rules out SSL)  
curl -k https://<target_IP>:<port>/some/endpoint

# 3. Test SSL strictly (confirms SSL is the issue)
curl https://<target_IP>:<port>/some/endpoint

# Only if both ping and curl work → investigate application layer

Related Concepts