OVHserver/opt/05-backups/RESTORE-GUIDE.md

AI-Impress Disaster Recovery Guide

Version: 2.2.0 Last Updated: 2025-11-13 Purpose: Complete step-by-step guide to restore full infrastructure from backups

---

Table of Contents

1. Overview
2. Prerequisites
3. Recovery Scenarios
4. Full System Restoration
5. Partial Recovery
6. Verification
7. Troubleshooting

---

Overview

This guide covers full disaster recovery for AI-Impress infrastructure. With backup version 2.2.0, we achieve 100% recovery coverage of all critical components.

What's Backed Up

Databases:

• PostgreSQL (postgres-main): n8n, Odoo, Vaultwarden, WikiJS, Evolution, Documenso, Supabase
• PostgreSQL (authentik-postgres): Authentik SSO users and configuration
• MariaDB (mautic-db): Mautic marketing automation
• MongoDB (if present)

Docker Volumes:

• authentik_authentik-postgres-data - Authentik database
• authentik_authentik-redis-data - Authentik sessions
• evolution-api_evolution-data - WhatsApp sessions and messages
• n8n-shared_n8n-data - n8n workflows and credentials
• odoo_odoo-data - Odoo file store and attachments
• odoo_odoo-addons - Custom Odoo modules
• vaultwarden_vaultwarden-data - Password vaults
• outline_outline-data - Outline wiki data
• wikijs_data - WikiJS data

Application Data:

• Vault secrets (/opt/00-infrastructure/vault/data)
• Docker Compose files and .env configs
• Supabase storage
• Grafana dashboards
• Documenso signed documents
• Evolution API WhatsApp instances
• Mautic sync data

Cloud Backups:

• HOT (R2): Last 3 days for quick recovery
• COLD (Google Drive): 7 days + 4 weeks + 3 months

---

Prerequisites

Required Information

1. Server Access:

   • New/replacement server IP address
   • SSH access (ubuntu user)
   • sudo privileges

2. Backup Credentials:

   • Restic password (from /opt/05-backups/restic/.env)
   • Cloudflare R2 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
   • Google Drive rclone configuration
   • Vault unseal keys (if using Vault)

3. DNS & Domain:

   • Domain: *.ai-impress.com
   • Cloudflare API token for SSL

4. Required Software:

   • Ubuntu 22.04 LTS (or compatible)
   • Docker & Docker Compose
   • Restic
   • rclone (for Google Drive)

---

Recovery Scenarios

Scenario 1: Complete Server Loss

Situation: Physical server destroyed, migrating to new hardware Recovery Time: 4-6 hours Procedure: Full System Restoration

Scenario 2: Single Service Failure

Situation: One service (e.g., n8n) corrupted or lost data Recovery Time: 30 minutes - 2 hours Procedure: Partial Recovery

Scenario 3: Database Corruption

Situation: PostgreSQL or MariaDB database corrupted Recovery Time: 1-2 hours Procedure: Database-Only Recovery

---

Full System Restoration

PHASE 1: Prepare New Server (30-60 minutes)

1.1 Install Base System

# Update system
sudo apt update && sudo apt upgrade -y

# Install required packages
sudo apt install -y \
    docker.io \
    docker-compose \
    git \
    curl \
    wget \
    restic \
    rclone \
    unzip

1.2 Create Directory Structure

# Create main directories
sudo mkdir -p /opt /mnt /data
sudo chown -R ubuntu:ubuntu /opt /mnt /data

# Create backup directories
sudo mkdir -p /mnt/backups/local-backups
sudo mkdir -p /opt/05-backups/{scripts,logs,reports,restic}

1.3 Setup Docker Networks

# Create external networks
docker network create traefik-public
docker network create database-internal

---

PHASE 2: Restore from Cloud Backup (1-2 hours)

2.1 Configure Restic

# Create Restic credentials file
cat > /opt/05-backups/restic/.env << 'EOF'
# Cloudflare R2 (HOT Storage)
export RESTIC_REPOSITORY="s3:https://6aff840a680098927b58beb93b59dd03.r2.cloudflarestorage.com/aimpress-backups"
export AWS_ACCESS_KEY_ID="YOUR_R2_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="YOUR_R2_SECRET_KEY"
export RESTIC_PASSWORD="YOUR_RESTIC_PASSWORD"

# Google Drive (COLD Storage) - alternative
# export RESTIC_REPOSITORY="rclone:gdrive:ai-impress-backups"
EOF

source /opt/05-backups/restic/.env

2.2 List Available Snapshots

# Check R2 snapshots (last 3 days)
restic -r "$RESTIC_REPOSITORY" snapshots

# Or check Google Drive (longer history)
restic -r "rclone:gdrive:ai-impress-backups" snapshots

2.3 Restore Latest Snapshot

# Restore to /mnt/backups
cd /mnt/backups
restic -r "$RESTIC_REPOSITORY" restore latest --target /mnt/backups

# Verify restoration
ls -lah /mnt/backups/local-backups/

---

PHASE 3: Restore Databases (1-2 hours)

3.1 Start Database Containers

# Start PostgreSQL main
cd /opt/00-infrastructure/postgres
docker compose up -d

# Wait for healthy status
docker ps | grep postgres-main

# Start Authentik PostgreSQL
cd /opt/01-security/authentik
docker compose up -d authentik-postgres

# Start MariaDB for Mautic (if used)
cd /opt/03-business/mautic
docker compose up -d mautic-db

3.2 Restore PostgreSQL Databases

# Find latest PostgreSQL dump
LATEST_PG_DUMP=$(ls -t /mnt/backups/local-backups/postgresql-postgres-main-*.sql.gz | head -1)

# Restore postgres-main
gunzip -c "$LATEST_PG_DUMP" | docker exec -i postgres-main psql -U aimpress_admin postgres

# Find and restore Authentik database
LATEST_AUTHENTIK_DUMP=$(ls -t /mnt/backups/local-backups/postgresql-authentik-postgres-*.sql.gz | head -1)

gunzip -c "$LATEST_AUTHENTIK_DUMP" | docker exec -i authentik-postgres psql -U authentik postgres

3.3 Restore MariaDB Database

# Find latest MariaDB dump
LATEST_MARIADB_DUMP=$(ls -t /mnt/backups/local-backups/mariadb-mautic-db-*.sql.gz | head -1)

# Restore
gunzip -c "$LATEST_MARIADB_DUMP" | docker exec -i mautic-db mariadb

---

PHASE 4: Restore Docker Volumes (1-2 hours)

4.1 Extract Volume Backups

cd /mnt/backups/local-backups

# Find latest volume backups
ls -t *-volume-*.tar.gz

4.2 Restore Critical Volumes

# Function to restore volume
restore_volume() {
    local volume_name=$1
    local backup_file=$2

    echo "Restoring $volume_name..."

    # Create volume if doesn't exist
    docker volume create "$volume_name"

    # Get volume mount point
    local volume_path=$(docker volume inspect "$volume_name" --format '{{.Mountpoint}}')

    # Extract backup to volume
    sudo tar xzf "$backup_file" -C "$(dirname "$volume_path")" --strip-components=1

    echo "✅ $volume_name restored"
}

# Restore Authentik volumes
restore_volume "authentik_authentik-postgres-data" "$(ls -t authentik-postgres-volume-*.tar.gz | head -1)"
restore_volume "authentik_authentik-redis-data" "$(ls -t authentik-redis-volume-*.tar.gz | head -1)"

# Restore Evolution API
restore_volume "evolution-api_evolution-data" "$(ls -t evolution-volume-*.tar.gz | head -1)"

# Restore n8n
restore_volume "n8n-shared_n8n-data" "$(ls -t n8n-volume-*.tar.gz | head -1)"

# Restore Odoo
restore_volume "odoo_odoo-data" "$(ls -t odoo-data-volume-*.tar.gz | head -1)"
restore_volume "odoo_odoo-addons" "$(ls -t odoo-addons-volume-*.tar.gz | head -1)"

# Restore Vaultwarden
restore_volume "vaultwarden_vaultwarden-data" "$(ls -t vaultwarden-volume-*.tar.gz | head -1)"

# Restore Outline & WikiJS
restore_volume "outline_outline-data" "$(ls -t outline-volume-*.tar.gz | head -1)"
restore_volume "wikijs_data" "$(ls -t wikijs-volume-*.tar.gz | head -1)"

---

PHASE 5: Restore Configurations (30-60 minutes)

5.1 Restore Docker Compose Files and .env

# Find latest configs backup
LATEST_CONFIGS=$(ls -t /mnt/backups/local-backups/docker-configs-*.tar.gz | head -1)

# Extract to /opt
cd /
sudo tar xzf "$LATEST_CONFIGS"

# Verify
ls -la /opt/*/docker-compose.yml

5.2 Restore Vault Data

# Find latest Vault backup
LATEST_VAULT=$(ls -t /mnt/backups/local-backups/vault-data-*.tar.gz | head -1)

# Extract
sudo tar xzf "$LATEST_VAULT" -C /opt/00-infrastructure/vault/

# Verify
ls -la /opt/00-infrastructure/vault/data/

5.3 Restore Application Data

# Find latest app data backup
LATEST_APP_DATA=$(ls -t /mnt/backups/local-backups/app-data-*.tar.gz | head -1)

# Extract
cd /
sudo tar xzf "$LATEST_APP_DATA"

# This restores:
# - Grafana dashboards
# - Supabase storage
# - Documenso documents
# - Evolution instances
# - Mautic data
# - And more

---

PHASE 6: Start Services (1-2 hours)

6.1 Start Infrastructure Services

# Start in order:

# 1. Traefik (reverse proxy)
cd /opt/00-infrastructure/traefik
docker compose up -d

# 2. PostgreSQL, Redis, RabbitMQ
cd /opt/00-infrastructure/postgres && docker compose up -d
cd /opt/00-infrastructure/redis && docker compose up -d
cd /opt/00-infrastructure/rabbitmq && docker compose up -d

# 3. Vault
cd /opt/00-infrastructure/vault && docker compose up -d

# Wait for services to be healthy
docker ps

6.2 Start Security & Authentication

# Authentik (SSO)
cd /opt/01-security/authentik
docker compose up -d

# Vaultwarden (Password Manager)
cd /opt/01-security/vaultwarden
docker compose up -d

# Wait for Authentik to be ready
curl -I https://auth.ai-impress.com

6.3 Start Core Services

# n8n automation
cd /opt/02-core/n8n-shared
docker compose up -d

# Evolution API (WhatsApp)
cd /opt/02-core/evolution-api
docker compose up -d

# Supabase
cd /opt/02-core/supabase/supabase/docker
docker compose up -d

# BigBlueButton (if used)
cd /opt/02-core/bigbluebutton
docker compose up -d

6.4 Start Business Services

# Odoo ERP
cd /opt/03-business/odoo
docker compose up -d

# Outline wiki
cd /opt/03-business/outline
docker compose up -d

# Documenso (document signing)
cd /opt/03-business/documenso
docker compose up -d

# WikiJS
cd /opt/03-business/wikijs
docker compose up -d

# Mautic (if used)
cd /opt/03-business/mautic
docker compose up -d

6.5 Start Monitoring & Tools

# Grafana
cd /opt/04-tools/monitoring/grafana
docker compose up -d

# Prometheus
cd /opt/04-tools/monitoring/prometheus
docker compose up -d

# Loki
cd /opt/04-tools/monitoring/loki
docker compose up -d

# Uptime Kuma
cd /opt/04-tools/uptime-kuma
docker compose up -d

# Portainer
cd /opt/04-tools/portainer
docker compose up -d

---

Verification

Check All Services

# View all running containers
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Check for any failed containers
docker ps -a | grep -v "Up"

# Check logs for errors
docker compose logs --tail=50 [service-name]

Test Key Services

# Test Traefik
curl -I https://traefik.ai-impress.com

# Test Authentik (SSO)
curl -I https://auth.ai-impress.com

# Test n8n
curl -I https://n8n.ai-impress.com

# Test Odoo
curl -I https://odoo.ai-impress.com

# Test Grafana
curl -I https://grafana.ai-impress.com

Verify Data Integrity

PostgreSQL:

# Check database sizes
docker exec postgres-main psql -U aimpress_admin -c "\l+"

# Verify n8n database
docker exec postgres-main psql -U aimpress_admin n8n_shared -c "SELECT COUNT(*) FROM workflow_entity;"

# Verify Odoo database
docker exec postgres-main psql -U aimpress_admin odoo -c "SELECT COUNT(*) FROM res_users;"

Authentik:

# Check Authentik users
docker exec authentik-postgres psql -U authentik authentik -c "SELECT COUNT(*) FROM authentik_core_user;"

Volumes:

# Check volume sizes
docker volume ls -q | xargs docker volume inspect --format '{{ .Name }}: {{ .Mountpoint }}' | while read vol; do
    du -sh $(echo $vol | cut -d: -f2)
done

---

Partial Recovery

Restore Single Service

Example: Restore n8n Only

# 1. Stop n8n
cd /opt/02-core/n8n-shared
docker compose down

# 2. Restore n8n database
LATEST_PG_DUMP=$(ls -t /mnt/backups/local-backups/postgresql-postgres-main-*.sql.gz | head -1)
gunzip -c "$LATEST_PG_DUMP" | docker exec -i postgres-main psql -U aimpress_admin -c "DROP DATABASE n8n_shared; CREATE DATABASE n8n_shared;"
gunzip -c "$LATEST_PG_DUMP" | docker exec -i postgres-main psql -U aimpress_admin n8n_shared

# 3. Restore n8n volume
docker volume rm n8n-shared_n8n-data
docker volume create n8n-shared_n8n-data
LATEST_N8N_VOL=$(ls -t /mnt/backups/local-backups/n8n-volume-*.tar.gz | head -1)
# ... extract volume ...

# 4. Restart n8n
docker compose up -d

---

Database-Only Recovery

# Stop services using the database
cd /opt/02-core/n8n-shared && docker compose stop
cd /opt/03-business/odoo && docker compose stop

# Restore database
LATEST_PG_DUMP=$(ls -t /mnt/backups/local-backups/postgresql-postgres-main-*.sql.gz | head -1)
gunzip -c "$LATEST_PG_DUMP" | docker exec -i postgres-main psql -U aimpress_admin postgres

# Restart services
cd /opt/02-core/n8n-shared && docker compose start
cd /opt/03-business/odoo && docker compose start

---

Troubleshooting

Issue: Container Won't Start

Problem: Service fails to start after restoration

Solution:

# Check logs
docker compose logs [service-name]

# Check if volume exists
docker volume ls | grep [volume-name]

# Check if database exists
docker exec postgres-main psql -U aimpress_admin -l

Issue: Database Connection Errors

Problem: Services can't connect to database

Solution:

# Verify database is running
docker ps | grep postgres

# Check database network
docker network inspect database-internal

# Test connection
docker exec postgres-main psql -U aimpress_admin -c "SELECT 1;"

Issue: SSL Certificate Errors

Problem: HTTPS not working

Solution:

# Check Traefik logs
docker compose -f /opt/00-infrastructure/traefik/docker-compose.yml logs

# Verify acme.json exists
ls -la /opt/00-infrastructure/traefik/acme/acme.json

# If missing, Traefik will regenerate (may take 5-10 minutes)

Issue: Authentik Users Missing

Problem: Can't log in to any service

Solution:

# Check Authentik PostgreSQL
docker ps | grep authentik-postgres

# Verify database restoration
docker exec authentik-postgres psql -U authentik authentik -c "SELECT email FROM authentik_core_user;"

# If empty, re-restore Authentik database

---

Recovery Time Estimates

Scenario	Minimum	Typical	Maximum
Full System	3 hours	4-6 hours	8 hours
Single Service	15 min	30-60 min	2 hours
Database Only	30 min	1 hour	2 hours
Volume Only	10 min	20-30 min	1 hour

---

Post-Recovery Checklist

• All containers running (docker ps)
• All services accessible via HTTPS
• Authentik SSO working (can log in)
• n8n workflows executing
• Odoo accessible with data
• Evolution API WhatsApp connected
• Grafana dashboards visible
• Vaultwarden accessible
• No errors in logs
• SSL certificates valid
• Backup script working (/opt/05-backups/scripts/backup-full-enhanced.sh)

---

Support & Contact

For assistance during recovery:

• Email: admin@ai-impress.com
• Backup Logs: /opt/05-backups/logs/
• Documentation: /opt/CLAUDE.md

---

Last Updated: 2025-11-13 Script Version: backup-full-enhanced.sh v2.2.0