#!/bin/bash # Module 6: Storage & Backups cat << 'EOF' --- ## 6️⃣ STORAGE & BACKUPS ### Disk Usage Summary ``` EOF df -h | grep -E '^Filesystem|^/dev/' cat << 'EOF' ``` ### Storage Breakdown by Directory ``` EOF du -sh /opt/* /mnt/* 2>/dev/null | sort -h | tail -20 cat << 'EOF' ``` ### Critical Directories EOF for dir in /opt /mnt/psql-data /mnt/backups /var/lib/docker; do if [[ -d "$dir" ]]; then SIZE=$(du -sh "$dir" 2>/dev/null | cut -f1) FILES=$(find "$dir" -type f 2>/dev/null | wc -l) DIRS=$(find "$dir" -type d 2>/dev/null | wc -l) echo "- **$dir**: $SIZE ($FILES files, $DIRS directories)" fi done cat << 'EOF' ### Backup Status EOF if [[ -d /mnt/backups ]]; then BACKUP_SIZE=$(du -sh /mnt/backups 2>/dev/null | cut -f1) BACKUP_COUNT=$(find /mnt/backups -type f 2>/dev/null | wc -l) echo "**Total Backup Size:** $BACKUP_SIZE " echo "**Total Files:** $BACKUP_COUNT " echo "" echo "### Latest 10 Backups" echo "" echo '```' find /mnt/backups -type f \( -name "*.tar.gz" -o -name "*.sql.gz" -o -name "*.zip" \) \ -printf '%T+ %s %p\n' 2>/dev/null | sort -r | head -10 | \ awk '{printf "%s %s %.2f MB %s\n", $1, $2, $3/1024/1024, $4}' echo '```' echo "" # Backup age check LATEST_BACKUP=$(find /mnt/backups -type f -name "*.tar.gz" -o -name "*.sql.gz" 2>/dev/null | sort | tail -1) if [[ -n "$LATEST_BACKUP" ]]; then BACKUP_AGE_DAYS=$(( ($(date +%s) - $(stat -c %Y "$LATEST_BACKUP")) / 86400 )) if [[ "$BACKUP_AGE_DAYS" -lt 1 ]]; then echo "✅ **Latest backup:** Less than 1 day old" elif [[ "$BACKUP_AGE_DAYS" -lt 3 ]]; then echo "⚠️ **Latest backup:** $BACKUP_AGE_DAYS days old" else echo "🔴 **Latest backup:** $BACKUP_AGE_DAYS days old - **CHECK BACKUP SYSTEM**" fi fi else echo "⚠️ Backup directory /mnt/backups not found" fi cat << 'EOF' ### Manual Backup Commands ```bash # Full system backup /opt/05-backups/scripts/backup-full-enhanced.sh # Quick database backup docker exec postgres-main pg_dumpall -U aimpress_admin | \ gzip > /mnt/backups/postgres-$(date +%Y%m%d).sql.gz # Backup specific service config tar -czf /mnt/backups/config-$(date +%Y%m%d).tar.gz /opt/ # Vault secrets export /opt/05-backups/scripts/vault-bitwarden-sync.sh export ``` ### Disk Cleanup Commands ```bash # Clean Docker (removes unused images, containers, volumes) docker system prune -af # Clean old logs (older than 7 days) find /opt -name "*.log" -mtime +7 -delete find /var/log -name "*.log" -mtime +7 -exec truncate -s 0 {} \; # Clean old backups (older than 60 days) find /mnt/backups -name "*.gz" -mtime +60 -delete # Find large files find /opt /mnt -type f -size +100M -exec ls -lh {} \; 2>/dev/null # Interactive disk usage analyzer ncdu /opt ``` ### Vault Secrets Backup Status EOF # Check Vault export status LATEST_VAULT_EXPORT=$(ls -t /opt/05-backups/vault-export-*.json 2>/dev/null | head -1) if [[ -n "$LATEST_VAULT_EXPORT" ]]; then EXPORT_SIZE=$(du -h "$LATEST_VAULT_EXPORT" | cut -f1) EXPORT_DATE=$(stat -c %y "$LATEST_VAULT_EXPORT" 2>/dev/null | cut -d' ' -f1,2 | cut -d'.' -f1) EXPORT_AGE_HOURS=$(( ($(date +%s) - $(stat -c %Y "$LATEST_VAULT_EXPORT")) / 3600 )) SECRET_COUNT=$(jq '[.. | objects | keys] | add | unique | length' "$LATEST_VAULT_EXPORT" 2>/dev/null || echo "unknown") echo "- **Latest Export:** $(basename $LATEST_VAULT_EXPORT)" echo "- **Date:** $EXPORT_DATE ($EXPORT_AGE_HOURS hours ago)" echo "- **Size:** $EXPORT_SIZE" echo "- **Secrets Backed Up:** $SECRET_COUNT" echo "- **Schedule:** Every 4 hours" echo "" if [[ "$EXPORT_AGE_HOURS" -lt 5 ]]; then echo "✅ Vault backups are up to date" else echo "⚠️ Last backup $EXPORT_AGE_HOURS hours ago" fi else echo "🔴 **ERROR:** No Vault exports found" fi cat << 'EOF' **Bitwarden Sync:** Not configured (CLI not installed) ### Cloudflare R2 Remote Backup EOF # Check R2 status if [[ -f /opt/05-backups/restic/.env ]]; then source /opt/05-backups/restic/.env 2>/dev/null R2_STATS=$(restic stats --mode restore-size 2>/dev/null | grep "Total Size" || echo "") if [[ -n "$R2_STATS" ]]; then R2_SIZE=$(echo "$R2_STATS" | grep -oP '\d+\.\d+' | head -1) R2_UNIT=$(echo "$R2_STATS" | grep -oP 'GiB|MiB|KiB' | head -1) R2_SNAPSHOTS=$(restic snapshots --compact 2>/dev/null | grep -c '^[a-f0-9]' || echo "0") echo "- **Used:** ${R2_SIZE} ${R2_UNIT} / 10 GB limit" echo "- **Snapshots:** $R2_SNAPSHOTS" echo "- **Retention:** Keep 3 daily + 1 weekly" echo "- **Schedule:** Daily at 03:00 AM" echo "" if awk -v size="$R2_SIZE" 'BEGIN{exit (size > 10) ? 0 : 1}' 2>/dev/null; then echo "🔴 R2 storage exceeds free tier!" elif awk -v size="$R2_SIZE" 'BEGIN{exit (size > 8) ? 0 : 1}' 2>/dev/null; then echo "⚠️ R2 at >80% of limit" else echo "✅ R2 storage OK" fi else echo "⚠️ Unable to query R2" fi else echo "⚠️ R2 not configured" fi cat << 'EOF' ### Storage Monitoring EOF # Check if disk is over 80% DISK_USAGE=$(df / | awk 'NR==2 {print $5}' | sed 's/%//') if [[ "$DISK_USAGE" -gt 80 ]]; then echo "🔴 **WARNING:** Disk usage is at ${DISK_USAGE}% - cleanup recommended" elif [[ "$DISK_USAGE" -gt 70 ]]; then echo "⚠️ **CAUTION:** Disk usage is at ${DISK_USAGE}% - monitor closely" else echo "✅ **GOOD:** Disk usage is at ${DISK_USAGE}%" fi