video-query/LOG_EXTRACTION_README.md

Log Extraction Scripts

These scripts extract user email addresses and prompts from systemd logs for the video generation service.

Scripts Overview

1. extract_user_logs.sh - Basic Extraction

Simple script that extracts basic information and creates a CSV report.

Usage:

./extract_user_logs.sh [output_file.csv]

Examples:

# Use default output file (video_generation_usage.csv)
./extract_user_logs.sh

# Specify custom output file
./extract_user_logs.sh my_usage_report.csv

2. extract_user_logs_robust.sh - Enhanced Extraction

More robust script with error handling, progress indicators, and detailed reporting.

Usage:

./extract_user_logs_robust.sh [output_file.csv] [service_name] [date_range]

Examples:

# Basic usage
./extract_user_logs_robust.sh

# Custom output file
./extract_user_logs_robust.sh usage_report.csv

# Different service name
./extract_user_logs_robust.sh report.csv my-video-service

# Specific date range (last 30 days)
./extract_user_logs_robust.sh report.csv veo-video-generator "--since=30 days ago"

# Specific date range (from June 1st, 2024)
./extract_user_logs_robust.sh report.csv veo-video-generator "--since=2024-06-01"

# Between specific dates
./extract_user_logs_robust.sh report.csv veo-video-generator "--since=2024-06-01 --until=2024-06-15"

3. quick_extract.sh - Quick & Simple

One-liner style extraction for quick checks.

Usage:

./quick_extract.sh [service_name]

Requirements

• jq - JSON processor (install with: sudo apt install jq or brew install jq)
• journalctl - systemd journal viewer (usually pre-installed on systemd systems)
• Bash shell

Output Format

The scripts generate CSV files with the following columns:

• timestamp - When the request was made
• user_email - Email address of the user
• prompt - The generation prompt used
• video_length_sec - Length of video requested (seconds)
• aspect_ratio - Video aspect ratio
• person_generation - Person generation setting

Log Format Expected

The scripts look for log lines matching this pattern:

Jun 14 20:23:20 optical-web-1 veo-video-generator[3458795]: DEBUG: Raw JSON data received: {'prompt': '...', 'user_email': '...', ...}

Troubleshooting

No records found

• Check if the service name is correct: systemctl list-units | grep video
• Verify logs exist: journalctl -u veo-video-generator | head
• Check if the log format matches what the script expects

Permission denied

• You may need to run with sudo to access systemd logs: sudo ./extract_user_logs.sh

jq command not found

• Install jq:
  • Ubuntu/Debian: sudo apt install jq
  • macOS: brew install jq
  • CentOS/RHEL: sudo yum install jq

Invalid JSON errors

• The robust script will show warnings for malformed JSON
• Check the log format to ensure it matches the expected pattern

Example Output

timestamp,user_email,prompt,video_length_sec,aspect_ratio,person_generation
"2024-06-14T20:23:20","MichaelClervi@oliver.agency","three adults telling jokes on the beach at sunset","8","16:9","allow_adult"
"2024-06-14T21:15:30","user@example.com","a cat playing piano","5","16:9","allow_adult"