Change token usage alert to 7-day rolling window and convert daily digest to weekly Monday digest
- High token usage notification now sums usage_timeline tokens over last 7 days instead of checking lifetime total - Daily agent digest converted to weekly digest, scheduled for Monday mornings - Email subjects and templates updated to reflect weekly timeframes - Env var DAILY_DIGEST_HOUR renamed to WEEKLY_DIGEST_HOUR Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
nickviljoen
parent
6c231cb094
commit
f7e88513b1
3 changed files with 80 additions and 64 deletions
27
CLAUDE.md
27
CLAUDE.md
|
|
@ -31,10 +31,10 @@ Create `.env` file with:
|
|||
- `MAILGUN_API_KEY`: Mailgun API key (if not set, notifications are silently disabled)
|
||||
- `MAILGUN_DOMAIN`: Mailgun sending domain (e.g. your-domain.mailgun.org)
|
||||
- `MAILGUN_FROM_EMAIL`: Sender address (default: `AgentHub <noreply@{MAILGUN_DOMAIN}>`)
|
||||
- `TOKEN_USAGE_THRESHOLD`: Token count that triggers an alert (default: 100000)
|
||||
- `TOKEN_USAGE_THRESHOLD`: Weekly token count that triggers an alert (default: 100000)
|
||||
- `NOTIFICATION_COOLDOWN_HOURS`: Hours between repeat alerts for the same agent (default: 24)
|
||||
- `CLIENT_AGENT_NOTIFY_EMAILS`: Comma-separated list of emails for client agent notifications
|
||||
- `DAILY_DIGEST_HOUR`: Hour (24h format) to send daily digest (default: 7)
|
||||
- `WEEKLY_DIGEST_HOUR`: Hour (24h format) to send weekly digest on Mondays (default: 7)
|
||||
|
||||
### Default Login Credentials
|
||||
- Admin: `admin@agenthub.com` / `admin123`
|
||||
|
|
@ -84,11 +84,11 @@ Create `.env` file with:
|
|||
- `is_mailgun_configured()`: Returns False if env vars not set (gracefully disabled)
|
||||
- `send_mailgun_email()`: POST to Mailgun HTTP API with 10s timeout
|
||||
- `build_threshold_email()`: HTML email template for threshold alerts
|
||||
- `check_and_notify_threshold()`: Checks token usage against threshold, enforces cooldown via `token_notifications` collection, sends to admin users
|
||||
- `check_and_notify_threshold()`: Checks 7-day token usage against threshold, enforces cooldown via `token_notifications` collection, sends to admin users
|
||||
- `send_client_agent_notification()`: Sends email when client-facing agent is created (to `CLIENT_AGENT_NOTIFY_EMAILS`)
|
||||
- `build_client_agent_email()`: HTML email template for client agent notifications
|
||||
- `send_daily_agent_digest()`: Queries agents created in last 24 hours, sends summary to all admin users
|
||||
- `build_daily_digest_email()`: HTML email template for daily digest
|
||||
- `send_weekly_agent_digest()`: Queries agents created in last 7 days, sends summary to all admin users
|
||||
- `build_weekly_digest_email()`: HTML email template for weekly digest
|
||||
|
||||
**auth.py**: JWT authentication with:
|
||||
- bcrypt password hashing
|
||||
|
|
@ -154,12 +154,12 @@ Located in `templates/` directory:
|
|||
- Body includes: Agent Name, Description, Purpose, Client Name, Studio Name, Tool, Created By
|
||||
- Non-blocking: failure does not break agent creation
|
||||
|
||||
### Daily Agent Digest Email
|
||||
- Scheduled via APScheduler at configured hour (default 7:00 AM, `DAILY_DIGEST_HOUR` env var)
|
||||
- Queries agents created in last 24 hours
|
||||
### Weekly Agent Digest Email
|
||||
- Scheduled via APScheduler to run every Monday morning (default 7:00 AM, `WEEKLY_DIGEST_HOUR` env var)
|
||||
- Queries agents created in last 7 days
|
||||
- Sends to all active admin users
|
||||
- Body includes: Agent Name, Purpose, Description, Created By (email)
|
||||
- Subject: "Agents Created Last 24 Hours"
|
||||
- Subject: "Agents Created in Last Week"
|
||||
- Skips sending if no agents created
|
||||
- Can be manually triggered via `POST /api/admin/digest/send` (admin-only)
|
||||
|
||||
|
|
@ -194,7 +194,8 @@ Located in `templates/` directory:
|
|||
|
||||
### High Usage Email Notifications
|
||||
- Entirely optional — silently disabled when Mailgun env vars are not set
|
||||
- Triggered from the Agent Collector endpoint (POST `/agents`) when `total_tokens` exceeds threshold
|
||||
- Triggered from the Agent Collector endpoint (POST `/agents`); checks 7-day rolling token usage from `usage_timeline`
|
||||
- Alerts when weekly token usage exceeds threshold (default 100,000, configurable via `TOKEN_USAGE_THRESHOLD`)
|
||||
- Non-blocking — notification failure never breaks the collector API
|
||||
- Cooldown tracking in MongoDB `token_notifications` collection (default 24h, configurable)
|
||||
- Sends to all active admin users' email addresses
|
||||
|
|
@ -260,7 +261,7 @@ Key dependencies from requirements.txt:
|
|||
- **jinja2**: Template engine
|
||||
- **python-multipart**: Form handling
|
||||
- **requests**: HTTP client (used for Mailgun API calls)
|
||||
- **apscheduler**: Task scheduling (daily digest email)
|
||||
- **apscheduler**: Task scheduling (weekly digest email)
|
||||
|
||||
## API Endpoints (New)
|
||||
|
||||
|
|
@ -268,5 +269,5 @@ Key dependencies from requirements.txt:
|
|||
- `GET /api/admin/agents/pending-verification` — List agents with verification status (admin + readonly_admin)
|
||||
- `PUT /api/admin/agents/{agent_id}/verify` — Approve/verify an agent (admin only)
|
||||
|
||||
### Daily Digest
|
||||
- `POST /api/admin/digest/send` — Manually trigger the daily agent digest email (admin only)
|
||||
### Weekly Digest
|
||||
- `POST /api/admin/digest/send` — Manually trigger the weekly agent digest email (admin only)
|
||||
43
main.py
43
main.py
|
|
@ -343,22 +343,23 @@ async def startup_event():
|
|||
if count > 0:
|
||||
print(f"Pencil Agents migration: updated {count} agent(s)")
|
||||
|
||||
# Start daily digest scheduler
|
||||
# Start weekly digest scheduler (runs Monday mornings)
|
||||
try:
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
digest_hour = int(os.getenv("DAILY_DIGEST_HOUR", "7"))
|
||||
digest_hour = int(os.getenv("WEEKLY_DIGEST_HOUR", "7"))
|
||||
scheduler = AsyncIOScheduler()
|
||||
scheduler.add_job(
|
||||
notifications.send_daily_agent_digest,
|
||||
notifications.send_weekly_agent_digest,
|
||||
'cron',
|
||||
day_of_week='mon',
|
||||
hour=digest_hour,
|
||||
minute=0,
|
||||
id='daily_agent_digest',
|
||||
id='weekly_agent_digest',
|
||||
)
|
||||
scheduler.start()
|
||||
print(f"Daily digest scheduler started (runs at {digest_hour}:00)")
|
||||
print(f"Weekly digest scheduler started (runs Monday at {digest_hour}:00)")
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to start daily digest scheduler: {e}")
|
||||
print(f"Warning: Failed to start weekly digest scheduler: {e}")
|
||||
|
||||
# HTML Routes
|
||||
@app.get("/")
|
||||
|
|
@ -1252,11 +1253,11 @@ async def verify_agent(agent_id: str, current_user: dict = Depends(require_admin
|
|||
raise HTTPException(status_code=500, detail="Failed to verify agent")
|
||||
|
||||
@app.post("/api/admin/digest/send")
|
||||
async def trigger_daily_digest(current_user: dict = Depends(require_admin)):
|
||||
"""Manually trigger the daily agent digest email"""
|
||||
async def trigger_weekly_digest(current_user: dict = Depends(require_admin)):
|
||||
"""Manually trigger the weekly agent digest email"""
|
||||
try:
|
||||
await notifications.send_daily_agent_digest()
|
||||
return {"message": "Daily digest sent successfully"}
|
||||
await notifications.send_weekly_agent_digest()
|
||||
return {"message": "Weekly digest sent successfully"}
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to send digest: {str(e)}")
|
||||
|
||||
|
|
@ -1639,12 +1640,11 @@ async def create_agent_collector(
|
|||
)
|
||||
print(f"🔗 URL DEBUG: Agent '{agent.name}' - Update executed: matched={result.matched_count}, modified={result.modified_count}")
|
||||
|
||||
# Check token threshold and send notification if needed (non-blocking)
|
||||
if internal_data.get("total_tokens"):
|
||||
try:
|
||||
await notifications.check_and_notify_threshold(agent.name, internal_data["total_tokens"])
|
||||
except Exception as notify_err:
|
||||
print(f"Notification check failed for '{agent.name}': {notify_err}")
|
||||
# Check weekly token threshold and send notification if needed (non-blocking)
|
||||
try:
|
||||
await notifications.check_and_notify_threshold(agent.name)
|
||||
except Exception as notify_err:
|
||||
print(f"Notification check failed for '{agent.name}': {notify_err}")
|
||||
|
||||
return models.AgentUsageTrackingResponse(
|
||||
status="usage_logged",
|
||||
|
|
@ -1664,12 +1664,11 @@ async def create_agent_collector(
|
|||
# Create agent using collector-specific function
|
||||
created_agent = await crud.create_agent_from_collector(internal_data)
|
||||
|
||||
# Check token threshold and send notification if needed (non-blocking)
|
||||
if internal_data.get("total_tokens"):
|
||||
try:
|
||||
await notifications.check_and_notify_threshold(agent.name, internal_data["total_tokens"])
|
||||
except Exception as notify_err:
|
||||
print(f"Notification check failed for '{agent.name}': {notify_err}")
|
||||
# Check weekly token threshold and send notification if needed (non-blocking)
|
||||
try:
|
||||
await notifications.check_and_notify_threshold(agent.name)
|
||||
except Exception as notify_err:
|
||||
print(f"Notification check failed for '{agent.name}': {notify_err}")
|
||||
|
||||
return models.AgentCollectorResponse(
|
||||
status="success",
|
||||
|
|
|
|||
|
|
@ -29,32 +29,32 @@ def send_mailgun_email(to_emails: list[str], subject: str, html_body: str) -> bo
|
|||
return response.status_code == 200
|
||||
|
||||
|
||||
def build_threshold_email(agent_name: str, total_tokens: int, threshold: int) -> str:
|
||||
"""Build HTML email body for a token threshold notification."""
|
||||
def build_threshold_email(agent_name: str, weekly_tokens: int, threshold: int) -> str:
|
||||
"""Build HTML email body for a weekly token threshold notification."""
|
||||
return f"""
|
||||
<html>
|
||||
<body style="font-family: Arial, sans-serif; max-width: 600px; margin: 0 auto;">
|
||||
<div style="background: linear-gradient(135deg, #f3ae3e, #e09520); padding: 20px; border-radius: 8px 8px 0 0;">
|
||||
<h2 style="color: white; margin: 0;">AgentHub - High Token Usage Alert</h2>
|
||||
<h2 style="color: white; margin: 0;">AgentHub - High Weekly Token Usage Alert</h2>
|
||||
</div>
|
||||
<div style="padding: 20px; border: 1px solid #e2e8f0; border-top: none; border-radius: 0 0 8px 8px;">
|
||||
<p>The following agent has exceeded the token usage threshold:</p>
|
||||
<p>The following agent has exceeded the weekly token usage threshold:</p>
|
||||
<table style="width: 100%; border-collapse: collapse; margin: 16px 0;">
|
||||
<tr>
|
||||
<td style="padding: 8px; font-weight: bold; border-bottom: 1px solid #eee;">Agent Name</td>
|
||||
<td style="padding: 8px; border-bottom: 1px solid #eee;">{agent_name}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="padding: 8px; font-weight: bold; border-bottom: 1px solid #eee;">Total Tokens</td>
|
||||
<td style="padding: 8px; border-bottom: 1px solid #eee;">{total_tokens:,}</td>
|
||||
<td style="padding: 8px; font-weight: bold; border-bottom: 1px solid #eee;">Tokens (Last 7 Days)</td>
|
||||
<td style="padding: 8px; border-bottom: 1px solid #eee;">{weekly_tokens:,}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="padding: 8px; font-weight: bold; border-bottom: 1px solid #eee;">Threshold</td>
|
||||
<td style="padding: 8px; font-weight: bold; border-bottom: 1px solid #eee;">Weekly Threshold</td>
|
||||
<td style="padding: 8px; border-bottom: 1px solid #eee;">{threshold:,}</td>
|
||||
</tr>
|
||||
</table>
|
||||
<p style="color: #666; font-size: 0.9em;">
|
||||
This is an automated notification from AgentHub. Please review the agent's token consumption.
|
||||
This is an automated notification from AgentHub. Please review the agent's recent token consumption.
|
||||
</p>
|
||||
</div>
|
||||
</body>
|
||||
|
|
@ -62,13 +62,29 @@ def build_threshold_email(agent_name: str, total_tokens: int, threshold: int) ->
|
|||
"""
|
||||
|
||||
|
||||
async def check_and_notify_threshold(agent_name: str, total_tokens: int):
|
||||
"""Check token threshold and send notification if exceeded. Non-blocking, safe to call."""
|
||||
async def check_and_notify_threshold(agent_name: str):
|
||||
"""Check 7-day token usage against threshold and send notification if exceeded. Non-blocking, safe to call."""
|
||||
if not is_mailgun_configured():
|
||||
return
|
||||
|
||||
threshold = int(os.getenv("TOKEN_USAGE_THRESHOLD", "100000"))
|
||||
if total_tokens < threshold:
|
||||
|
||||
# Calculate token usage over the last 7 days from usage_timeline
|
||||
agent = await agents_collection.find_one(
|
||||
{"agent_name": agent_name},
|
||||
{"usage_timeline": 1}
|
||||
)
|
||||
if not agent or not agent.get("usage_timeline"):
|
||||
return
|
||||
|
||||
cutoff_date = (datetime.utcnow() - timedelta(days=7)).strftime("%Y-%m-%d")
|
||||
weekly_tokens = sum(
|
||||
entry.get("token_count", 0)
|
||||
for entry in agent["usage_timeline"]
|
||||
if entry.get("date", "") >= cutoff_date
|
||||
)
|
||||
|
||||
if weekly_tokens < threshold:
|
||||
return
|
||||
|
||||
cooldown_hours = int(os.getenv("NOTIFICATION_COOLDOWN_HOURS", "24"))
|
||||
|
|
@ -92,14 +108,14 @@ async def check_and_notify_threshold(agent_name: str, total_tokens: int):
|
|||
return
|
||||
|
||||
# Send email
|
||||
subject = f"[AgentHub] High Token Usage: {agent_name} ({total_tokens:,} tokens)"
|
||||
html_body = build_threshold_email(agent_name, total_tokens, threshold)
|
||||
subject = f"[AgentHub] High Weekly Token Usage: {agent_name} ({weekly_tokens:,} tokens in 7 days)"
|
||||
html_body = build_threshold_email(agent_name, weekly_tokens, threshold)
|
||||
success = send_mailgun_email(admin_emails, subject, html_body)
|
||||
|
||||
# Record the notification attempt
|
||||
await notifications_collection.insert_one({
|
||||
"agent_name": agent_name,
|
||||
"total_tokens": total_tokens,
|
||||
"weekly_tokens": weekly_tokens,
|
||||
"threshold": threshold,
|
||||
"recipients": admin_emails,
|
||||
"success": success,
|
||||
|
|
@ -181,8 +197,8 @@ def send_client_agent_notification(agent_data: dict):
|
|||
print(f"Failed to send client agent notification: {e}")
|
||||
|
||||
|
||||
def build_daily_digest_email(agents: list) -> str:
|
||||
"""Build HTML email body for the daily agent digest."""
|
||||
def build_weekly_digest_email(agents: list) -> str:
|
||||
"""Build HTML email body for the weekly agent digest."""
|
||||
rows = ""
|
||||
for i, agent in enumerate(agents, 1):
|
||||
rows += f"""
|
||||
|
|
@ -200,10 +216,10 @@ def build_daily_digest_email(agents: list) -> str:
|
|||
<html>
|
||||
<body style="font-family: Arial, sans-serif; max-width: 600px; margin: 0 auto;">
|
||||
<div style="background: linear-gradient(135deg, #2d6a4f, #40916c); padding: 20px; border-radius: 8px 8px 0 0;">
|
||||
<h2 style="color: white; margin: 0;">Agents Created Last 24 Hours</h2>
|
||||
<h2 style="color: white; margin: 0;">Agents Created in Last Week</h2>
|
||||
</div>
|
||||
<div style="padding: 20px; border: 1px solid #e2e8f0; border-top: none; border-radius: 0 0 8px 8px;">
|
||||
<p>{len(agents)} agent(s) created in the last 24 hours:</p>
|
||||
<p>{len(agents)} agent(s) created in the last week:</p>
|
||||
<table style="width: 100%; border-collapse: collapse; margin: 16px 0;">
|
||||
{rows}
|
||||
</table>
|
||||
|
|
@ -216,15 +232,15 @@ def build_daily_digest_email(agents: list) -> str:
|
|||
"""
|
||||
|
||||
|
||||
async def send_daily_agent_digest():
|
||||
"""Send daily digest of agents created in the last 24 hours to all admins."""
|
||||
async def send_weekly_agent_digest():
|
||||
"""Send weekly digest of agents created in the last 7 days to all admins."""
|
||||
if not is_mailgun_configured():
|
||||
print("Daily digest: Mailgun not configured, skipping.")
|
||||
print("Weekly digest: Mailgun not configured, skipping.")
|
||||
return
|
||||
|
||||
cutoff = datetime.utcnow() - timedelta(hours=24)
|
||||
cutoff = datetime.utcnow() - timedelta(days=7)
|
||||
|
||||
# Find agents created in last 24 hours
|
||||
# Find agents created in last 7 days
|
||||
cursor = agents_collection.find(
|
||||
{"created_at": {"$gte": cutoff}},
|
||||
{"agent_name": 1, "agent_purpose": 1, "agent_description": 1, "created_by": 1}
|
||||
|
|
@ -232,7 +248,7 @@ async def send_daily_agent_digest():
|
|||
agents = await cursor.to_list(length=None)
|
||||
|
||||
if not agents:
|
||||
print("Daily digest: No agents created in last 24 hours, skipping.")
|
||||
print("Weekly digest: No agents created in last 7 days, skipping.")
|
||||
return
|
||||
|
||||
# Resolve creator emails
|
||||
|
|
@ -255,14 +271,14 @@ async def send_daily_agent_digest():
|
|||
)
|
||||
admin_emails = [doc["email"] async for doc in admin_cursor]
|
||||
if not admin_emails:
|
||||
print("Daily digest: No admin emails found, skipping.")
|
||||
print("Weekly digest: No admin emails found, skipping.")
|
||||
return
|
||||
|
||||
subject = "Agents Created Last 24 Hours"
|
||||
html_body = build_daily_digest_email(agents)
|
||||
subject = "Agents Created in Last Week"
|
||||
html_body = build_weekly_digest_email(agents)
|
||||
|
||||
try:
|
||||
success = send_mailgun_email(admin_emails, subject, html_body)
|
||||
print(f"Daily digest: Sent to {len(admin_emails)} admins, success={success}")
|
||||
print(f"Weekly digest: Sent to {len(admin_emails)} admins, success={success}")
|
||||
except Exception as e:
|
||||
print(f"Daily digest: Failed to send: {e}")
|
||||
print(f"Weekly digest: Failed to send: {e}")
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue