# Wrike Import Tools

Python tools for automatically importing project structures and deliverable tasks into Wrike from JSON files.

## Tools Overview

### 1. wrike_monitor.py - Real-time Monitor Service (Staging)
A continuous monitoring service that watches a folder 24/7, processes files as they arrive, and sends daily email reports. Designed to run as a systemd service on the **Staging board**.

### 2. wrike_monitor_lgl.py - Real-time Monitor Service (LGL Team - Production)
Production version configured for the **LGL Team board**. Features recursive duplicate detection across the entire Business Areas folder and stores OMG# as HTML links matching the existing board format.

### 3. discover_board_info.py - Board Discovery Tool
Automated tool to discover Space IDs, custom field IDs, and item types for any Wrike board. Generates configuration snippets for easy setup.

## Common Features

Both tools create the following Wrike structure:
- **Folders** (product categories like "Dry Specialty", "Air", "PDC")
- **Projects** (campaigns with timelines and metadata)
- **Deliverables** (created as projects with custom fields and dates)

Both intelligently handle duplicates by checking OMG numbers before creating new items.

### Known Limitations

⚠️ **Custom Item Type Display**: Due to Wrike API v4 limitations, deliverables are created with the correct structure and fields but may display as "Project" type in the UI instead of "Deliverable" type. The `customItemTypeId` parameter is set in the API, but Wrike UI doesn't always recognize it. If needed, you can manually change the item type in the Wrike UI after creation. All functionality (fields, dates, duplicate detection) works correctly regardless of display type.

## Features Comparison

### wrike_import.py (Batch Script)
- ✅ Simple command-line usage
- ✅ Process all files and exit
- ✅ Good for one-time imports or manual runs
- ✅ Moves processed files to Processed/
- ✅ Auto-cleanup of old files

### wrike_monitor.py (Service)
- ✅ **Real-time folder monitoring** with watchdog
- ✅ **Automatic processing** as files arrive
- ✅ **Daily email reports** at 7PM
- ✅ **Periodic scanning** for missed files
- ✅ **Failed file handling** with error logs
- ✅ **Auto-cleanup** of processed files (24h)
- ✅ **Statistics tracking** and performance monitoring
- ✅ **Systemd service** support for production
- ✅ **Sequential processing** to prevent race conditions

### Common Features (Both)
- ✅ **Automatic folder creation** from BusinessArea hierarchy
- ✅ **Project management** with start/end dates and campaign codes
- ✅ **Deliverable creation** as projects with comprehensive custom fields
- ✅ **OMG# field** with plain job number for duplicate detection
- ✅ **OMG URL field** with clickable link to OMG system
- ✅ **Duplicate prevention** via OMG number checking
- ✅ **Smart caching** to minimize API calls
- ✅ **Detailed logging** with progress tracking

## Quick Start

### For Batch Processing (wrike_import.py)
```bash
# Install dependencies
pip install requests

# Run on a folder
python wrike_import.py /path/to/json/files/
```

### For Real-time Monitoring (wrike_monitor.py)
```bash
# Install dependencies
pip install -r requirements.txt

# Configure paths in wrike_monitor.py (lines 41-44)
# Run the monitor
python wrike_monitor.py
```

See [INSTALLATION.md](INSTALLATION.md) for systemd service setup.

## Requirements

### Python Dependencies

**For wrike_import.py:**
```bash
pip install requests
```

**For wrike_monitor.py:**
```bash
pip install requests watchdog schedule
# Or use requirements.txt:
pip install -r requirements.txt
```

### Python Version
- Python 3.6 or higher

### Wrike Requirements
- Valid Wrike API token with write permissions
- Access to the target Wrike space
- Custom fields configured in the space (see Configuration section)

## Installation

1. Clone or download this repository
2. Install dependencies:
   ```bash
   pip install requests
   ```
3. Configure your Wrike API token in the script (see Configuration)

## Configuration

### 1. API Token

Edit `wrike_import.py` and update the `WRIKE_TOKEN` constant:

```python
WRIKE_TOKEN = "your_wrike_api_token_here"
```

### 2. Target Space

Update the `STAGING_SPACE_ID` to point to your target Wrike space:

```python
STAGING_SPACE_ID = "MQAAAABpz7l_"  # Your space ID
```

To find your space ID:
```bash
curl -X GET "https://www.wrike.com/api/v4/spaces" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

### 3. Custom Fields

The script expects the following custom fields to exist in your Wrike space:

| Field Name | Type | ID Variable | Usage |
|------------|------|-------------|-------|
| Budget | Currency | `budget` | Project budgets |
| Impact | Dropdown | `impact` | Priority level |
| Notes | Text | `notes` | Additional information |
| RAG | Dropdown | `rag` | Status (Red/Amber/Green) |
| Deliverable Category | Dropdown | `deliverable_category` | Type of deliverable |
| Actions | Text | `actions` | Next steps |
| Shoot date | Date | `shoot_date` | Photography date |
| OMG # | Text | `omg_number` | Job/Campaign number |
| Box Link | Text | `box_link` | Link to assets |
| Owner | Contacts | `owner` | Task owner |

To create these fields, use the Wrike API or create them manually in the Wrike interface.

**Update custom field IDs** in the script after creating them:

```python
CUSTOM_FIELDS = {
    "budget": "YOUR_BUDGET_FIELD_ID",
    "impact": "YOUR_IMPACT_FIELD_ID",
    # ... etc
}
```

## JSON File Format

### Required Structure

```json
{
  "JobSpecification": {
    "ProjectDetails": {
      "BusinessArea": "BISSELL > PRODUCT MARKETING > Dry Specialty",
      "Title": "PowerClean FurFinder + FurGuard 2025",
      "Description": "<p>PowerClean FurFinder + FurGuard 2025-related workflows</p>",
      "StartDate": "2025-04-08 13:00:00+00",
      "EndDate": "2025-12-31 17:06:00+00"
    },
    "JobDetails": {
      "Number": "5791330",
      "Title": "PowerClean Corded Mass User Guide",
      "CampaignCode": "1647476",
      "JobCategory": "UX User Guide",
      "MediaType": "Creative development",
      "Type": "Creative development",
      "Notes": "User guide for mass market",
      "BriefDate": "2025-05-23 16:00:00+00",
      "DueDate": "2025-09-17 16:04:00+00",
      "BusinessArea": "BISSELL > PRODUCT MARKETING > Dry Specialty"
    }
  }
}
```

### Field Mapping

#### Project Level
| JSON Field | Maps To | Notes |
|------------|---------|-------|
| `ProjectDetails.BusinessArea` | Folder name | Last segment (e.g., "Dry Specialty") |
| `ProjectDetails.Title` | Project title | |
| `ProjectDetails.Description` | Project description | HTML tags removed |
| `ProjectDetails.StartDate` | Project start date | Converted to YYYY-MM-DD |
| `ProjectDetails.EndDate` | Project end date | Converted to YYYY-MM-DD |
| `JobDetails.CampaignCode` | Project OMG # | Custom field |

#### Task/Deliverable Level
| JSON Field | Maps To | Notes |
|------------|---------|-------|
| `JobDetails.Number` | Task OMG # | Used for duplicate detection |
| `JobDetails.Title` | Task title | |
| `JobDetails.Notes` | Task description | |
| `JobDetails.JobCategory` | Deliverable Category | Custom field |
| `JobDetails.MediaType` | Deliverable Category | Fallback if JobCategory empty |
| `JobDetails.Type` | Notes field | Combined with Details |
| `JobDetails.BriefDate` | Task start date | |
| `JobDetails.DueDate` | Task due date | |

## Usage

### Basic Usage

Process all JSON files in a directory:

```bash
python wrike_import.py /path/to/json/files/
```

### Example

```bash
# Process files in the current directory
python wrike_import.py ./jobs/

# Process files in a specific folder
python wrike_import.py ~/Documents/wrike_imports/
```

### Output Example

```
Found 3 JSON file(s) to process
Target: Wrike Staging Space (MQAAAABpz7l_)

================================================================================
Processing: 5791330.json
================================================================================

1. Processing folder: 'Dry Specialty'
  Found existing folder 'Dry Specialty': MQAAAABpz123

2. Processing project: 'PowerClean FurFinder + FurGuard 2025'
   Campaign Code: 1647476
  Found existing project 'PowerClean FurFinder + FurGuard 2025': MQAAAABpz456

3. Processing deliverable task
  Checking for existing task with OMG #: 5791330
  Creating new task 'PowerClean Corded Mass User Guide' (Job #5791330)
  ✓ Created task 'PowerClean Corded Mass User Guide': MAAAAABpz789

✓ Successfully processed 5791330.json
  → Moved to: Processed/5791330.json

================================================================================
Processing: 5791331.json
================================================================================

1. Processing folder: 'Air'
  Found existing folder 'Air': MQAAAABpz124

2. Processing project: 'Air Purifier 2025'
   Campaign Code: 1647477
  Found existing project 'Air Purifier 2025': MQAAAABpz457

3. Processing deliverable task
  Checking for existing task with OMG #: 5791331
  Found existing task: MAAAAABpz790
  ⊙ Task 'Air Purifier Hero Image' already exists (Job #5791331) - skipping

⊙ Successfully processed 5791331.json (task already exists)
  → Moved to: Processed/5791331.json

================================================================================
CLEANUP
================================================================================
  Deleted old file: 5791329.json
  Deleted old file: 5791328.json
Deleted 2 file(s) older than 24 hours from Processed folder

================================================================================
SUMMARY
================================================================================
Total files: 3
Successful: 3
Skipped (already exists): 1
Failed: 0
Moved to Processed: 3

Folders created/found: 2
Projects created/found: 3
```

## How It Works

### 1. Folder Management
- Extracts the last segment from `BusinessArea` (e.g., "BISSELL > PRODUCT MARKETING > Dry Specialty" → "Dry Specialty")
- Checks if folder exists in the target space
- Creates folder if not found
- Caches folder IDs for performance

### 2. Project Management
- Searches for existing project by title in the folder
- Creates new project if not found
- Converts folder to project with start/end dates
- Adds campaign code as OMG # custom field
- Caches project IDs for performance

### 3. Task Management
- Checks for existing task by OMG # (job number)
- **If found**: Updates existing task with new data
- **If not found**: Creates new task
- Populates all custom fields and dates

### 4. Duplicate Handling

The script prevents duplicates using OMG numbers:

- **Projects**: Matched by title within folder
- **Tasks**: Matched by OMG # custom field (job number)

When processing the same JSON file twice:
- 1st run: Creates folder, project, and task
- 2nd run:
  - Finds existing folder and project (reuses them)
  - Finds existing task with same OMG # → **Skips task creation**
  - Marks file as "skipped" and moves to Processed folder

**Important**: Tasks are never updated once created. If a task with the same OMG # exists, it's left unchanged.

### 5. File Management

After successful processing:
- **Moves file** to `Processed/` subfolder (auto-created)
- **Failed files** remain in the source directory for retry
- **Cleanup**: Deletes files older than 24 hours from Processed folder

Directory structure:
```
json_files/
├── 5791331.json          # Pending
├── 5791332.json          # Pending
└── Processed/
    ├── 5791330.json      # Processed today (kept)
    └── 5791329.json      # Processed >24h ago (deleted)
```

## Troubleshooting

### Common Issues

#### 1. Authentication Error
```
Error making Wrike request: 401 Unauthorized
```
**Solution**: Verify your API token is correct and has write permissions.

#### 2. Custom Field Not Found
```
Error: Custom field ID not found
```
**Solution**: Ensure all custom fields are created in Wrike and IDs are updated in the script.

#### 3. No JSON Files Found
```
No JSON files found in '/path/to/directory'
```
**Solution**:
- Verify the directory path is correct
- Ensure files have `.json` extension
- Check file permissions

#### 4. Invalid Space ID
```
Error: Space 'MQAAAABpz7l_' not found
```
**Solution**:
- Get your space ID from the API: `GET /spaces`
- Update `STAGING_SPACE_ID` in the script

### Debug Mode

To see detailed API responses, modify the `make_wrike_request` function to print full responses:

```python
def make_wrike_request(method, endpoint, data=None):
    # ... existing code ...
    print(f"Response: {response.json()}")  # Add this line
    return response.json()
```

## API Rate Limits

Wrike API has rate limits:
- **100 requests per minute** per token
- **1000 requests per hour** per token

The script is optimized with caching to minimize API calls, but for very large batches (100+ files), you may need to:
1. Process in smaller batches
2. Add delays between requests

## Data Mapping Reference

### Date Format Conversion
Input: `"2025-05-23 16:00:00+00"`
Output: `"2025-05-23"`

### HTML Cleaning
Input: `"<p>PowerClean FurFinder + FurGuard 2025-related workflows</p>"`
Output: `"PowerClean FurFinder + FurGuard 2025-related workflows"`

### Business Area Parsing
Input: `"BISSELL > PRODUCT MARKETING > Dry Specialty"`
Output: `"Dry Specialty"` (folder name)

## Advanced Usage

### Custom Field Mapping

To add more custom fields, update the script:

1. Add field ID to `CUSTOM_FIELDS` dictionary
2. Update `create_or_update_deliverable_task()` function:

```python
# Add your custom field
custom_fields.append({
    "id": CUSTOM_FIELDS["your_field_name"],
    "value": job_details.get("YourJsonField", "")
})
```

### Filter by Business Area

To process only specific business areas, add a filter in `process_json_file()`:

```python
folder_name = parse_business_area(business_area)

# Only process specific folders
if folder_name not in ["Dry Specialty", "Air"]:
    print(f"  Skipping folder '{folder_name}'")
    return False
```

### Batch Processing Script

Create a wrapper script for automated processing:

```bash
#!/bin/bash
# process_all.sh

IMPORT_DIR="/path/to/json/files"

# Run import (automatically moves to Processed/ and cleans up old files)
python wrike_import.py "$IMPORT_DIR"

# Optional: Send notification
if [ $? -eq 0 ]; then
    echo "Wrike import completed successfully"
fi
```

### Scheduled Processing with Cron

Set up automatic processing every hour:

```bash
# Edit crontab
crontab -e

# Add this line to run every hour
0 * * * * /usr/bin/python3 /path/to/wrike_import.py /path/to/json/files >> /var/log/wrike_import.log 2>&1
```

### Change Cleanup Retention Period

To change the 24-hour retention to a different period, modify the cleanup call in the script:

```python
# Delete files older than 48 hours instead
deleted_count = cleanup_old_files(json_dir, hours=48)

# Delete files older than 7 days
deleted_count = cleanup_old_files(json_dir, hours=168)
```

## Security Notes

⚠️ **Important Security Considerations:**

1. **Never commit API tokens** to version control
2. Store tokens in environment variables:
   ```python
   import os
   WRIKE_TOKEN = os.environ.get('WRIKE_API_TOKEN')
   ```
3. Use a `.env` file for local development (add to `.gitignore`)
4. Rotate tokens regularly
5. Use read-only tokens for testing

## Support

### Getting Help

1. Check the [Wrike API Documentation](https://developers.wrike.com/api/v4/)
2. Review error messages in script output
3. Enable debug mode for detailed logging
4. Check Wrike API status at [status.wrike.com](https://status.wrike.com)

### Reporting Issues

When reporting issues, include:
- Script version
- Python version (`python --version`)
- Full error message
- Sample JSON structure (redacted)
- Wrike space configuration

## License

This script is provided as-is for internal use.

## Which Tool Should I Use?

### Use wrike_import.py if you want to:
- Import a batch of JSON files once
- Run manually when needed
- Test imports locally
- Simple command-line operation

### Use wrike_monitor.py if you want to:
- Continuous 24/7 monitoring
- Automatic processing as files arrive
- Daily email reports
- Production deployment as a service
- Detailed statistics and monitoring

## LGL Team Production Version (wrike_monitor_lgl.py)

### Key Differences from Staging Version

**OMG# Format:**
- LGL Team stores OMG# as **HTML links** instead of plain text
- Format: `<a href="https://bissell.omg.oliver.solutions/projects/{id}">{number}</a>`
- Script automatically generates and parses HTML format

**Duplicate Detection:**
- **Recursive search** across entire "Business Areas" folder
- Uses `descendants=true` API parameter for efficient searching
- Searches **ALL subfolders** at any depth
- If OMG# exists anywhere in Business Areas → Skip (no duplicate)
- Other folders (Templates, Production Tracker, etc.) are ignored

**Configuration:**
- API Token: Updated for production access
- Space: "LGL Team" (MQAAAABoHcTY)
- All 11 custom fields mapped to LGL Team IDs
- Custom item types not required (set to None)

**Running the LGL Team Version:**
```bash
# Install dependencies
pip install -r requirements.txt

# Configure paths in wrike_monitor_lgl.py (lines 41-44)
# Run the monitor
python wrike_monitor_lgl.py
```

### Discovery Tool Usage

To discover configuration for any board:

```bash
# Run discovery tool
python discover_board_info.py

# Edit TARGET_SPACE in the script to discover different boards
# Generates config_{space_name}.py with all IDs
```

## Changelog

### Version 3.0 (December 2025) - LGL Team Migration
- **NEW**: Added wrike_monitor_lgl.py for LGL Team (production) board
- **NEW**: HTML link format for OMG# field (matching existing entries)
- **NEW**: Recursive duplicate detection across entire Business Areas folder
- **NEW**: discover_board_info.py - automated board configuration discovery
- **NEW**: HTML extraction and comparison for accurate duplicate detection
- **NEW**: Global OMG# uniqueness within Business Areas folder
- **IMPROVED**: Duplicate detection now searches recursively, not just parent project
- **IMPROVED**: OMG# format matches existing board entries (clickable links)
- **FIXED**: Handles OMG# stored as HTML `<a>` tags
- **ADDED**: config_lgl_team.py - LGL Team configuration reference
- **ADDED**: test_duplicate_detection.py - testing tool

### Version 2.0
- **NEW**: Added wrike_monitor.py - real-time monitoring service
- **NEW**: Folder watching with watchdog
- **NEW**: Daily email reports at 7PM
- **NEW**: Failed file handling with error logs
- **NEW**: Periodic scanning for missed files
- **NEW**: Statistics tracking and performance monitoring
- **NEW**: systemd service support
- **FIXED**: Sequential processing to prevent race conditions
- **FIXED**: Proper parent/child folder matching to avoid duplicates
- **IMPROVED**: Logging for skipped deliverables

### Version 1.2
- Added automatic file movement to Processed subfolder
- Implemented auto-cleanup of files older than 24 hours
- Improved file management and organization
- Enhanced output with file movement tracking

### Version 1.1
- Added duplicate detection via OMG numbers
- Separated project and task OMG numbers
- Improved error handling
- Added caching for performance
- Enhanced logging output

### Version 1.0
- Initial release
- Basic folder/project/task creation
- Custom field mapping

---

**Last Updated**: October 2025
**Author**: Dave Porter
**Repository**: https://bitbucket.org/zlalani/bissell-wrike-python