bissell-wrike-python/README.md

# Wrike Import Tools

Two Python tools for automatically importing project structures and deliverable tasks into Wrike from JSON files.

## Tools Overview

### 1. wrike_import.py - Simple Batch Processor
A standalone script for one-time or manual batch imports. Processes all JSON files in a directory and exits.

### 2. wrike_monitor.py - Real-time Monitor Service
A continuous monitoring service that watches a folder 24/7, processes files as they arrive, and sends daily email reports. Designed to run as a systemd service.

## Common Features

Both tools create the following Wrike structure:
- **Folders** (product categories like "Dry Specialty", "Air", "PDC")
- **Projects** (campaigns with timelines and metadata)
- **Tasks** (deliverables with custom fields and due dates)

Both intelligently handle duplicates by checking OMG numbers before creating new items.

## Features Comparison

### wrike_import.py (Batch Script)
- ✅ Simple command-line usage
- ✅ Process all files and exit
- ✅ Good for one-time imports or manual runs
- ✅ Moves processed files to Processed/
- ✅ Auto-cleanup of old files

### wrike_monitor.py (Service)
- ✅ **Real-time folder monitoring** with watchdog
- ✅ **Automatic processing** as files arrive
- ✅ **Daily email reports** at 7PM
- ✅ **Periodic scanning** for missed files
- ✅ **Failed file handling** with error logs
- ✅ **Auto-cleanup** of processed files (24h)
- ✅ **Statistics tracking** and performance monitoring
- ✅ **Systemd service** support for production
- ✅ **Sequential processing** to prevent race conditions

### Common Features (Both)
- ✅ **Automatic folder creation** from BusinessArea hierarchy
- ✅ **Project management** with start/end dates and campaign codes
- ✅ **Task creation** with comprehensive custom fields
- ✅ **Duplicate prevention** via OMG number checking
- ✅ **Smart caching** to minimize API calls
- ✅ **Detailed logging** with progress tracking

## Quick Start

### For Batch Processing (wrike_import.py)
```bash
# Install dependencies
pip install requests

# Run on a folder
python wrike_import.py /path/to/json/files/
```

### For Real-time Monitoring (wrike_monitor.py)
```bash
# Install dependencies
pip install -r requirements.txt

# Configure paths in wrike_monitor.py (lines 41-44)
# Run the monitor
python wrike_monitor.py
```

See [INSTALLATION.md](INSTALLATION.md) for systemd service setup.

## Requirements

### Python Dependencies

**For wrike_import.py:**
```bash
pip install requests
```

**For wrike_monitor.py:**
```bash
pip install requests watchdog schedule
# Or use requirements.txt:
pip install -r requirements.txt
```

### Python Version
- Python 3.6 or higher

### Wrike Requirements
- Valid Wrike API token with write permissions
- Access to the target Wrike space
- Custom fields configured in the space (see Configuration section)

## Installation

1. Clone or download this repository
2. Install dependencies:
   ```bash
   pip install requests
   ```
3. Configure your Wrike API token in the script (see Configuration)

## Configuration

### 1. API Token

Edit `wrike_import.py` and update the `WRIKE_TOKEN` constant:

```python
WRIKE_TOKEN = "your_wrike_api_token_here"
```

### 2. Target Space

Update the `STAGING_SPACE_ID` to point to your target Wrike space:

```python
STAGING_SPACE_ID = "MQAAAABpz7l_"  # Your space ID
```

To find your space ID:
```bash
curl -X GET "https://www.wrike.com/api/v4/spaces" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

### 3. Custom Fields

The script expects the following custom fields to exist in your Wrike space:

| Field Name | Type | ID Variable | Usage |
|------------|------|-------------|-------|
| Budget | Currency | `budget` | Project budgets |
| Impact | Dropdown | `impact` | Priority level |
| Notes | Text | `notes` | Additional information |
| RAG | Dropdown | `rag` | Status (Red/Amber/Green) |
| Deliverable Category | Dropdown | `deliverable_category` | Type of deliverable |
| Actions | Text | `actions` | Next steps |
| Shoot date | Date | `shoot_date` | Photography date |
| OMG # | Text | `omg_number` | Job/Campaign number |
| Box Link | Text | `box_link` | Link to assets |
| Owner | Contacts | `owner` | Task owner |

To create these fields, use the Wrike API or create them manually in the Wrike interface.

**Update custom field IDs** in the script after creating them:

```python
CUSTOM_FIELDS = {
    "budget": "YOUR_BUDGET_FIELD_ID",
    "impact": "YOUR_IMPACT_FIELD_ID",
    # ... etc
}
```

## JSON File Format

### Required Structure

```json
{
  "JobSpecification": {
    "ProjectDetails": {
      "BusinessArea": "BISSELL > PRODUCT MARKETING > Dry Specialty",
      "Title": "PowerClean FurFinder + FurGuard 2025",
      "Description": "<p>PowerClean FurFinder + FurGuard 2025-related workflows</p>",
      "StartDate": "2025-04-08 13:00:00+00",
      "EndDate": "2025-12-31 17:06:00+00"
    },
    "JobDetails": {
      "Number": "5791330",
      "Title": "PowerClean Corded Mass User Guide",
      "CampaignCode": "1647476",
      "JobCategory": "UX User Guide",
      "MediaType": "Creative development",
      "Type": "Creative development",
      "Notes": "User guide for mass market",
      "BriefDate": "2025-05-23 16:00:00+00",
      "DueDate": "2025-09-17 16:04:00+00",
      "BusinessArea": "BISSELL > PRODUCT MARKETING > Dry Specialty"
    }
  }
}
```

### Field Mapping

#### Project Level
| JSON Field | Maps To | Notes |
|------------|---------|-------|
| `ProjectDetails.BusinessArea` | Folder name | Last segment (e.g., "Dry Specialty") |
| `ProjectDetails.Title` | Project title | |
| `ProjectDetails.Description` | Project description | HTML tags removed |
| `ProjectDetails.StartDate` | Project start date | Converted to YYYY-MM-DD |
| `ProjectDetails.EndDate` | Project end date | Converted to YYYY-MM-DD |
| `JobDetails.CampaignCode` | Project OMG # | Custom field |

#### Task/Deliverable Level
| JSON Field | Maps To | Notes |
|------------|---------|-------|
| `JobDetails.Number` | Task OMG # | Used for duplicate detection |
| `JobDetails.Title` | Task title | |
| `JobDetails.Notes` | Task description | |
| `JobDetails.JobCategory` | Deliverable Category | Custom field |
| `JobDetails.MediaType` | Deliverable Category | Fallback if JobCategory empty |
| `JobDetails.Type` | Notes field | Combined with Details |
| `JobDetails.BriefDate` | Task start date | |
| `JobDetails.DueDate` | Task due date | |

## Usage

### Basic Usage

Process all JSON files in a directory:

```bash
python wrike_import.py /path/to/json/files/
```

### Example

```bash
# Process files in the current directory
python wrike_import.py ./jobs/

# Process files in a specific folder
python wrike_import.py ~/Documents/wrike_imports/
```

### Output Example

```
Found 3 JSON file(s) to process
Target: Wrike Staging Space (MQAAAABpz7l_)

================================================================================
Processing: 5791330.json
================================================================================

1. Processing folder: 'Dry Specialty'
  Found existing folder 'Dry Specialty': MQAAAABpz123

2. Processing project: 'PowerClean FurFinder + FurGuard 2025'
   Campaign Code: 1647476
  Found existing project 'PowerClean FurFinder + FurGuard 2025': MQAAAABpz456

3. Processing deliverable task
  Checking for existing task with OMG #: 5791330
  Creating new task 'PowerClean Corded Mass User Guide' (Job #5791330)
  ✓ Created task 'PowerClean Corded Mass User Guide': MAAAAABpz789

✓ Successfully processed 5791330.json
  → Moved to: Processed/5791330.json

================================================================================
Processing: 5791331.json
================================================================================

1. Processing folder: 'Air'
  Found existing folder 'Air': MQAAAABpz124

2. Processing project: 'Air Purifier 2025'
   Campaign Code: 1647477
  Found existing project 'Air Purifier 2025': MQAAAABpz457

3. Processing deliverable task
  Checking for existing task with OMG #: 5791331
  Found existing task: MAAAAABpz790
  ⊙ Task 'Air Purifier Hero Image' already exists (Job #5791331) - skipping

⊙ Successfully processed 5791331.json (task already exists)
  → Moved to: Processed/5791331.json

================================================================================
CLEANUP
================================================================================
  Deleted old file: 5791329.json
  Deleted old file: 5791328.json
Deleted 2 file(s) older than 24 hours from Processed folder

================================================================================
SUMMARY
================================================================================
Total files: 3
Successful: 3
Skipped (already exists): 1
Failed: 0
Moved to Processed: 3

Folders created/found: 2
Projects created/found: 3
```

## How It Works

### 1. Folder Management
- Extracts the last segment from `BusinessArea` (e.g., "BISSELL > PRODUCT MARKETING > Dry Specialty" → "Dry Specialty")
- Checks if folder exists in the target space
- Creates folder if not found
- Caches folder IDs for performance

### 2. Project Management
- Searches for existing project by title in the folder
- Creates new project if not found
- Converts folder to project with start/end dates
- Adds campaign code as OMG # custom field
- Caches project IDs for performance

### 3. Task Management
- Checks for existing task by OMG # (job number)
- **If found**: Updates existing task with new data
- **If not found**: Creates new task
- Populates all custom fields and dates

### 4. Duplicate Handling

The script prevents duplicates using OMG numbers:

- **Projects**: Matched by title within folder
- **Tasks**: Matched by OMG # custom field (job number)

When processing the same JSON file twice:
- 1st run: Creates folder, project, and task
- 2nd run:
  - Finds existing folder and project (reuses them)
  - Finds existing task with same OMG # → **Skips task creation**
  - Marks file as "skipped" and moves to Processed folder

**Important**: Tasks are never updated once created. If a task with the same OMG # exists, it's left unchanged.

### 5. File Management

After successful processing:
- **Moves file** to `Processed/` subfolder (auto-created)
- **Failed files** remain in the source directory for retry
- **Cleanup**: Deletes files older than 24 hours from Processed folder

Directory structure:
```
json_files/
├── 5791331.json          # Pending
├── 5791332.json          # Pending
└── Processed/
    ├── 5791330.json      # Processed today (kept)
    └── 5791329.json      # Processed >24h ago (deleted)
```

## Troubleshooting

### Common Issues

#### 1. Authentication Error
```
Error making Wrike request: 401 Unauthorized
```
**Solution**: Verify your API token is correct and has write permissions.

#### 2. Custom Field Not Found
```
Error: Custom field ID not found
```
**Solution**: Ensure all custom fields are created in Wrike and IDs are updated in the script.

#### 3. No JSON Files Found
```
No JSON files found in '/path/to/directory'
```
**Solution**:
- Verify the directory path is correct
- Ensure files have `.json` extension
- Check file permissions

#### 4. Invalid Space ID
```
Error: Space 'MQAAAABpz7l_' not found
```
**Solution**:
- Get your space ID from the API: `GET /spaces`
- Update `STAGING_SPACE_ID` in the script

### Debug Mode

To see detailed API responses, modify the `make_wrike_request` function to print full responses:

```python
def make_wrike_request(method, endpoint, data=None):
    # ... existing code ...
    print(f"Response: {response.json()}")  # Add this line
    return response.json()
```

## API Rate Limits

Wrike API has rate limits:
- **100 requests per minute** per token
- **1000 requests per hour** per token

The script is optimized with caching to minimize API calls, but for very large batches (100+ files), you may need to:
1. Process in smaller batches
2. Add delays between requests

## Data Mapping Reference

### Date Format Conversion
Input: `"2025-05-23 16:00:00+00"`
Output: `"2025-05-23"`

### HTML Cleaning
Input: `"<p>PowerClean FurFinder + FurGuard 2025-related workflows</p>"`
Output: `"PowerClean FurFinder + FurGuard 2025-related workflows"`

### Business Area Parsing
Input: `"BISSELL > PRODUCT MARKETING > Dry Specialty"`
Output: `"Dry Specialty"` (folder name)

## Advanced Usage

### Custom Field Mapping

To add more custom fields, update the script:

1. Add field ID to `CUSTOM_FIELDS` dictionary
2. Update `create_or_update_deliverable_task()` function:

```python
# Add your custom field
custom_fields.append({
    "id": CUSTOM_FIELDS["your_field_name"],
    "value": job_details.get("YourJsonField", "")
})
```

### Filter by Business Area

To process only specific business areas, add a filter in `process_json_file()`:

```python
folder_name = parse_business_area(business_area)

# Only process specific folders
if folder_name not in ["Dry Specialty", "Air"]:
    print(f"  Skipping folder '{folder_name}'")
    return False
```

### Batch Processing Script

Create a wrapper script for automated processing:

```bash
#!/bin/bash
# process_all.sh

IMPORT_DIR="/path/to/json/files"

# Run import (automatically moves to Processed/ and cleans up old files)
python wrike_import.py "$IMPORT_DIR"

# Optional: Send notification
if [ $? -eq 0 ]; then
    echo "Wrike import completed successfully"
fi
```

### Scheduled Processing with Cron

Set up automatic processing every hour:

```bash
# Edit crontab
crontab -e

# Add this line to run every hour
0 * * * * /usr/bin/python3 /path/to/wrike_import.py /path/to/json/files >> /var/log/wrike_import.log 2>&1
```

### Change Cleanup Retention Period

To change the 24-hour retention to a different period, modify the cleanup call in the script:

```python
# Delete files older than 48 hours instead
deleted_count = cleanup_old_files(json_dir, hours=48)

# Delete files older than 7 days
deleted_count = cleanup_old_files(json_dir, hours=168)
```

## Security Notes

⚠️ **Important Security Considerations:**

1. **Never commit API tokens** to version control
2. Store tokens in environment variables:
   ```python
   import os
   WRIKE_TOKEN = os.environ.get('WRIKE_API_TOKEN')
   ```
3. Use a `.env` file for local development (add to `.gitignore`)
4. Rotate tokens regularly
5. Use read-only tokens for testing

## Support

### Getting Help

1. Check the [Wrike API Documentation](https://developers.wrike.com/api/v4/)
2. Review error messages in script output
3. Enable debug mode for detailed logging
4. Check Wrike API status at [status.wrike.com](https://status.wrike.com)

### Reporting Issues

When reporting issues, include:
- Script version
- Python version (`python --version`)
- Full error message
- Sample JSON structure (redacted)
- Wrike space configuration

## License

This script is provided as-is for internal use.

## Which Tool Should I Use?

### Use wrike_import.py if you want to:
- Import a batch of JSON files once
- Run manually when needed
- Test imports locally
- Simple command-line operation

### Use wrike_monitor.py if you want to:
- Continuous 24/7 monitoring
- Automatic processing as files arrive
- Daily email reports
- Production deployment as a service
- Detailed statistics and monitoring

## Changelog

### Version 2.0 (Current)
- **NEW**: Added wrike_monitor.py - real-time monitoring service
- **NEW**: Folder watching with watchdog
- **NEW**: Daily email reports at 7PM
- **NEW**: Failed file handling with error logs
- **NEW**: Periodic scanning for missed files
- **NEW**: Statistics tracking and performance monitoring
- **NEW**: systemd service support
- **FIXED**: Sequential processing to prevent race conditions
- **FIXED**: Proper parent/child folder matching to avoid duplicates
- **IMPROVED**: Logging for skipped deliverables

### Version 1.2
- Added automatic file movement to Processed subfolder
- Implemented auto-cleanup of files older than 24 hours
- Improved file management and organization
- Enhanced output with file movement tracking

### Version 1.1
- Added duplicate detection via OMG numbers
- Separated project and task OMG numbers
- Improved error handling
- Added caching for performance
- Enhanced logging output

### Version 1.0
- Initial release
- Basic folder/project/task creation
- Custom field mapping

---

**Last Updated**: October 2025
**Author**: Dave Porter
**Repository**: https://bitbucket.org/zlalani/bissell-wrike-python