nfoguard/SUMMARY.md

# NFOGuard Project Summary

## 📋 Current Status: v0.6.1 (Latest Improvements)

### 🎯 Project Goal
Preserve movie and TV import dates in Emby/Jellyfin/Plex by preventing upgraded files from appearing as "recently added". NFOGuard listens to Radarr/Sonarr webhooks and manages `.nfo` files and file timestamps to maintain chronological consistency.

### 🚀 Major Achievements

**Database-Only Architecture (v0.5.0+)**
- **Performance**: 10x faster than API-based approach 
- **Reliability**: Direct PostgreSQL database queries eliminate API limitations
- **Scalability**: Handles 1500+ movies without pagination issues

**🎯 Configurable Release Date Priority System (v0.6.0)**  
- **Three-Tier Priority System**: Digital → Physical → Theatrical (fully configurable)
- **Smart Fallbacks**: "The Craft (1996)" gets 1996 theatrical, "Top Gun Maverick (2022)" gets digital
- **Per-Movie Intelligence**: Automatically adapts to movie era and available release data
- **Source Tracking**: NFO files annotated with chosen date source (tmdb:theatrical, tmdb:digital, etc.)
- **Quality Control**: Prevents unrealistic dates through smart comparison logic

### 🏗 Current Architecture

**Core Components:**
- `nfoguard.py` - Main FastAPI webhook server
- `clients/radarr_db_client.py` - Direct database access client
- `clients/external_clients.py` - TMDB/OMDb integration
- `core/nfo_manager.py` - NFO file creation and management
- `bulk_update_movies.py` - Mass movie processing

**Data Flow:**
1. Radarr webhook → NFOGuard processes → Database queries for import dates
2. Smart fallback logic for manual imports → TMDB digital release dates
3. NFO file creation with preserved dates → File timestamp updates

### 🔧 Configuration Management

**Environment-Based Setup:**
- `.env.template` - Generic configuration template
- `docker-compose.yml` - Production deployment configuration
- Supports multiple media paths and database types

**Key Settings:**
- `MOVIE_PRIORITY=import_then_digital` - Prioritizes real import history
- `PREFER_RELEASE_DATES_OVER_FILE_DATES=true` - Smart fallback for manual imports
- `RELEASE_DATE_PRIORITY=digital,physical,theatrical` - Configurable fallback order
- Database connection parameters for PostgreSQL/SQLite

### 📊 Performance Metrics

**Before (API-based):**
- Took 30+ seconds for complex movies
- Failed on movies with extensive history
- API pagination caused timeouts

**After (Database-only):**
- Sub-second response times
- Handles any history size
- Reliable July 2025 date detection maintained

### 🧪 Testing Infrastructure

**Comprehensive Test Suite:**
- `test_bulk_update.py` - Database connection validation
- `test_movie_scan.py` - Directory scanning logic testing
- `test_end_to_end.py` - Complete workflow validation

**Debug Endpoints:**
- `/debug/movie/{imdb_id}` - Import date analysis
- `/debug/movie/{imdb_id}/priority` - Date selection logic
- `/debug/movie/{imdb_id}/history` - Complete import history

### 🎉 Success Metrics

**Production Results:**
- ✅ Correct July 2025 dates preserved for legitimate imports
- ✅ Manual imports now use **intelligent release date selection**:
  - "The Craft (1996)" → 1996 theatrical date (not 2025 file date)
  - "Top Gun Maverick (2022)" → digital release date per user preference
- ✅ Zero API timeout issues with database-only approach
- ✅ Complete webhook-based operation (no manual CLI required)
- ✅ NFO source annotations for full transparency

### 🚧 Current Development Focus

**Completed (v0.6.0):**
- ✅ **Revolutionary Priority System**: Configurable digital/physical/theatrical fallbacks
- ✅ **Per-Movie Intelligence**: Adapts to movie era and available release data
- ✅ **Complete Documentation**: README, testing guides, troubleshooting
- ✅ **Source Transparency**: NFO annotations show exactly which source was used

**Next Priorities:**
- TV series processing optimization
- Additional external API integrations (OMDb, Jellyseerr)
- Advanced configuration options
- Community feedback integration

### 📈 Project Maturity

**Ready for Production:**
- ✅ Stable database architecture
- ✅ Comprehensive error handling
- ✅ Docker deployment ready
- ✅ Extensive testing coverage
- ✅ Complete documentation

**Community Ready:**
- ✅ Environment-based configuration
- ✅ Portable Docker setup
- ✅ Detailed API documentation
- ✅ Troubleshooting guides

---

### 🆕 v0.6.1 Recent Improvements (September 2025)

**🔧 Enhanced Radarr History Detection:**
- **Smart Upgrade Detection**: Detects movies where first event is `movieFileRenamed` (upgrade scenarios)
- **Improved Logic**: For movies like The Matrix (1999), prefer digital release dates over recent upgrade dates
- **True Import Preservation**: Still honors actual import dates when they exist - only enhances edge cases

**🎯 Improved IMDb ID Extraction:**  
- **Flexible Regex**: Now supports both `[imdb-tt123]` and `[tt123]` formats
- **NFO Fallback**: Scans `.nfo` files when path extraction fails (handles Radarr auto-generated files)
- **Broader Compatibility**: Works with various folder naming conventions

**🧠 Enhanced Priority Logic:**
- Movies with rename→upgrade history now get better chronological dates
- True import dates always take precedence (no regression)
- Smart fallbacks only engage when appropriate

---

### 🆕 v0.6.2 TV Episode Date Handling Revolution (September 2025)

**🎯 Problem Identified:**
- TV episodes downloaded today were showing historical air dates (1951) as `dateadded`
- Episodes appeared as old content instead of "recently added" in Plex/Jellyfin
- Incorrect fallback logic was using air dates instead of actual download times

**⚡ Revolutionary Webhook-First Processing:**
- **Webhook = Truth**: When Sonarr webhook fires, episode was just downloaded → use current timestamp
- **Database Priority**: Existing NFOguard entries preserved (no re-processing of same episode)
- **Smart Backfill**: Manual scans check Sonarr import history, only fall back to air dates when no history exists

**🔧 Technical Implementation:**
- **New Method**: `_get_webhook_episode_date()` - webhook-specific date handling
- **Enhanced Method**: `_get_single_episode_date()` - proper backfill scan priority
- **Clear Separation**: Webhook processing vs. backfill scanning use different logic paths
- **Database-First**: Always check NFOguard database before external APIs

**📋 Correct Processing Flow:**
1. **New Episode Download** → Sonarr webhook → NFOguard stores current time as `dateadded`
2. **Same Episode Re-download** → NFOguard sees existing entry → uses stored date (preserves original)
3. **Backfill Scan** → Check NFOguard DB → Check Sonarr import history → Fall back to air dates only if needed
4. **First Time Setup** → No bulk import - database populated only through webhooks and manual scans

**🏆 Results:**
- Episodes downloaded today now correctly show today's date as `dateadded`
- Historical air dates preserved in `aired` field for accuracy
- "Recently Added" functionality restored in media servers
- No regression for existing properly-dated episodes

**🐛 Bug Fixes:**
- Fixed f-string syntax error in `core/nfo_manager.py:324`
- Resolved backslash escaping issue preventing startup

---

### 🆕 v0.6.3 NFO Management & Date Accuracy Improvements (September 2025)

**💾 Smart NFO File Management:**
- **Problem**: NFOguard was rewriting NFO files for every episode on every scan, even untouched ones
- **Solution**: Intelligent content comparison - only updates NFOs when content actually changes
- **Performance**: Dramatically reduces unnecessary file system writes and processing time
- **Both Media Types**: Applied to both TV episodes and movies for comprehensive efficiency

**🗓️ Correct Date Field Mapping:**
- **Problem**: Episode NFOs showing import dates (2025) for historical fields `<aired>` and `<premiered>`
- **Root Cause**: Logic incorrectly used `dateadded` for all date fields instead of proper separation
- **Fix**: Clear separation of date meanings:
  - `<aired>`: Historical air date (e.g., 1951-10-15 for I Love Lucy)
  - `<premiered>`: Historical premiere date (same as aired)
  - `<dateadded>`: Actual download/import date (e.g., 2025-09-13T21:39:28+00:00)

**🎯 Technical Implementation:**
- **New Method**: `_nfo_content_matches()` - smart content comparison ignoring timestamps
- **Enhanced Logic**: Episodes now correctly separate historical vs import dates
- **Skip Logic**: Files marked "already up-to-date" when no meaningful changes detected
- **Logging**: Clear debug output showing when files are skipped vs updated

**🏆 User Experience Improvements:**
- **Correct Metadata**: Shows historically accurate air dates in media servers
- **Recently Added**: Still works properly using `dateadded` field
- **Performance**: Faster scans with fewer unnecessary file writes
- **Accuracy**: No more confusion between when show aired vs when you downloaded it

**📋 Example Before/After:**
```xml
<!-- Before (Wrong) -->
<aired>2025-09-13</aired>           <!-- Import date in wrong field -->
<dateadded>2025-09-13T21:39:28+00:00</dateadded>
<premiered>2025-09-13</premiered>   <!-- Import date in wrong field -->

<!-- After (Correct) -->
<aired>1951-10-15</aired>           <!-- Historical air date ✅ -->
<dateadded>2025-09-13T21:39:28+00:00</dateadded>  <!-- When downloaded ✅ -->
<premiered>1951-10-15</premiered>   <!-- Historical premiere ✅ -->
```

---

### 🆕 v0.6.4 Timezone-Aware Logging Fix (September 2025)

**🕰️ Universal Timezone Consistency:**
- **Problem**: Mixed timestamp formats in logs made troubleshooting difficult
  - Some logs: `[2025-09-14T09:37:00.338045]` (local time without timezone info)
  - Other logs: `[2025-09-14T13:37:00+00:00]` (UTC with timezone info)
  - Docker `TZ=America/New_York` environment variable not respected by all logging systems

**⚡ Comprehensive Logging Fix:**
- **Unified Timezone Handling**: All logging systems now respect the `TZ` environment variable
- **Custom TimezoneAwareFormatter**: Python's standard logging now uses container timezone
- **Enhanced _log Function**: Custom logging function updated with robust timezone support
- **Centralized Logging**: Replaced placeholder logging functions with consistent implementation

**🔧 Technical Implementation:**
- **New Class**: `TimezoneAwareFormatter` for Python's standard logging module
- **Enhanced Function**: `_get_local_timezone()` with fallbacks for different Python versions
- **Cross-Compatibility**: Supports both `zoneinfo` (Python 3.9+) and `pytz` (older versions)
- **Consistent Import**: All client modules now use centralized `core.logging._log`

**🏆 User Experience Improvements:**
- **Consistent Timestamps**: All logs AND NFO files now show the same timezone format
- **Easier Troubleshooting**: Timestamps match user's local environment everywhere
- **No More Confusion**: No need to mentally convert between UTC and local time
- **Production Ready**: Robust fallbacks ensure timezone handling works in all environments
- **NFO File Accuracy**: Movie and TV episode `<dateadded>` fields now respect container timezone

**📋 Before/After Example:**
```
# Before (Inconsistent)
sonarr-nfo-cache  | [2025-09-14T09:37:00.338045] DEBUG: Mapped Radarr path...
sonarr-nfo-cache  | [2025-09-14T13:37:00+00:00] INFO: Batched movie webhook...
<dateadded>2025-09-14T13:49:00+00:00</dateadded>  <!-- UTC in NFO files -->

# After (Consistent)  
sonarr-nfo-cache  | [2025-09-14T09:37:00-04:00] DEBUG: Mapped Radarr path...
sonarr-nfo-cache  | [2025-09-14T09:37:00-04:00] INFO: Batched movie webhook...
<dateadded>2025-09-14T09:49:00-04:00</dateadded>  <!-- Local timezone in NFO files -->
```

**🔧 Additional NFO Timezone Fixes:**
- **Movie Import Dates**: Convert UTC timestamps from Radarr to local timezone in NFO files
- **TV Episode Downloads**: Webhook-triggered episodes now use local time for `<dateadded>`
- **File Modification Times**: Fallback file mtime dates now respect container timezone
- **Historical Date Preservation**: `<aired>` and `<premiered>` dates remain historically accurate
- **Centralized Logging**: All modules now use consistent timezone-aware logging system

---

## 📝 Development Workflow Instructions

**For Future Updates:**
- **VERSION File**: Always update the `VERSION` file to match the release version being documented
- **Commit Authority**: Claude Code is authorized to write commit messages and push changes directly
- **SUMMARY Updates**: This file serves as the primary project documentation and should be updated with each significant change
- **Version Consistency**: Ensure VERSION file, SUMMARY.md version, and git tags all align

---

---

### 🆕 v0.6.6 NFO Timestamp & Duplicate Processing Fixes (September 2025)

**🕰️ NFO Management Timestamp Fix:**
- **Problem**: NFO management comments still showed UTC timestamps despite container timezone setting
- **Fix**: All NFO file comments now respect local timezone
- **Before**: `<!-- managed by NFOGuard at 2025-09-14T13:29:06+00:00 -->`
- **After**: `<!-- managed by NFOGuard at 2025-09-14T09:29:06-04:00 -->`

**🔍 Enhanced Episode Processing Debug:**
- **Enhanced Logging**: Added comprehensive debug logging for episode database lookups
- **Duplicate Detection**: Improved tracking of why episodes might be processed multiple times
- **Database Verification**: Added verification logging after database writes
- **IMDb ID Tracking**: Enhanced logging shows IMDb ID in processing messages

**🛠️ Technical Improvements:**
- **NFO Manager**: Updated both movie and TV episode NFO timestamp generation
- **Webhook Processing**: Added detailed debug logging for episode lookup failures
- **Database Integrity**: Added immediate verification of database writes
- **Troubleshooting Ready**: Enhanced logging will help identify duplicate processing causes

---

---

### 🆕 v0.6.7 Webhook Episode Import History Fix (September 2025)

**🎯 Critical Webhook Processing Fix:**
- **Problem**: Webhook episodes used current time instead of actual Sonarr import history
- **Issue**: `<dateadded>2025-09-14T12:40:07+00:00</dateadded>` showed webhook time (UTC) instead of real import time (local)
- **Root Cause**: `_get_webhook_episode_date()` wasn't querying Sonarr import history like manual scans

**⚡ Enhanced Webhook Episode Processing:**
- **Sonarr History Lookup**: Webhooks now query Sonarr import history for real import dates
- **Local Timezone Conversion**: Import dates converted to container timezone (Eastern Time)
- **Proper Source Attribution**: Episodes now show `sonarr:history.import` instead of `webhook:new_download` when history exists
- **Fallback Logic**: Only uses current time when no Sonarr history found (true new downloads)

**📋 Expected Results:**
```xml
<!-- Before (Wrong) -->
<dateadded>2025-09-14T12:40:07+00:00</dateadded>  <!-- Current webhook time in UTC -->
<!-- source: TV Episode; webhook:new_download -->

<!-- After (Correct) -->  
<dateadded>2025-09-14T08:40:07-04:00</dateadded>   <!-- Real import time in Eastern -->
<!-- source: TV Episode; sonarr:history.import -->
```

**🛠️ Technical Implementation:**
- **Enhanced Logic**: Webhooks now follow same import history priority as manual scans
- **Timezone Consistency**: All import dates converted to local timezone before NFO generation  
- **Better Logging**: Shows whether import history was found or current time used as fallback
- **Upgrade Handling**: Episodes downloaded multiple times now preserve original import dates

---

---

### 🆕 v0.7.0 MAJOR: Webhook-First Architecture & Database Priority (September 2025)

**🎯 Revolutionary Workflow Changes:**
This is a **major architectural change** that fundamentally alters how NFOGuard handles timestamps.

**📡 Webhooks = Source of Truth:**
- **First webhook seen** → Use current timestamp → Store as **permanent source of truth**
- **Subsequent webhooks** (upgrades) → Check database → **Use original first-seen timestamp**  
- **No more API calls during webhooks** → Webhook timing is the ultimate authority

**🔍 Manual Scans = Smart Fallback Logic:**
- **Priority 1**: Our database (webhook timestamps) → **Database always wins**
- **Priority 2**: Sonarr/Radarr import history (first import only)
- **Priority 3**: Air date as dateadded (final fallback)

**⚡ Implementation Details:**
- **TV Episodes**: Enhanced `_get_webhook_episode_date()` → uses current timestamp as source of truth
- **Movies**: New webhook mode in `process_movie()` → separate logic for webhooks vs manual scans
- **Database Priority**: Manual scans now **always** check database first before API calls
- **Local Timezone**: All timestamps converted to container timezone (Eastern Time)

**🛠️ Technical Changes:**
- **Webhook Sources**: Episodes/movies show `webhook:first_seen` instead of complex API sources
- **Database Verification**: Added immediate verification logging after database writes
- **Enhanced Debug**: Comprehensive logging shows database lookups and timestamp decisions
- **Clean Separation**: Webhook processing vs manual scan processing use different code paths

**📋 Expected Workflow:**
```
# First Download (8:30am EST)
Webhook fires → Use 8:30am timestamp → Store in database → NFO shows 8:30am

# Upgrade Download (2:00pm EST)  
Webhook fires → Check database → Find 8:30am entry → NFO still shows 8:30am

# Manual Scan
Curl scan → Check database → Find 8:30am webhook entry → NFO shows 8:30am
```

**🏆 Benefits:**
- **Consistent Timestamps**: First download time is preserved forever
- **Simplified Logic**: Webhooks don't query APIs, just use current time
- **Performance**: Faster webhook processing with database-first approach
- **Predictable**: Clear hierarchy - database beats everything else

---

---

### 🆕 v0.7.1 NFO Organization Improvement (September 2025)

**📝 Better NFO File Organization:**
- **Problem**: NFOGuard elements mixed throughout existing Radarr metadata made files hard to read
- **Solution**: Move all NFOGuard elements (`<dateadded>`, `<lockdata>`, comments) to bottom of NFO files
- **Benefit**: Easier to read NFO files with clean separation between media metadata and NFOGuard management

**🔧 Technical Implementation:**
- **Smart Element Management**: Remove existing NFOGuard elements and re-add at bottom
- **Preserved Functionality**: All existing behavior maintained, just better organization
- **Both Media Types**: Applied to both movie and TV episode NFO files

**📋 Before/After NFO Structure:**
```xml
<!-- Before (Mixed) -->
<plot>Movie plot...</plot>
<dateadded>2025-09-14T10:02:00-04:00</dateadded>  <!-- Mixed in middle -->
<title>Movie Title</title>
<lockdata>true</lockdata>                         <!-- Mixed in middle -->
<director>Director Name</director>
<!-- source: Movie; webhook:first_seen -->        <!-- At end -->

<!-- After (Organized) -->
<plot>Movie plot...</plot>
<title>Movie Title</title>
<director>Director Name</director>
<!-- All media metadata first, then NFOGuard elements at bottom -->
<dateadded>2025-09-14T10:02:00-04:00</dateadded>
<lockdata>true</lockdata>
<!-- source: Movie; webhook:first_seen -->
<!-- managed by NFOGuard at 2025-09-14T11:30:00-04:00 -->
```

---

---

### 🆕 v0.7.2 Complete API Keys Documentation (September 2025)

**📚 Enhanced README.md Documentation:**
- **Missing TVDB_API_KEY**: Added TVDB API key to all relevant sections
- **Comprehensive API Table**: Complete reference for all API keys with purposes and sources
- **Configuration Examples**: Clear examples for .env.secrets setup  
- **Warning Resolution**: Documentation for resolving "TVDB API key not configured" warnings
- **Easy Reference**: Centralized API keys section with direct links to get keys

**🔧 What This Fixes:**
- **TVDB Warning**: Users can now easily resolve `[WARNING] TVDB API key not configured, skipping TVDB ID lookup`
- **Complete Documentation**: All API keys (TMDB, TVDB, Radarr, Sonarr) properly documented
- **User Experience**: Clear table showing which keys are required vs optional

---

**Last Updated:** September 14, 2025  
**Version:** v0.7.2  
**Status:** Production Ready - Major Architecture