nfoguard/SUMMARY.md

NFOGuard Project Summary

📋 Current Status: v0.6.1 (Latest Improvements)

🎯 Project Goal

Preserve movie and TV import dates in Emby/Jellyfin/Plex by preventing upgraded files from appearing as "recently added". NFOGuard listens to Radarr/Sonarr webhooks and manages .nfo files and file timestamps to maintain chronological consistency.

🚀 Major Achievements

Database-Only Architecture (v0.5.0+)

• Performance: 10x faster than API-based approach
• Reliability: Direct PostgreSQL database queries eliminate API limitations
• Scalability: Handles 1500+ movies without pagination issues

🎯 Configurable Release Date Priority System (v0.6.0)

• Three-Tier Priority System: Digital → Physical → Theatrical (fully configurable)
• Smart Fallbacks: "The Craft (1996)" gets 1996 theatrical, "Top Gun Maverick (2022)" gets digital
• Per-Movie Intelligence: Automatically adapts to movie era and available release data
• Source Tracking: NFO files annotated with chosen date source (tmdb:theatrical, tmdb:digital, etc.)
• Quality Control: Prevents unrealistic dates through smart comparison logic

🏗 Current Architecture

Core Components:

• nfoguard.py - Main FastAPI webhook server
• clients/radarr_db_client.py - Direct database access client
• clients/external_clients.py - TMDB/OMDb integration
• core/nfo_manager.py - NFO file creation and management
• bulk_update_movies.py - Mass movie processing

Data Flow:

1. Radarr webhook → NFOGuard processes → Database queries for import dates
2. Smart fallback logic for manual imports → TMDB digital release dates
3. NFO file creation with preserved dates → File timestamp updates

🔧 Configuration Management

Environment-Based Setup:

• .env.template - Generic configuration template
• docker-compose.yml - Production deployment configuration
• Supports multiple media paths and database types

Key Settings:

• MOVIE_PRIORITY=import_then_digital - Prioritizes real import history
• PREFER_RELEASE_DATES_OVER_FILE_DATES=true - Smart fallback for manual imports
• RELEASE_DATE_PRIORITY=digital,physical,theatrical - Configurable fallback order
• Database connection parameters for PostgreSQL/SQLite

📊 Performance Metrics

Before (API-based):

• Took 30+ seconds for complex movies
• Failed on movies with extensive history
• API pagination caused timeouts

After (Database-only):

• Sub-second response times
• Handles any history size
• Reliable July 2025 date detection maintained

🧪 Testing Infrastructure

Comprehensive Test Suite:

• test_bulk_update.py - Database connection validation
• test_movie_scan.py - Directory scanning logic testing
• test_end_to_end.py - Complete workflow validation

Debug Endpoints:

• /debug/movie/{imdb_id} - Import date analysis
• /debug/movie/{imdb_id}/priority - Date selection logic
• /debug/movie/{imdb_id}/history - Complete import history

🎉 Success Metrics

Production Results:

• ✅ Correct July 2025 dates preserved for legitimate imports
• ✅ Manual imports now use intelligent release date selection:
  • "The Craft (1996)" → 1996 theatrical date (not 2025 file date)
  • "Top Gun Maverick (2022)" → digital release date per user preference
• ✅ Zero API timeout issues with database-only approach
• ✅ Complete webhook-based operation (no manual CLI required)
• ✅ NFO source annotations for full transparency

🚧 Current Development Focus

Completed (v0.6.0):

• ✅ Revolutionary Priority System: Configurable digital/physical/theatrical fallbacks
• ✅ Per-Movie Intelligence: Adapts to movie era and available release data
• ✅ Complete Documentation: README, testing guides, troubleshooting
• ✅ Source Transparency: NFO annotations show exactly which source was used

Next Priorities:

• TV series processing optimization
• Additional external API integrations (OMDb, Jellyseerr)
• Advanced configuration options
• Community feedback integration

📈 Project Maturity

Ready for Production:

• ✅ Stable database architecture
• ✅ Comprehensive error handling
• ✅ Docker deployment ready
• ✅ Extensive testing coverage
• ✅ Complete documentation

Community Ready:

• ✅ Environment-based configuration
• ✅ Portable Docker setup
• ✅ Detailed API documentation
• ✅ Troubleshooting guides

---

🆕 v0.6.1 Recent Improvements (September 2025)

🔧 Enhanced Radarr History Detection:

• Smart Upgrade Detection: Detects movies where first event is movieFileRenamed (upgrade scenarios)
• Improved Logic: For movies like The Matrix (1999), prefer digital release dates over recent upgrade dates
• True Import Preservation: Still honors actual import dates when they exist - only enhances edge cases

🎯 Improved IMDb ID Extraction:

• Flexible Regex: Now supports both [imdb-tt123] and [tt123] formats
• NFO Fallback: Scans .nfo files when path extraction fails (handles Radarr auto-generated files)
• Broader Compatibility: Works with various folder naming conventions

🧠 Enhanced Priority Logic:

• Movies with rename→upgrade history now get better chronological dates
• True import dates always take precedence (no regression)
• Smart fallbacks only engage when appropriate

---

🆕 v0.6.2 TV Episode Date Handling Revolution (September 2025)

🎯 Problem Identified:

• TV episodes downloaded today were showing historical air dates (1951) as dateadded
• Episodes appeared as old content instead of "recently added" in Plex/Jellyfin
• Incorrect fallback logic was using air dates instead of actual download times

⚡ Revolutionary Webhook-First Processing:

• Webhook = Truth: When Sonarr webhook fires, episode was just downloaded → use current timestamp
• Database Priority: Existing NFOguard entries preserved (no re-processing of same episode)
• Smart Backfill: Manual scans check Sonarr import history, only fall back to air dates when no history exists

🔧 Technical Implementation:

• New Method: _get_webhook_episode_date() - webhook-specific date handling
• Enhanced Method: _get_single_episode_date() - proper backfill scan priority
• Clear Separation: Webhook processing vs. backfill scanning use different logic paths
• Database-First: Always check NFOguard database before external APIs

📋 Correct Processing Flow:

1. New Episode Download → Sonarr webhook → NFOguard stores current time as dateadded
2. Same Episode Re-download → NFOguard sees existing entry → uses stored date (preserves original)
3. Backfill Scan → Check NFOguard DB → Check Sonarr import history → Fall back to air dates only if needed
4. First Time Setup → No bulk import - database populated only through webhooks and manual scans

🏆 Results:

• Episodes downloaded today now correctly show today's date as dateadded
• Historical air dates preserved in aired field for accuracy
• "Recently Added" functionality restored in media servers
• No regression for existing properly-dated episodes

🐛 Bug Fixes:

• Fixed f-string syntax error in core/nfo_manager.py:324
• Resolved backslash escaping issue preventing startup

---

🆕 v0.6.3 NFO Management & Date Accuracy Improvements (September 2025)

💾 Smart NFO File Management:

• Problem: NFOguard was rewriting NFO files for every episode on every scan, even untouched ones
• Solution: Intelligent content comparison - only updates NFOs when content actually changes
• Performance: Dramatically reduces unnecessary file system writes and processing time
• Both Media Types: Applied to both TV episodes and movies for comprehensive efficiency

🗓️ Correct Date Field Mapping:

• Problem: Episode NFOs showing import dates (2025) for historical fields <aired> and <premiered>
• Root Cause: Logic incorrectly used dateadded for all date fields instead of proper separation
• Fix: Clear separation of date meanings:
  • <aired>: Historical air date (e.g., 1951-10-15 for I Love Lucy)
  • <premiered>: Historical premiere date (same as aired)
  • <dateadded>: Actual download/import date (e.g., 2025-09-13T21:39:28+00:00)

🎯 Technical Implementation:

• New Method: _nfo_content_matches() - smart content comparison ignoring timestamps
• Enhanced Logic: Episodes now correctly separate historical vs import dates
• Skip Logic: Files marked "already up-to-date" when no meaningful changes detected
• Logging: Clear debug output showing when files are skipped vs updated

🏆 User Experience Improvements:

• Correct Metadata: Shows historically accurate air dates in media servers
• Recently Added: Still works properly using dateadded field
• Performance: Faster scans with fewer unnecessary file writes
• Accuracy: No more confusion between when show aired vs when you downloaded it

📋 Example Before/After:

<!-- Before (Wrong) -->
<aired>2025-09-13</aired>           <!-- Import date in wrong field -->
<dateadded>2025-09-13T21:39:28+00:00</dateadded>
<premiered>2025-09-13</premiered>   <!-- Import date in wrong field -->

<!-- After (Correct) -->
<aired>1951-10-15</aired>           <!-- Historical air date ✅ -->
<dateadded>2025-09-13T21:39:28+00:00</dateadded>  <!-- When downloaded ✅ -->
<premiered>1951-10-15</premiered>   <!-- Historical premiere ✅ -->

---

🆕 v0.6.4 Timezone-Aware Logging Fix (September 2025)

🕰️ Universal Timezone Consistency:

• Problem: Mixed timestamp formats in logs made troubleshooting difficult
  • Some logs: [2025-09-14T09:37:00.338045] (local time without timezone info)
  • Other logs: [2025-09-14T13:37:00+00:00] (UTC with timezone info)
  • Docker TZ=America/New_York environment variable not respected by all logging systems

⚡ Comprehensive Logging Fix:

• Unified Timezone Handling: All logging systems now respect the TZ environment variable
• Custom TimezoneAwareFormatter: Python's standard logging now uses container timezone
• Enhanced _log Function: Custom logging function updated with robust timezone support
• Centralized Logging: Replaced placeholder logging functions with consistent implementation

🔧 Technical Implementation:

• New Class: TimezoneAwareFormatter for Python's standard logging module
• Enhanced Function: _get_local_timezone() with fallbacks for different Python versions
• Cross-Compatibility: Supports both zoneinfo (Python 3.9+) and pytz (older versions)
• Consistent Import: All client modules now use centralized core.logging._log

🏆 User Experience Improvements:

• Consistent Timestamps: All logs AND NFO files now show the same timezone format
• Easier Troubleshooting: Timestamps match user's local environment everywhere
• No More Confusion: No need to mentally convert between UTC and local time
• Production Ready: Robust fallbacks ensure timezone handling works in all environments
• NFO File Accuracy: Movie and TV episode <dateadded> fields now respect container timezone

📋 Before/After Example:

# Before (Inconsistent)
sonarr-nfo-cache  | [2025-09-14T09:37:00.338045] DEBUG: Mapped Radarr path...
sonarr-nfo-cache  | [2025-09-14T13:37:00+00:00] INFO: Batched movie webhook...
<dateadded>2025-09-14T13:49:00+00:00</dateadded>  <!-- UTC in NFO files -->

# After (Consistent)  
sonarr-nfo-cache  | [2025-09-14T09:37:00-04:00] DEBUG: Mapped Radarr path...
sonarr-nfo-cache  | [2025-09-14T09:37:00-04:00] INFO: Batched movie webhook...
<dateadded>2025-09-14T09:49:00-04:00</dateadded>  <!-- Local timezone in NFO files -->

🔧 Additional NFO Timezone Fixes:

• Movie Import Dates: Convert UTC timestamps from Radarr to local timezone in NFO files
• TV Episode Downloads: Webhook-triggered episodes now use local time for <dateadded>
• File Modification Times: Fallback file mtime dates now respect container timezone
• Historical Date Preservation: <aired> and <premiered> dates remain historically accurate
• Centralized Logging: All modules now use consistent timezone-aware logging system

---

📝 Development Workflow Instructions

For Future Updates:

• VERSION File: Always update the VERSION file to match the release version being documented
• Commit Authority: Claude Code is authorized to write commit messages and push changes directly
• SUMMARY Updates: This file serves as the primary project documentation and should be updated with each significant change
• Version Consistency: Ensure VERSION file, SUMMARY.md version, and git tags all align

---

---

🆕 v0.6.6 NFO Timestamp & Duplicate Processing Fixes (September 2025)

🕰️ NFO Management Timestamp Fix:

• Problem: NFO management comments still showed UTC timestamps despite container timezone setting
• Fix: All NFO file comments now respect local timezone
• Before: <!-- managed by NFOGuard at 2025-09-14T13:29:06+00:00 -->
• After: <!-- managed by NFOGuard at 2025-09-14T09:29:06-04:00 -->

🔍 Enhanced Episode Processing Debug:

• Enhanced Logging: Added comprehensive debug logging for episode database lookups
• Duplicate Detection: Improved tracking of why episodes might be processed multiple times
• Database Verification: Added verification logging after database writes
• IMDb ID Tracking: Enhanced logging shows IMDb ID in processing messages

🛠️ Technical Improvements:

• NFO Manager: Updated both movie and TV episode NFO timestamp generation
• Webhook Processing: Added detailed debug logging for episode lookup failures
• Database Integrity: Added immediate verification of database writes
• Troubleshooting Ready: Enhanced logging will help identify duplicate processing causes

---

---

🆕 v0.6.7 Webhook Episode Import History Fix (September 2025)

🎯 Critical Webhook Processing Fix:

• Problem: Webhook episodes used current time instead of actual Sonarr import history
• Issue: <dateadded>2025-09-14T12:40:07+00:00</dateadded> showed webhook time (UTC) instead of real import time (local)
• Root Cause: _get_webhook_episode_date() wasn't querying Sonarr import history like manual scans

⚡ Enhanced Webhook Episode Processing:

• Sonarr History Lookup: Webhooks now query Sonarr import history for real import dates
• Local Timezone Conversion: Import dates converted to container timezone (Eastern Time)
• Proper Source Attribution: Episodes now show sonarr:history.import instead of webhook:new_download when history exists
• Fallback Logic: Only uses current time when no Sonarr history found (true new downloads)

📋 Expected Results:

<!-- Before (Wrong) -->
<dateadded>2025-09-14T12:40:07+00:00</dateadded>  <!-- Current webhook time in UTC -->
<!-- source: TV Episode; webhook:new_download -->

<!-- After (Correct) -->  
<dateadded>2025-09-14T08:40:07-04:00</dateadded>   <!-- Real import time in Eastern -->
<!-- source: TV Episode; sonarr:history.import -->

🛠️ Technical Implementation:

• Enhanced Logic: Webhooks now follow same import history priority as manual scans
• Timezone Consistency: All import dates converted to local timezone before NFO generation
• Better Logging: Shows whether import history was found or current time used as fallback
• Upgrade Handling: Episodes downloaded multiple times now preserve original import dates

---

---

🆕 v0.7.0 MAJOR: Webhook-First Architecture & Database Priority (September 2025)

🎯 Revolutionary Workflow Changes: This is a major architectural change that fundamentally alters how NFOGuard handles timestamps.

📡 Webhooks = Source of Truth:

• First webhook seen → Use current timestamp → Store as permanent source of truth
• Subsequent webhooks (upgrades) → Check database → Use original first-seen timestamp
• No more API calls during webhooks → Webhook timing is the ultimate authority

🔍 Manual Scans = Smart Fallback Logic:

• Priority 1: Our database (webhook timestamps) → Database always wins
• Priority 2: Sonarr/Radarr import history (first import only)
• Priority 3: Air date as dateadded (final fallback)

⚡ Implementation Details:

• TV Episodes: Enhanced _get_webhook_episode_date() → uses current timestamp as source of truth
• Movies: New webhook mode in process_movie() → separate logic for webhooks vs manual scans
• Database Priority: Manual scans now always check database first before API calls
• Local Timezone: All timestamps converted to container timezone (Eastern Time)

🛠️ Technical Changes:

• Webhook Sources: Episodes/movies show webhook:first_seen instead of complex API sources
• Database Verification: Added immediate verification logging after database writes
• Enhanced Debug: Comprehensive logging shows database lookups and timestamp decisions
• Clean Separation: Webhook processing vs manual scan processing use different code paths

📋 Expected Workflow:

# First Download (8:30am EST)
Webhook fires → Use 8:30am timestamp → Store in database → NFO shows 8:30am

# Upgrade Download (2:00pm EST)  
Webhook fires → Check database → Find 8:30am entry → NFO still shows 8:30am

# Manual Scan
Curl scan → Check database → Find 8:30am webhook entry → NFO shows 8:30am

🏆 Benefits:

• Consistent Timestamps: First download time is preserved forever
• Simplified Logic: Webhooks don't query APIs, just use current time
• Performance: Faster webhook processing with database-first approach
• Predictable: Clear hierarchy - database beats everything else

---

---

🆕 v0.7.1 NFO Organization Improvement (September 2025)

📝 Better NFO File Organization:

• Problem: NFOGuard elements mixed throughout existing Radarr metadata made files hard to read
• Solution: Move all NFOGuard elements (<dateadded>, <lockdata>, comments) to bottom of NFO files
• Benefit: Easier to read NFO files with clean separation between media metadata and NFOGuard management

🔧 Technical Implementation:

• Smart Element Management: Remove existing NFOGuard elements and re-add at bottom
• Preserved Functionality: All existing behavior maintained, just better organization
• Both Media Types: Applied to both movie and TV episode NFO files

📋 Before/After NFO Structure:

<!-- Before (Mixed) -->
<plot>Movie plot...</plot>
<dateadded>2025-09-14T10:02:00-04:00</dateadded>  <!-- Mixed in middle -->
<title>Movie Title</title>
<lockdata>true</lockdata>                         <!-- Mixed in middle -->
<director>Director Name</director>
<!-- source: Movie; webhook:first_seen -->        <!-- At end -->

<!-- After (Organized) -->
<plot>Movie plot...</plot>
<title>Movie Title</title>
<director>Director Name</director>
<!-- All media metadata first, then NFOGuard elements at bottom -->
<dateadded>2025-09-14T10:02:00-04:00</dateadded>
<lockdata>true</lockdata>
<!-- source: Movie; webhook:first_seen -->
<!-- managed by NFOGuard at 2025-09-14T11:30:00-04:00 -->

---

---

🆕 v0.7.2 Complete API Keys Documentation (September 2025)

📚 Enhanced README.md Documentation:

• Missing TVDB_API_KEY: Added TVDB API key to all relevant sections
• Comprehensive API Table: Complete reference for all API keys with purposes and sources
• Configuration Examples: Clear examples for .env.secrets setup
• Warning Resolution: Documentation for resolving "TVDB API key not configured" warnings
• Easy Reference: Centralized API keys section with direct links to get keys

🔧 What This Fixes:

• TVDB Warning: Users can now easily resolve [WARNING] TVDB API key not configured, skipping TVDB ID lookup
• Complete Documentation: All API keys (TMDB, TVDB, Radarr, Sonarr) properly documented
• User Experience: Clear table showing which keys are required vs optional

---

Last Updated: September 14, 2025
Version: v0.7.2
Status: Production Ready - Major Architecture