diff --git a/Project.md b/Project.md new file mode 100644 index 0000000..ee8d563 --- /dev/null +++ b/Project.md @@ -0,0 +1,331 @@ +# NFOGuard Project Overview + +## Project Purpose +NFOGuard is an automated NFO file management system that integrates with Radarr and Sonarr to maintain accurate metadata and release dates in media library NFO files. It serves as a webhook-driven middleware that preserves existing metadata while adding clean, standardized date information. + +## Core Architecture + +### Technology Stack +- **Language**: Python 3 +- **Database**: PostgreSQL (direct database access to Radarr/Sonarr) +- **Deployment**: Docker containers with Docker Compose +- **API Framework**: Built-in HTTP webhook server +- **File Processing**: XML parsing with xml.etree.ElementTree +- **Logging**: Custom logging system with configurable debug levels + +### Key Components +1. **nfoguard.py** - Main application with webhook server and processing logic + - Lines 975-990: Webhook date decision logic (fixed in v1.5.4) + - Lines 1429, 1479: Event type filtering for Sonarr/Radarr webhooks + - Lines 366, 555, 607, 813: Episode NFO creation calls + - Lines 586-592: Episode parsing from filename logic +2. **core/nfo_manager.py** - NFO file creation, parsing, and management + - Lines 336-375: `_find_existing_episode_nfo()` - Smart NFO detection (v1.6.0) + - Lines 377-491: `create_episode_nfo()` - Episode NFO creation with renaming + - Lines 341: Episode filename pattern `S{season_num:02d}E{episode_num:02d}.nfo` + - Lines 480-487: NFO renaming logic (removes old files after rename) + - Lines 85-149: `parse_imdb_from_path_or_nfo()` - Dual identification system +3. **core/database.py** - Database operations and date management + - Lines 173-186: `upsert_movie_dates()` - CRITICAL FIX: INSERT OR REPLACE (v1.5.5) + - Database connection handling for Radarr/Sonarr PostgreSQL +4. **core/logging.py** - Centralized logging system with `_log()` function +5. **core/config.py** - Configuration management + +## Primary Features + +### Webhook Integration +- **Radarr Webhooks**: `/webhook/radarr` - Handles movie import, upgrade, rename events +- **Sonarr Webhooks**: `/webhook/sonarr` - Handles TV episode import, upgrade, rename events +- **Event Filtering**: Processes Download, Upgrade, Rename, Import events only +- **Batch Processing**: Configurable delays to handle multiple rapid events + +### Media Identification (Dual Method System) +1. **Primary**: Directory names with IMDb IDs in brackets `[tt1234567]` or `[imdb-tt1234567]` +2. **Fallback**: NFO file parsing for IMDb IDs using XML tags: + - `tt1234567` + - `tt1234567` + - `tt1234567` + +### NFO File Management + +#### Movies +- **File**: `movie.nfo` in movie directory +- **Metadata**: Preserves existing Radarr metadata, adds release dates at bottom +- **Date Priority**: Digital → Physical → Theatrical release dates +- **Locking**: Adds `true` to prevent overwrites + +#### TV Shows +- **Series File**: `tvshow.nfo` in series root directory +- **Episode Files**: Standardized `S##E##.nfo` format (e.g., `S01E01.nfo`) +- **Smart Renaming**: Detects existing non-standard NFO files, extracts metadata, renames to standard format +- **Metadata**: Preserves existing metadata, adds air dates and import dates + +### Database Integration +- **Direct Access**: Connects to Radarr/Sonarr PostgreSQL databases +- **Date Tracking**: Maintains `dateadded` timestamps for import dates +- **Upsert Operations**: Proper `INSERT OR REPLACE` for data integrity +- **History Queries**: Retrieves earliest import dates from application databases + +## Directory Structure Requirements + +### Movies +``` +/movies/ +└── Movie Title (2024) [tt1234567]/ + ├── movie.mkv + └── movie.nfo (created/updated by NFOGuard) +``` + +### TV Shows +``` +/tv/ +└── Series Title (2024) [tt1234567]/ + ├── Season 01/ + │ ├── Series S01E01.mkv + │ ├── S01E01.nfo (created/updated by NFOGuard) + │ └── S01E02.nfo + ├── Season 02/ + └── tvshow.nfo (created/updated by NFOGuard) +``` + +## Configuration System + +### Environment Files +- **`.env`**: Main configuration (paths, behavior, non-sensitive options) +- **`.env.secrets`**: Sensitive data (API keys, database credentials) + +### Key Settings +- **Media Paths**: Container and application path mappings +- **Release Date Priority**: Configurable date preference order +- **Debug Modes**: `DEBUG`, `PATH_DEBUG`, `SUPPRESS_TVDB_WARNINGS` +- **Database Credentials**: PostgreSQL connection details +- **API Keys**: TMDB, TVDB integration (TVDB optional) + +## API Endpoints + +| Endpoint | Method | Purpose | +|----------|--------|---------| +| `/health` | GET | Health check and status | +| `/webhook/radarr` | POST | Radarr webhook handler | +| `/webhook/sonarr` | POST | Sonarr webhook handler | +| `/manual/scan` | POST | Manual media scanning | +| `/bulk/update` | POST | Bulk movie updates from Radarr DB | + +## Critical Bug Fixes & Technical Details + +### Database Upsert Bug (v1.5.5) - CRITICAL +- **Location**: `core/database.py:173-186` +- **Problem**: `upsert_movie_dates()` was UPDATE-only, not proper upsert +- **Symptoms**: Manual scans weren't saving `dateadded`, webhooks fell back to current timestamp +- **Fix**: Changed to `INSERT OR REPLACE` with `COALESCE` for path preservation +- **Impact**: Fixed webhook date accuracy and database integrity + +### Webhook Date Logic Bug (v1.5.4) +- **Location**: `nfoguard.py:975-990` +- **Problem**: Upgrade webhooks bypassed full date decision logic +- **Symptoms**: Used current timestamp instead of existing Radarr import dates +- **Fix**: Webhook mode now uses same date priority as manual scans +- **Flow**: Database → Radarr/Sonarr API → Release dates → Current timestamp + +### Radarr Event Filtering (v1.5.3) +- **Location**: `nfoguard.py:1479` +- **Problem**: Missing event type filtering (Sonarr had it, Radarr didn't) +- **Fix**: Added `if event_type not in ["Download", "Upgrade", "Rename"]:` +- **Result**: Both systems now follow identical webhook processing + +### NFO-Based Identification System (v1.6.0) +- **Location**: `core/nfo_manager.py:85-149` +- **Method**: `parse_imdb_from_path_or_nfo()` tries directory first, then NFO files +- **Supports**: ``, ``, `` XML tags +- **Files**: movie.nfo and tvshow.nfo parsing + +### Smart Episode NFO Renaming (v1.6.0) +- **Location**: `core/nfo_manager.py:336-491` +- **Detection**: `_find_existing_episode_nfo()` scans season directories +- **Process**: Extract metadata → Write to standard format → Remove old file +- **Pattern**: Always creates `S##E##.nfo` format +- **Safety**: Only removes old file after successful write + +## Development History & Major Fixes + +### v1.6.0 (September 16, 2025) - Current +- **NFO-Based Identification**: Added dual identification system (directory + NFO parsing) +- **Smart NFO Renaming**: Detects and renames non-standard episode NFO files to `S##E##.nfo` format +- **Metadata Preservation**: Maintains existing metadata during NFO operations +- **Enhanced Documentation**: Comprehensive README and project documentation + +### v1.5.5 (September 15, 2025) +- **CRITICAL DATABASE FIX**: Fixed `upsert_movie_dates()` from UPDATE-only to proper `INSERT OR REPLACE` +- **Data Integrity**: Manual scans now properly save `dateadded` to database +- **Webhook Fix**: Webhooks now find existing database entries correctly + +### v1.5.4 (September 15, 2025) +- **Webhook Date Logic Fix**: Fixed webhook processing to use proper date decision logic +- **Smart Fallback**: Database → Radarr import dates → release dates → timestamp priority + +### v1.5.3 (September 15, 2025) +- **Radarr Rename Fix**: Added missing event type filtering to Radarr webhooks +- **Consistency**: Radarr now matches Sonarr's event handling + +### v1.5.2 +- **TVDB Warning Suppression**: Added `SUPPRESS_TVDB_WARNINGS` for cleaner logs +- **Production Ready**: Reduced log noise for production environments + +## Date Decision Logic + +### Movies +1. **Database Check**: Existing NFOGuard database entry +2. **Radarr History**: Earliest import date from Radarr database +3. **Release Dates**: Digital → Physical → Theatrical (configurable priority) +4. **File Dates**: File system timestamps (fallback) +5. **Current Timestamp**: Absolute last resort + +### TV Shows +1. **Database Check**: Existing NFOGuard database entry +2. **Sonarr History**: Earliest import date from Sonarr database +3. **Air Dates**: Episode air date from metadata +4. **File Dates**: File system timestamps (fallback) +5. **Current Timestamp**: Absolute last resort + +## Workflow Patterns + +### Webhook Processing +1. **Event Validation**: Filter supported event types +2. **Media Detection**: Parse file paths and identify media +3. **Database Lookup**: Check for existing entries +4. **Date Resolution**: Apply decision logic hierarchy +5. **NFO Processing**: Create/update NFO files with metadata +6. **Database Update**: Store processed information + +### Manual Scanning +1. **Directory Traversal**: Scan configured media paths +2. **Media Identification**: Directory names and NFO files +3. **Batch Processing**: Process multiple items efficiently +4. **Database Persistence**: Store all discovered media +5. **NFO Creation**: Generate/update all NFO files + +## Integration Points + +### External Services +- **Radarr**: Movie management, webhook triggers, database queries +- **Sonarr**: TV show management, webhook triggers, database queries +- **TMDB**: Movie metadata and release dates (primary) +- **TVDB**: TV show metadata (optional, failures suppressed) +- **Emby/Jellyfin**: NFO consumption for media servers + +### File System +- **Read Access**: Media directories for scanning and identification +- **Write Access**: NFO file creation and updates +- **Path Mapping**: Container paths vs application paths + +## Development Patterns + +### Code Organization +- **Modular Design**: Separate concerns (database, NFO, logging, config) +- **Error Handling**: Comprehensive exception handling with logging +- **Configuration Driven**: Environment-based configuration management +- **Docker Native**: Designed for containerized deployment + +### Testing Strategy +- **Manual Testing**: API endpoints and webhook simulation +- **Real-world Integration**: Actual Radarr/Sonarr webhook testing +- **Database Validation**: Direct database query verification +- **File System Testing**: NFO file creation and parsing validation + +## Common Issues & Solutions + +### Path Mapping +- **Problem**: Container paths don't match application paths +- **Solution**: Proper volume mapping and path configuration in `.env` +- **Debug**: Enable `PATH_DEBUG=true` to see path resolution + +### Database Connectivity +- **Problem**: Cannot connect to Radarr/Sonarr databases +- **Solution**: Verify credentials in `.env.secrets` and network connectivity +- **Check**: Database host, port, username, password in secrets file + +### Permission Issues +- **Problem**: Cannot write NFO files +- **Solution**: Ensure proper file permissions and volume mount configurations +- **Fix**: `chmod 755` on media directories, `rw` mount permissions + +### TVDB Failures +- **Problem**: TVDB API errors causing log noise +- **Solution**: Set `SUPPRESS_TVDB_WARNINGS=true` (system works without TVDB) +- **Note**: TVDB is optional - IMDb and Sonarr data are primary sources + +### Webhook Issues +- **Problem**: Webhooks not triggering NFO updates +- **Debug Steps**: + 1. Check webhook URLs: `http://nfoguard:8080/webhook/radarr|sonarr` + 2. Verify event types: Download, Upgrade, Rename only + 3. Enable `DEBUG=true` for detailed webhook logging + 4. Check database entries with manual scan first + +### Date Problems +- **Problem**: Wrong dates in NFO files (current timestamp instead of import date) +- **Root Cause**: Usually database upsert issues or missing database entries +- **Solution**: Run manual scan first to populate database, then test webhooks +- **Priority**: Database → API → Release dates → File dates → Current timestamp + +## Future Considerations + +### Potential Enhancements +- **Movie NFO Renaming**: Similar to TV show episode renaming +- **Multiple NFO Formats**: Support for different media server preferences +- **Metadata Enhancement**: Additional metadata sources and fields +- **Performance Optimization**: Caching and batch processing improvements + +### Monitoring & Maintenance +- **Health Checks**: Regular endpoint monitoring +- **Log Analysis**: Debug mode for troubleshooting +- **Database Maintenance**: Regular cleanup of old entries +- **Version Management**: Semantic versioning and change tracking + +## Development Environment + +### Prerequisites +- Docker and Docker Compose +- Python 3.x development environment +- Access to Radarr/Sonarr instances with PostgreSQL +- Git workflow familiarity + +### Development Workflow +1. **Local Testing**: Use development Docker image +2. **Code Changes**: Implement features with proper logging +3. **Version Updates**: Update VERSION file and SUMMARY.md +4. **Documentation**: Update README.md and Project.md +5. **Deployment**: Push to dev branch, then production + +## Quick Context Recovery + +### When Starting a New Session +1. **Read Project.md** - This file for full context +2. **Check SUMMARY.md** - Latest changes and current version +3. **Review VERSION** - Current version number +4. **Key Files to Understand**: + - `nfoguard.py` - Main logic, webhook handlers + - `core/nfo_manager.py` - NFO creation, parsing, renaming + - `core/database.py` - Database operations (watch the upsert bug fix!) + +### Critical Code Locations to Remember +- **Database Upsert**: `core/database.py:173-186` (fixed UPDATE → INSERT OR REPLACE) +- **Webhook Date Logic**: `nfoguard.py:975-990` (fixed fallback behavior) +- **Event Filtering**: `nfoguard.py:1429,1479` (Download/Upgrade/Rename only) +- **NFO Renaming**: `core/nfo_manager.py:336-491` (smart episode renaming) +- **Dual Identification**: `core/nfo_manager.py:85-149` (directory + NFO parsing) + +### User Context +- **User has**: Radarr, Sonarr, Unmanic integration +- **Environment**: Docker deployment, PostgreSQL databases +- **Focus Areas**: Webhook accuracy, NFO standardization, date integrity +- **Testing Method**: Real webhooks from Radarr/Sonarr, manual scans for validation + +### Recent Work Pattern +- User reports issues with specific examples (log snippets, webhook behavior) +- We debug by analyzing code at specific line numbers +- Implement fixes with proper error handling and logging +- Update documentation (SUMMARY.md, README.md, Project.md) +- Version bump and prepare for deployment + +This project represents a mature, production-ready system for automated NFO management with comprehensive webhook integration and intelligent metadata handling. \ No newline at end of file diff --git a/SUMMARY.md b/SUMMARY.md index ea67e4a..7669629 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,8 +1,21 @@ # NFOGuard Development Summary -## Current Status: v1.5.5 - Database Upsert Fix! 🔧 +## Current Status: v1.6.0 - NFO-Based Identification! 📄 -### Latest Updates (v1.5.5) - September 15, 2025 +### Latest Updates (v1.6.0) - September 16, 2025 +- **📄 NFO IDENTIFICATION**: Added support for identifying movies/TV shows from NFO files +- **🔄 DUAL METHOD**: Directory name IMDb IDs (primary) + NFO file IMDb IDs (fallback) +- **📋 NFO FORMATS**: Supports ``, ``, and `` XML tags +- **🎬 MOVIE SUPPORT**: Reads movie.nfo files for IMDb ID extraction +- **📺 TV SUPPORT**: Reads tvshow.nfo files for series identification +- **✅ SMART FALLBACK**: Only checks NFO files when directory name lacks IMDb ID +- **📝 DOCUMENTATION**: Updated README with NFO identification examples and formats +- **🔄 NFO RENAMING**: Added smart episode NFO file renaming to standardized `S##E##.nfo` format +- **🎯 SONARR INTEGRATION**: Detects existing NFO files created by Sonarr/other tools, extracts metadata, and renames to standard format +- **📁 METADATA PRESERVATION**: Preserves all existing metadata when renaming non-standard NFO files +- **🧹 CLEANUP**: Removes old non-standard NFO files after successful rename to prevent duplicates + +### Previous Updates (v1.5.5) - September 15, 2025 - **🔧 DATABASE BUG FIX**: Fixed `upsert_movie_dates` to be a proper upsert operation - **💾 DATA INTEGRITY**: Manual scans now properly save `dateadded` to database - **🎯 ROOT CAUSE**: Fixed UPDATE-only operation that was leaving `dateadded` fields NULL diff --git a/VERSION b/VERSION index 9075be4..dc1e644 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.5.5 +1.6.0