Add NFO-based identification and smart episode renaming v1.6.0
- Add dual identification system: directory names + NFO file parsing - Support <uniqueid type="imdb">, <imdbid>, <imdb> XML tags - Implement smart episode NFO renaming to S##E##.nfo format - Detect existing Sonarr/other NFO files and preserve metadata - Add comprehensive Project.md for context recovery - Update documentation with new capabilities
This commit is contained in:
+331
@@ -0,0 +1,331 @@
|
||||
# NFOGuard Project Overview
|
||||
|
||||
## Project Purpose
|
||||
NFOGuard is an automated NFO file management system that integrates with Radarr and Sonarr to maintain accurate metadata and release dates in media library NFO files. It serves as a webhook-driven middleware that preserves existing metadata while adding clean, standardized date information.
|
||||
|
||||
## Core Architecture
|
||||
|
||||
### Technology Stack
|
||||
- **Language**: Python 3
|
||||
- **Database**: PostgreSQL (direct database access to Radarr/Sonarr)
|
||||
- **Deployment**: Docker containers with Docker Compose
|
||||
- **API Framework**: Built-in HTTP webhook server
|
||||
- **File Processing**: XML parsing with xml.etree.ElementTree
|
||||
- **Logging**: Custom logging system with configurable debug levels
|
||||
|
||||
### Key Components
|
||||
1. **nfoguard.py** - Main application with webhook server and processing logic
|
||||
- Lines 975-990: Webhook date decision logic (fixed in v1.5.4)
|
||||
- Lines 1429, 1479: Event type filtering for Sonarr/Radarr webhooks
|
||||
- Lines 366, 555, 607, 813: Episode NFO creation calls
|
||||
- Lines 586-592: Episode parsing from filename logic
|
||||
2. **core/nfo_manager.py** - NFO file creation, parsing, and management
|
||||
- Lines 336-375: `_find_existing_episode_nfo()` - Smart NFO detection (v1.6.0)
|
||||
- Lines 377-491: `create_episode_nfo()` - Episode NFO creation with renaming
|
||||
- Lines 341: Episode filename pattern `S{season_num:02d}E{episode_num:02d}.nfo`
|
||||
- Lines 480-487: NFO renaming logic (removes old files after rename)
|
||||
- Lines 85-149: `parse_imdb_from_path_or_nfo()` - Dual identification system
|
||||
3. **core/database.py** - Database operations and date management
|
||||
- Lines 173-186: `upsert_movie_dates()` - CRITICAL FIX: INSERT OR REPLACE (v1.5.5)
|
||||
- Database connection handling for Radarr/Sonarr PostgreSQL
|
||||
4. **core/logging.py** - Centralized logging system with `_log()` function
|
||||
5. **core/config.py** - Configuration management
|
||||
|
||||
## Primary Features
|
||||
|
||||
### Webhook Integration
|
||||
- **Radarr Webhooks**: `/webhook/radarr` - Handles movie import, upgrade, rename events
|
||||
- **Sonarr Webhooks**: `/webhook/sonarr` - Handles TV episode import, upgrade, rename events
|
||||
- **Event Filtering**: Processes Download, Upgrade, Rename, Import events only
|
||||
- **Batch Processing**: Configurable delays to handle multiple rapid events
|
||||
|
||||
### Media Identification (Dual Method System)
|
||||
1. **Primary**: Directory names with IMDb IDs in brackets `[tt1234567]` or `[imdb-tt1234567]`
|
||||
2. **Fallback**: NFO file parsing for IMDb IDs using XML tags:
|
||||
- `<uniqueid type="imdb">tt1234567</uniqueid>`
|
||||
- `<imdbid>tt1234567</imdbid>`
|
||||
- `<imdb>tt1234567</imdb>`
|
||||
|
||||
### NFO File Management
|
||||
|
||||
#### Movies
|
||||
- **File**: `movie.nfo` in movie directory
|
||||
- **Metadata**: Preserves existing Radarr metadata, adds release dates at bottom
|
||||
- **Date Priority**: Digital → Physical → Theatrical release dates
|
||||
- **Locking**: Adds `<lockdata>true</lockdata>` to prevent overwrites
|
||||
|
||||
#### TV Shows
|
||||
- **Series File**: `tvshow.nfo` in series root directory
|
||||
- **Episode Files**: Standardized `S##E##.nfo` format (e.g., `S01E01.nfo`)
|
||||
- **Smart Renaming**: Detects existing non-standard NFO files, extracts metadata, renames to standard format
|
||||
- **Metadata**: Preserves existing metadata, adds air dates and import dates
|
||||
|
||||
### Database Integration
|
||||
- **Direct Access**: Connects to Radarr/Sonarr PostgreSQL databases
|
||||
- **Date Tracking**: Maintains `dateadded` timestamps for import dates
|
||||
- **Upsert Operations**: Proper `INSERT OR REPLACE` for data integrity
|
||||
- **History Queries**: Retrieves earliest import dates from application databases
|
||||
|
||||
## Directory Structure Requirements
|
||||
|
||||
### Movies
|
||||
```
|
||||
/movies/
|
||||
└── Movie Title (2024) [tt1234567]/
|
||||
├── movie.mkv
|
||||
└── movie.nfo (created/updated by NFOGuard)
|
||||
```
|
||||
|
||||
### TV Shows
|
||||
```
|
||||
/tv/
|
||||
└── Series Title (2024) [tt1234567]/
|
||||
├── Season 01/
|
||||
│ ├── Series S01E01.mkv
|
||||
│ ├── S01E01.nfo (created/updated by NFOGuard)
|
||||
│ └── S01E02.nfo
|
||||
├── Season 02/
|
||||
└── tvshow.nfo (created/updated by NFOGuard)
|
||||
```
|
||||
|
||||
## Configuration System
|
||||
|
||||
### Environment Files
|
||||
- **`.env`**: Main configuration (paths, behavior, non-sensitive options)
|
||||
- **`.env.secrets`**: Sensitive data (API keys, database credentials)
|
||||
|
||||
### Key Settings
|
||||
- **Media Paths**: Container and application path mappings
|
||||
- **Release Date Priority**: Configurable date preference order
|
||||
- **Debug Modes**: `DEBUG`, `PATH_DEBUG`, `SUPPRESS_TVDB_WARNINGS`
|
||||
- **Database Credentials**: PostgreSQL connection details
|
||||
- **API Keys**: TMDB, TVDB integration (TVDB optional)
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/health` | GET | Health check and status |
|
||||
| `/webhook/radarr` | POST | Radarr webhook handler |
|
||||
| `/webhook/sonarr` | POST | Sonarr webhook handler |
|
||||
| `/manual/scan` | POST | Manual media scanning |
|
||||
| `/bulk/update` | POST | Bulk movie updates from Radarr DB |
|
||||
|
||||
## Critical Bug Fixes & Technical Details
|
||||
|
||||
### Database Upsert Bug (v1.5.5) - CRITICAL
|
||||
- **Location**: `core/database.py:173-186`
|
||||
- **Problem**: `upsert_movie_dates()` was UPDATE-only, not proper upsert
|
||||
- **Symptoms**: Manual scans weren't saving `dateadded`, webhooks fell back to current timestamp
|
||||
- **Fix**: Changed to `INSERT OR REPLACE` with `COALESCE` for path preservation
|
||||
- **Impact**: Fixed webhook date accuracy and database integrity
|
||||
|
||||
### Webhook Date Logic Bug (v1.5.4)
|
||||
- **Location**: `nfoguard.py:975-990`
|
||||
- **Problem**: Upgrade webhooks bypassed full date decision logic
|
||||
- **Symptoms**: Used current timestamp instead of existing Radarr import dates
|
||||
- **Fix**: Webhook mode now uses same date priority as manual scans
|
||||
- **Flow**: Database → Radarr/Sonarr API → Release dates → Current timestamp
|
||||
|
||||
### Radarr Event Filtering (v1.5.3)
|
||||
- **Location**: `nfoguard.py:1479`
|
||||
- **Problem**: Missing event type filtering (Sonarr had it, Radarr didn't)
|
||||
- **Fix**: Added `if event_type not in ["Download", "Upgrade", "Rename"]:`
|
||||
- **Result**: Both systems now follow identical webhook processing
|
||||
|
||||
### NFO-Based Identification System (v1.6.0)
|
||||
- **Location**: `core/nfo_manager.py:85-149`
|
||||
- **Method**: `parse_imdb_from_path_or_nfo()` tries directory first, then NFO files
|
||||
- **Supports**: `<uniqueid type="imdb">`, `<imdbid>`, `<imdb>` XML tags
|
||||
- **Files**: movie.nfo and tvshow.nfo parsing
|
||||
|
||||
### Smart Episode NFO Renaming (v1.6.0)
|
||||
- **Location**: `core/nfo_manager.py:336-491`
|
||||
- **Detection**: `_find_existing_episode_nfo()` scans season directories
|
||||
- **Process**: Extract metadata → Write to standard format → Remove old file
|
||||
- **Pattern**: Always creates `S##E##.nfo` format
|
||||
- **Safety**: Only removes old file after successful write
|
||||
|
||||
## Development History & Major Fixes
|
||||
|
||||
### v1.6.0 (September 16, 2025) - Current
|
||||
- **NFO-Based Identification**: Added dual identification system (directory + NFO parsing)
|
||||
- **Smart NFO Renaming**: Detects and renames non-standard episode NFO files to `S##E##.nfo` format
|
||||
- **Metadata Preservation**: Maintains existing metadata during NFO operations
|
||||
- **Enhanced Documentation**: Comprehensive README and project documentation
|
||||
|
||||
### v1.5.5 (September 15, 2025)
|
||||
- **CRITICAL DATABASE FIX**: Fixed `upsert_movie_dates()` from UPDATE-only to proper `INSERT OR REPLACE`
|
||||
- **Data Integrity**: Manual scans now properly save `dateadded` to database
|
||||
- **Webhook Fix**: Webhooks now find existing database entries correctly
|
||||
|
||||
### v1.5.4 (September 15, 2025)
|
||||
- **Webhook Date Logic Fix**: Fixed webhook processing to use proper date decision logic
|
||||
- **Smart Fallback**: Database → Radarr import dates → release dates → timestamp priority
|
||||
|
||||
### v1.5.3 (September 15, 2025)
|
||||
- **Radarr Rename Fix**: Added missing event type filtering to Radarr webhooks
|
||||
- **Consistency**: Radarr now matches Sonarr's event handling
|
||||
|
||||
### v1.5.2
|
||||
- **TVDB Warning Suppression**: Added `SUPPRESS_TVDB_WARNINGS` for cleaner logs
|
||||
- **Production Ready**: Reduced log noise for production environments
|
||||
|
||||
## Date Decision Logic
|
||||
|
||||
### Movies
|
||||
1. **Database Check**: Existing NFOGuard database entry
|
||||
2. **Radarr History**: Earliest import date from Radarr database
|
||||
3. **Release Dates**: Digital → Physical → Theatrical (configurable priority)
|
||||
4. **File Dates**: File system timestamps (fallback)
|
||||
5. **Current Timestamp**: Absolute last resort
|
||||
|
||||
### TV Shows
|
||||
1. **Database Check**: Existing NFOGuard database entry
|
||||
2. **Sonarr History**: Earliest import date from Sonarr database
|
||||
3. **Air Dates**: Episode air date from metadata
|
||||
4. **File Dates**: File system timestamps (fallback)
|
||||
5. **Current Timestamp**: Absolute last resort
|
||||
|
||||
## Workflow Patterns
|
||||
|
||||
### Webhook Processing
|
||||
1. **Event Validation**: Filter supported event types
|
||||
2. **Media Detection**: Parse file paths and identify media
|
||||
3. **Database Lookup**: Check for existing entries
|
||||
4. **Date Resolution**: Apply decision logic hierarchy
|
||||
5. **NFO Processing**: Create/update NFO files with metadata
|
||||
6. **Database Update**: Store processed information
|
||||
|
||||
### Manual Scanning
|
||||
1. **Directory Traversal**: Scan configured media paths
|
||||
2. **Media Identification**: Directory names and NFO files
|
||||
3. **Batch Processing**: Process multiple items efficiently
|
||||
4. **Database Persistence**: Store all discovered media
|
||||
5. **NFO Creation**: Generate/update all NFO files
|
||||
|
||||
## Integration Points
|
||||
|
||||
### External Services
|
||||
- **Radarr**: Movie management, webhook triggers, database queries
|
||||
- **Sonarr**: TV show management, webhook triggers, database queries
|
||||
- **TMDB**: Movie metadata and release dates (primary)
|
||||
- **TVDB**: TV show metadata (optional, failures suppressed)
|
||||
- **Emby/Jellyfin**: NFO consumption for media servers
|
||||
|
||||
### File System
|
||||
- **Read Access**: Media directories for scanning and identification
|
||||
- **Write Access**: NFO file creation and updates
|
||||
- **Path Mapping**: Container paths vs application paths
|
||||
|
||||
## Development Patterns
|
||||
|
||||
### Code Organization
|
||||
- **Modular Design**: Separate concerns (database, NFO, logging, config)
|
||||
- **Error Handling**: Comprehensive exception handling with logging
|
||||
- **Configuration Driven**: Environment-based configuration management
|
||||
- **Docker Native**: Designed for containerized deployment
|
||||
|
||||
### Testing Strategy
|
||||
- **Manual Testing**: API endpoints and webhook simulation
|
||||
- **Real-world Integration**: Actual Radarr/Sonarr webhook testing
|
||||
- **Database Validation**: Direct database query verification
|
||||
- **File System Testing**: NFO file creation and parsing validation
|
||||
|
||||
## Common Issues & Solutions
|
||||
|
||||
### Path Mapping
|
||||
- **Problem**: Container paths don't match application paths
|
||||
- **Solution**: Proper volume mapping and path configuration in `.env`
|
||||
- **Debug**: Enable `PATH_DEBUG=true` to see path resolution
|
||||
|
||||
### Database Connectivity
|
||||
- **Problem**: Cannot connect to Radarr/Sonarr databases
|
||||
- **Solution**: Verify credentials in `.env.secrets` and network connectivity
|
||||
- **Check**: Database host, port, username, password in secrets file
|
||||
|
||||
### Permission Issues
|
||||
- **Problem**: Cannot write NFO files
|
||||
- **Solution**: Ensure proper file permissions and volume mount configurations
|
||||
- **Fix**: `chmod 755` on media directories, `rw` mount permissions
|
||||
|
||||
### TVDB Failures
|
||||
- **Problem**: TVDB API errors causing log noise
|
||||
- **Solution**: Set `SUPPRESS_TVDB_WARNINGS=true` (system works without TVDB)
|
||||
- **Note**: TVDB is optional - IMDb and Sonarr data are primary sources
|
||||
|
||||
### Webhook Issues
|
||||
- **Problem**: Webhooks not triggering NFO updates
|
||||
- **Debug Steps**:
|
||||
1. Check webhook URLs: `http://nfoguard:8080/webhook/radarr|sonarr`
|
||||
2. Verify event types: Download, Upgrade, Rename only
|
||||
3. Enable `DEBUG=true` for detailed webhook logging
|
||||
4. Check database entries with manual scan first
|
||||
|
||||
### Date Problems
|
||||
- **Problem**: Wrong dates in NFO files (current timestamp instead of import date)
|
||||
- **Root Cause**: Usually database upsert issues or missing database entries
|
||||
- **Solution**: Run manual scan first to populate database, then test webhooks
|
||||
- **Priority**: Database → API → Release dates → File dates → Current timestamp
|
||||
|
||||
## Future Considerations
|
||||
|
||||
### Potential Enhancements
|
||||
- **Movie NFO Renaming**: Similar to TV show episode renaming
|
||||
- **Multiple NFO Formats**: Support for different media server preferences
|
||||
- **Metadata Enhancement**: Additional metadata sources and fields
|
||||
- **Performance Optimization**: Caching and batch processing improvements
|
||||
|
||||
### Monitoring & Maintenance
|
||||
- **Health Checks**: Regular endpoint monitoring
|
||||
- **Log Analysis**: Debug mode for troubleshooting
|
||||
- **Database Maintenance**: Regular cleanup of old entries
|
||||
- **Version Management**: Semantic versioning and change tracking
|
||||
|
||||
## Development Environment
|
||||
|
||||
### Prerequisites
|
||||
- Docker and Docker Compose
|
||||
- Python 3.x development environment
|
||||
- Access to Radarr/Sonarr instances with PostgreSQL
|
||||
- Git workflow familiarity
|
||||
|
||||
### Development Workflow
|
||||
1. **Local Testing**: Use development Docker image
|
||||
2. **Code Changes**: Implement features with proper logging
|
||||
3. **Version Updates**: Update VERSION file and SUMMARY.md
|
||||
4. **Documentation**: Update README.md and Project.md
|
||||
5. **Deployment**: Push to dev branch, then production
|
||||
|
||||
## Quick Context Recovery
|
||||
|
||||
### When Starting a New Session
|
||||
1. **Read Project.md** - This file for full context
|
||||
2. **Check SUMMARY.md** - Latest changes and current version
|
||||
3. **Review VERSION** - Current version number
|
||||
4. **Key Files to Understand**:
|
||||
- `nfoguard.py` - Main logic, webhook handlers
|
||||
- `core/nfo_manager.py` - NFO creation, parsing, renaming
|
||||
- `core/database.py` - Database operations (watch the upsert bug fix!)
|
||||
|
||||
### Critical Code Locations to Remember
|
||||
- **Database Upsert**: `core/database.py:173-186` (fixed UPDATE → INSERT OR REPLACE)
|
||||
- **Webhook Date Logic**: `nfoguard.py:975-990` (fixed fallback behavior)
|
||||
- **Event Filtering**: `nfoguard.py:1429,1479` (Download/Upgrade/Rename only)
|
||||
- **NFO Renaming**: `core/nfo_manager.py:336-491` (smart episode renaming)
|
||||
- **Dual Identification**: `core/nfo_manager.py:85-149` (directory + NFO parsing)
|
||||
|
||||
### User Context
|
||||
- **User has**: Radarr, Sonarr, Unmanic integration
|
||||
- **Environment**: Docker deployment, PostgreSQL databases
|
||||
- **Focus Areas**: Webhook accuracy, NFO standardization, date integrity
|
||||
- **Testing Method**: Real webhooks from Radarr/Sonarr, manual scans for validation
|
||||
|
||||
### Recent Work Pattern
|
||||
- User reports issues with specific examples (log snippets, webhook behavior)
|
||||
- We debug by analyzing code at specific line numbers
|
||||
- Implement fixes with proper error handling and logging
|
||||
- Update documentation (SUMMARY.md, README.md, Project.md)
|
||||
- Version bump and prepare for deployment
|
||||
|
||||
This project represents a mature, production-ready system for automated NFO management with comprehensive webhook integration and intelligent metadata handling.
|
||||
+15
-2
@@ -1,8 +1,21 @@
|
||||
# NFOGuard Development Summary
|
||||
|
||||
## Current Status: v1.5.5 - Database Upsert Fix! 🔧
|
||||
## Current Status: v1.6.0 - NFO-Based Identification! 📄
|
||||
|
||||
### Latest Updates (v1.5.5) - September 15, 2025
|
||||
### Latest Updates (v1.6.0) - September 16, 2025
|
||||
- **📄 NFO IDENTIFICATION**: Added support for identifying movies/TV shows from NFO files
|
||||
- **🔄 DUAL METHOD**: Directory name IMDb IDs (primary) + NFO file IMDb IDs (fallback)
|
||||
- **📋 NFO FORMATS**: Supports `<uniqueid type="imdb">`, `<imdbid>`, and `<imdb>` XML tags
|
||||
- **🎬 MOVIE SUPPORT**: Reads movie.nfo files for IMDb ID extraction
|
||||
- **📺 TV SUPPORT**: Reads tvshow.nfo files for series identification
|
||||
- **✅ SMART FALLBACK**: Only checks NFO files when directory name lacks IMDb ID
|
||||
- **📝 DOCUMENTATION**: Updated README with NFO identification examples and formats
|
||||
- **🔄 NFO RENAMING**: Added smart episode NFO file renaming to standardized `S##E##.nfo` format
|
||||
- **🎯 SONARR INTEGRATION**: Detects existing NFO files created by Sonarr/other tools, extracts metadata, and renames to standard format
|
||||
- **📁 METADATA PRESERVATION**: Preserves all existing metadata when renaming non-standard NFO files
|
||||
- **🧹 CLEANUP**: Removes old non-standard NFO files after successful rename to prevent duplicates
|
||||
|
||||
### Previous Updates (v1.5.5) - September 15, 2025
|
||||
- **🔧 DATABASE BUG FIX**: Fixed `upsert_movie_dates` to be a proper upsert operation
|
||||
- **💾 DATA INTEGRITY**: Manual scans now properly save `dateadded` to database
|
||||
- **🎯 ROOT CAUSE**: Fixed UPDATE-only operation that was leaving `dateadded` fields NULL
|
||||
|
||||
Reference in New Issue
Block a user