sbcrumb/nfoguard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
SBCrumb
2025-09-09 10:59:45 -04:00
parent 49a2c1788a
commit 4a9aecbc22
12 changed files with 81 additions and 4121 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.template
+8
View File
@@ -90,6 +90,14 @@ ALLOW_FILE_DATE_FALLBACK=false
# digital = VOD/streaming release, physical = DVD/Blu-ray, theatrical = cinema release
RELEASE_DATE_PRIORITY=digital,physical,theatrical
# Smart date validation: automatically prefer theatrical over unreasonably late digital/physical dates
# Example: "The Craft (1996)" theatrical vs "The Craft (2022)" physical → chooses 1996 theatrical
ENABLE_SMART_DATE_VALIDATION=true
# Maximum reasonable gap between theatrical and digital/physical release (in years)
# If digital/physical is more than this many years after theatrical, prefer theatrical instead
MAX_RELEASE_DATE_GAP_YEARS=10
# When to query APIs: always, if_missing, never
MOVIE_POLL_MODE=always
API_ENDPOINTS.md
-218
View File
@@ -1,218 +0,0 @@
# NFOGuard API Endpoints
## Health & Status Endpoints
### GET `/health`
Health check endpoint with comprehensive status information.
**Example:**
```bash
curl http://localhost:8080/health
```
**Response:**
```json
{
"status": "healthy",
"version": "0.5.0",
"uptime": "0:15:23.456789",
"database_status": "healthy",
"radarr_database": {
"status": "healthy",
"database_type": "postgresql",
"connection": "readable",
"readable": true,
"tables_exist": true,
"sample_data": true,
"functional": true,
"movie_count": 1250,
"history_count": 5847,
"movies_with_imdb": 1248,
"connection_info": {
"type": "postgresql",
"host": "postgres",
"port": 5432,
"database": "radarr-main"
},
"tested_at": "2025-09-08T17:30:15+00:00"
}
}
```
### GET `/stats`
Database statistics for NFOGuard's internal database.
**Example:**
```bash
curl http://localhost:8080/stats
```
### GET `/batch/status`
Current batch processing queue status.
**Example:**
```bash
curl http://localhost:8080/batch/status
```
## Debug & Testing Endpoints
### GET `/debug/movie/{imdb_id}`
**Primary testing endpoint** - Debug movie import date detection for a specific IMDb ID.
**Examples:**
```bash
# Test with your existing command
curl http://localhost:8080/debug/movie/tt1674782
# Test other movies
curl http://localhost:8080/debug/movie/tt1596343 # Fast & Furious 6
curl http://localhost:8080/debug/movie/tt0468569 # The Dark Knight
```
**Response includes:**
- Movie lookup results (database vs API)
- Import date detection with performance metrics
- Source information (radarr:db.history.import vs radarr:api.history.import)
- Fallback chain details
### GET `/debug/movie/{imdb_id}/history`
Detailed history analysis for a movie (if available in your version).
## Webhook Endpoints
### POST `/webhook/radarr`
Radarr webhook endpoint for real-time processing.
**Test webhook manually:**
```bash
# Simulate a Radarr Download event
curl -X POST http://localhost:8080/webhook/radarr \
-H "Content-Type: application/json" \
-d '{
"eventType": "Download",
"movie": {
"id": 123,
"imdbId": "tt1674782",
"title": "The Hobbit: An Unexpected Journey",
"year": 2012
},
"movieFile": {
"path": "/movies/The Hobbit An Unexpected Journey (2012) [imdb-tt1674782]/The Hobbit An Unexpected Journey (2012).mkv"
}
}'
```
### POST `/webhook/sonarr`
Sonarr webhook endpoint for TV shows.
**Test webhook manually:**
```bash
# Simulate a Sonarr Download event
curl -X POST http://localhost:8080/webhook/sonarr \
-H "Content-Type: application/json" \
-d '{
"eventType": "Download",
"series": {
"imdbId": "tt0944947",
"title": "Game of Thrones"
},
"episodes": [{
"seasonNumber": 1,
"episodeNumber": 1
}]
}'
```
## Manual Processing Endpoints
### POST `/manual/scan`
Trigger manual library scans.
**Examples:**
```bash
# Scan everything
curl -X POST "http://localhost:8080/manual/scan?scan_type=both"
# Scan only movies
curl -X POST "http://localhost:8080/manual/scan?scan_type=movies"
# Scan only TV shows
curl -X POST "http://localhost:8080/manual/scan?scan_type=tv"
# Scan specific path
curl -X POST "http://localhost:8080/manual/scan?path=/media/movies"
```
## Testing Database Performance
### Run Performance Tests
Use the included test script to compare database vs API performance:
```bash
# Inside NFOGuard container or with proper environment
python test_db_performance.py
```
### Environment Setup for Testing
**For PostgreSQL (Recommended):**
```env
RADARR_DB_TYPE=postgresql
RADARR_DB_HOST=postgres_host
RADARR_DB_PORT=5432
RADARR_DB_NAME=radarr-main
RADARR_DB_USER=postgres
RADARR_DB_PASSWORD=your_password
```
**For SQLite:**
```env
RADARR_DB_TYPE=sqlite
RADARR_DB_PATH=/config/radarr.db
```
## Performance Testing Workflow
1. **Test your existing command** (should now use database if configured):
```bash
curl http://localhost:8080/debug/movie/tt1674782
```
2. **Check health** to verify database connectivity:
```bash
curl http://localhost:8080/health
```
3. **Run performance comparison**:
```bash
python test_db_performance.py
```
4. **Test webhook simulation** with a movie in your library:
```bash
curl -X POST http://localhost:8080/webhook/radarr \
-H "Content-Type: application/json" \
-d '{
"eventType": "Download",
"movie": {"imdbId": "tt1674782", "title": "Test Movie"},
"movieFile": {"path": "/path/to/movie.mkv"}
}'
```
## Expected Performance Improvements
With database access enabled, you should see:
- **Movie lookups**: 5-10x faster
- **Import date detection**: 10-20x faster
- **Bulk operations**: 50x+ faster
- **Reduced API calls**: 90%+ reduction
- **Log output**: Much cleaner, fewer "pages and pages" messages
## Troubleshooting
If database access fails, NFOGuard automatically falls back to API methods. Check the logs for:
- `"Enhanced performance mode: Direct database access enabled"` (success)
- `"Failed to initialize database client, using API fallback"` (fallback)
The `/health` endpoint will show detailed database status and any connection issues.
CHANGELOG.md
+9
View File
@@ -11,6 +11,15 @@ All notable changes to this project will be documented in this file.
- **Automatic Loading**: Both files loaded at startup with python-dotenv
- **Log Security**: API keys automatically masked in all log output
- **Git Protection**: Updated .gitignore prevents accidental commits of sensitive data
- **🧠 Smart Date Validation**: Intelligent release date selection with configurable gap detection
- **Automatic Detection**: Prefers theatrical dates when digital/physical are unreasonably late
- **Configurable**: `MAX_RELEASE_DATE_GAP_YEARS=10` (default: 10 years maximum gap)
- **Example**: "The Craft (1996)" prefers 1996 theatrical over 2022 physical release
- **Toggle**: `ENABLE_SMART_DATE_VALIDATION=true` to enable/disable feature
- **🧹 Repository Cleanup**: Removed legacy and duplicate files
- **Removed**: `media_date_cache.py`, `webhook_server.py` (replaced by nfoguard.py)
- **Removed**: Outdated documentation files (`API_ENDPOINTS.md`, `MIGRATION.md`, `STRUCTURE.md`)
- **Streamlined**: Cleaner codebase with only essential files
- **🎯 Configurable Release Date Priority System**: Revolutionary smart fallback for manual imports
- **Configurable Priority Order**: `RELEASE_DATE_PRIORITY=digital,physical,theatrical` (customizable)
- **Digital Releases**: VOD/streaming dates (Netflix, iTunes, etc.) - TMDB type 4
MIGRATION.md
-163
View File
@@ -1,163 +0,0 @@
# NFOGuard v0.2.0 - Migration Guide
## 🚨 **BREAKING CHANGES** - Complete Refactor
### **What Changed**
- **Complete modular rewrite** addressing path mapping and Radarr performance issues
- **Single consolidated application** (`nfoguard.py`) replaces `webhook_server.py` + `media_date_cache.py`
- **Enhanced path mapping** between container and Radarr/Sonarr paths
- **Optimized Radarr client** with early termination and better import detection
- **Updated environment variables** for better configuration
---
## 📋 **Required Actions for Deployment**
### **1. Update Environment Variables**
Add these **CRITICAL** new variables to your `.env`:
```bash
# Path mapping - REQUIRED for proper operation
RADARR_ROOT_FOLDERS=/mnt/unionfs/Media/Movies/movies,/mnt/unionfs/Media/Movies/movies6
SONARR_ROOT_FOLDERS=/mnt/unionfs/Media/TV/tv,/mnt/unionfs/Media/TV/tv6
# Download detection - helps identify real imports vs renames
DOWNLOAD_PATH_INDICATORS=/downloads/,/nzbget/,/sabnzbd/,/completed/,/torrents/
# Movie processing strategy - NEW options
MOVIE_PRIORITY=import_then_digital # or digital_then_import
MOVIE_POLL_MODE=always # always, if_missing, never
MOVIE_DATE_UPDATE_MODE=backfill_only # or overwrite
```
### **2. Update Docker Configuration**
Your build will automatically use the new structure, but verify:
- ✅ Dockerfile now copies `/clients/`, `/core/`, `/scripts/` directories
- ✅ Main command changed to `python nfoguard.py`
- ✅ Version management system in place
### **3. Git Runner Build**
Your existing CI/CD should work with these changes:
- All new modular files will be included
- Version automatically incremented to `0.2.0`
- Container will start with `nfoguard.py` instead of `webhook_server.py`
---
## 🔧 **Key Problem Fixes**
### **Path Mapping Issue** ✅ **SOLVED**
**Problem:** Container sees `/media/movies` but Radarr sees `/mnt/unionfs/Media/Movies/movies`
**Solution:** New `PathMapper` class handles translation:
```python
# Automatically converts between:
Container: /media/movies/Movie [imdb-tt1234567]
Radarr: /mnt/unionfs/Media/Movies/movies/Movie [imdb-tt1234567]
```
### **Radarr History Performance** ✅ **SOLVED**
**Problem:** "Pages and pages of Radarr imports"
**Solution:** New `RadarrClient` with:
- **Early termination** - stops at first real import (80-90% fewer API calls)
- **Better event detection** - distinguishes downloads from renames
- **Smart path analysis** - identifies nzbget/download sources vs media paths
### **Import Date Accuracy** ✅ **SOLVED**
**Problem:** Hard to find the "right date"
**Solution:** Enhanced detection chain:
1. **Real import events** from downloads (nzbget → Radarr)
2. **Digital release dates** from TMDB/Jellyseerr/OMDb
3. **Grab events** as fallback
4. **File mtime** as last resort
---
## 🗂️ **File Structure Changes**
### **NEW Structure:**
```
NFOguard/
├── nfoguard.py # ← NEW main application
├── clients/ # ← NEW API clients
│ ├── radarr_client.py # ← Enhanced Radarr client
│ ├── sonarr_client.py # ← Extracted Sonarr client
│ └── external_clients.py # ← TMDB/OMDb/Jellyseerr
├── core/ # ← NEW business logic
│ ├── database.py # ← Database operations
│ ├── nfo_manager.py # ← NFO file handling
│ └── path_mapper.py # ← Path translation
├── VERSION # ← Version file
├── Dockerfile # ← Updated for new structure
└── .env.example # ← Updated variables
```
### **OLD Files (can remove after successful deployment):**
- ~~`webhook_server.py`~~ → Replaced by `nfoguard.py`
- ~~`media_date_cache.py`~~ → Split into `/clients/` and `/core/`
---
## 🧪 **Testing Checklist**
### **Before Deployment:**
- [ ] Update `.env` with new variables (especially `RADARR_ROOT_FOLDERS`, `SONARR_ROOT_FOLDERS`)
- [ ] Commit all new files to git
- [ ] Verify git runner builds successfully
- [ ] Check container logs for startup
### **After Deployment:**
- [ ] Test webhook endpoints: `/health`, `/stats`
- [ ] Send test webhook from Radarr/Sonarr
- [ ] Check logs for path mapping success
- [ ] Verify movie/TV processing works
- [ ] Monitor Radarr API call reduction
### **Rollback Plan (if needed):**
1. Revert to previous git commit
2. Rebuild container with old `webhook_server.py`
3. Restore old environment variables
---
## 📊 **Expected Improvements**
### **Performance:**
- **80-90% reduction** in Radarr API calls (early termination)
- **Faster webhook processing** (better caching and event detection)
- **Reduced container memory usage** (modular architecture)
### **Reliability:**
- **Proper path mapping** (no more path mismatch issues)
- **Better import detection** (distinguishes real downloads from renames)
- **Enhanced error handling** (modular components with clear interfaces)
### **Maintainability:**
- **Smaller, focused modules** (easier debugging)
- **Clear separation of concerns** (API clients, business logic, database)
- **Comprehensive logging** (better troubleshooting)
---
## 🆘 **Support**
### **Common Issues:**
1. **"Movie directory not found"** → Check `RADARR_ROOT_FOLDERS` mapping
2. **"No real import events found"** → Check `DOWNLOAD_PATH_INDICATORS`
3. **"Series directory not found"** → Check `SONARR_ROOT_FOLDERS` mapping
### **Debug Commands:**
```bash
# Check version
curl http://localhost:8080/health
# Check path mapping
docker logs nfoguard_container | grep "Mapped.*path"
# Check API connectivity
docker logs nfoguard_container | grep "Radarr\|Sonarr"
```
This refactor addresses all your core issues while maintaining compatibility with your git-based deployment workflow.
STRUCTURE.md
-137
View File
@@ -1,137 +0,0 @@
# NFOGuard Code Organization
## New Modular Structure
### `/clients/` - API Clients
- **`radarr_client.py`** - Enhanced Radarr API client with optimized history parsing
- Improved event type detection for real imports vs upgrades/renames
- Early termination when first import found (stops scanning unnecessary history)
- Better path analysis for download source detection
- Fallback strategies for missing data
- **`sonarr_client.py`** - Sonarr API client extracted from media_date_cache.py
- Series lookup by IMDb ID or title
- Episode import history analysis
- Enhanced error handling and retries
### `/core/` - Core Functionality
- **`nfo_manager.py`** - NFO file creation and management
- Movie and TV episode NFO creation
- XML parsing and pretty-printing
- IMDb ID extraction from paths and NFO files
- File/directory timestamp management
- Orphaned NFO cleanup
- **`database.py`** - Database operations
- TV series and episode management
- Movie metadata storage
- Statistics and health checks
- Orphaned record cleanup
- Schema management and migrations
### Current Files (to be refactored)
- **`webhook_server.py`** - Main FastAPI application (1977+ lines)
- Should be split into smaller modules
- Contains duplicate client code that can now use `/clients/`
- Mix of concerns that should use `/core/` modules
- **`media_date_cache.py`** - TV processing logic
- Can be simplified to use new client modules
- Focus on coordination rather than API calls
## Benefits of New Structure
### 1. **Separation of Concerns**
- API clients handle only API communication
- Core modules handle only business logic
- Database operations isolated and testable
- NFO management centralized
### 2. **Improved Radarr Data Extraction**
- **Early Termination**: Stops querying when first real import found
- **Better Event Detection**: Distinguishes real downloads from renames/upgrades
- **Path Analysis**: Improved detection of download vs existing file operations
- **Reduced API Calls**: Optimized pagination and caching
### 3. **Enhanced Testing & Debugging**
- Each module can be tested independently
- Clear interfaces between components
- Easier to debug specific API issues
- Modular replacement of components
### 4. **Better Maintainability**
- Smaller, focused files
- Clear responsibilities
- Easier to add new features
- Simpler code reviews
## Migration Complete ✅
### **OLD FILES (can be removed):**
- `webhook_server.py` → Replaced by `nfoguard.py`
- `media_date_cache.py` → Functionality distributed to `/clients/` and `/core/`
### **NEW MAIN APPLICATION:**
- **`nfoguard.py`** - Single consolidated application
- Uses all new modular components
- Handles both TV and movie processing
- Improved path mapping and date detection
- Cleaner webhook handling
### **BREAKING CHANGES in v0.2.0:**
1. **Environment Variables Changed:**
- Added `RADARR_ROOT_FOLDERS` and `SONARR_ROOT_FOLDERS`
- Added `DOWNLOAD_PATH_INDICATORS`
- Updated `MOVIE_PRIORITY`, `MOVIE_POLL_MODE`, `MOVIE_DATE_UPDATE_MODE`
2. **Docker Command Changed:**
- OLD: `CMD ["python", "webhook_server.py"]`
- NEW: `CMD ["python", "nfoguard.py"]`
3. **File Structure:**
- Must copy `/clients/` and `/core/` directories
- Must copy `VERSION` file
## Key Improvements for Radarr Issues
### **Problem**: "Pages and pages of Radarr history"
**Solution**: `RadarrClient.earliest_import_event_optimized()`
- Processes events chronologically (oldest first)
- Stops immediately when first real import found
- Uses smaller page sizes (50 vs 1000)
- Early termination reduces API calls by 80-90%
### **Problem**: "Determining the right date"
**Solution**: Enhanced event analysis
- Distinguishes download imports from renames/upgrades
- Path-based detection of download sources
- Fallback chain: real import → grab event → file dateAdded
### **Problem**: "Matching right movieID to right file on disk"
**Solution**: Multiple lookup strategies
- Direct IMDb lookup with validation
- All-movies scan with filtering
- Lookup endpoint with verification
- Prevents wrong movie matching
## Usage Examples
```python
# New Radarr client
from clients.radarr_client import RadarrClient
client = RadarrClient(base_url, api_key)
movie = client.movie_by_imdb("tt1596343")
import_date, source = client.get_movie_import_date(movie_id)
# NFO management
from core.nfo_manager import NFOManager
nfo = NFOManager("NFOGuard")
nfo.create_movie_nfo(movie_dir, imdb_id, dateadded, released, source)
# Database operations
from core.database import NFOGuardDatabase
db = NFOGuardDatabase(db_path)
db.upsert_movie_dates(imdb_id, released, dateadded, source, has_video=True)
```
This modular structure addresses the core issues while making the codebase more maintainable and testable.
TEST_PRIORITY_SYSTEM.md
-137
View File
@@ -1,137 +0,0 @@
# NFOGuard Release Date Priority System - Testing Guide
## 🧪 Test Your Configured Priority System
### 1. **Test "The Craft (1996)" - Theatrical Fallback**
```bash
# Test step-by-step TMDB lookup
curl "http://localhost:8080/debug/tmdb/tt0115963"
# Expected: digital=0, physical=multiple, theatrical=1996-05-03
# Should show: no digital releases, DVD from 2000, theatrical from 1996
# Test priority logic
curl "http://localhost:8080/debug/movie/tt0115963/priority"
# Expected result with your config (digital,physical,theatrical):
# - digital: not available
# - physical: 2000-09-12 (too late - DVD release)
# - theatrical: 1996-05-03 ← SELECTED (best option)
```
### 2. **Test Modern Movie - Digital Priority**
```bash
# Test a recent movie like Top Gun Maverick
curl "http://localhost:8080/debug/tmdb/tt1745960"
# Expected: digital, physical, and theatrical dates all available
# Should show your priority order preference
curl "http://localhost:8080/debug/movie/tt1745960/priority"
# Expected with your config (digital,physical,theatrical):
# - digital: 2022-08-23 ← SELECTED (first in priority)
# - physical: 2022-10-31 (skipped)
# - theatrical: 2022-05-27 (skipped)
```
### 3. **Test Different Priority Orders**
**Your Current Config**: `RELEASE_DATE_PRIORITY=digital,physical,theatrical`
```bash
curl "http://localhost:8080/debug/movie/tt1745960/priority"
# Should select: digital date
```
**Alternative Config** (edit .env temporarily): `RELEASE_DATE_PRIORITY=theatrical,digital,physical`
```bash
# Restart container after changing .env
docker-compose restart
curl "http://localhost:8080/debug/movie/tt1745960/priority"
# Should now select: theatrical date (earliest option)
```
### 4. **Test File Date Fallback Logic**
**Movies with manual imports** (showing `radarr:db.file.dateAdded`):
```bash
curl "http://localhost:8080/debug/movie/tt0115963/priority"
# Expected response:
# {
# "release_date_priority": ["digital", "physical", "theatrical"],
# "date_sources": {
# "radarr_import": {"date": "2025-08-29...", "source": "radarr:db.file.dateAdded"},
# "digital_release": {"date": "1996-05-03...", "source": "tmdb:theatrical"}
# },
# "file_date_detected": true,
# "would_prefer_digital": true,
# "selected_date": "1996-05-03T00:00:00+00:00",
# "selected_source": "tmdb:theatrical (preferred over file date)"
# }
```
### 5. **Verify NFO Source Annotations**
After processing a movie:
```bash
# Process the movie
curl -X POST "http://localhost:8080/manual/scan?scan_type=movies"
# Check the created NFO file
cat "/media/Movies/movies/The Craft (1996) [imdb-tt0115963]/movie.nfo"
# Should contain at the end:
# <!-- source: Movie; tmdb:theatrical -->
# <!-- managed by NFOGuard -->
```
## 🎯 Expected Behavior by Movie Type
### **Pre-Digital Era Movies (1990s-early 2000s)**
- **Digital**: ❌ Not available
- **Physical**: ⚠️ Often years later (DVD era)
- **Theatrical**: ✅ **Most accurate** (selected)
- **Example**: The Craft, Titanic, The Matrix
### **Transition Era Movies (2000s-2010s)**
- **Digital**: ⚠️ Limited early VOD
- **Physical**: ✅ **Often most relevant** (DVD/Blu-ray boom)
- **Theatrical**: ✅ Available
- **Example**: Dark Knight, Avatar, Iron Man
### **Modern Movies (2010s+)**
- **Digital**: ✅ **Most relevant** (streaming era)
- **Physical**: ✅ Available but later
- **Theatrical**: ✅ Available
- **Example**: Top Gun Maverick, Avengers, Everything Everywhere
## 🔧 Troubleshooting Priority System
### No Release Dates Found
```bash
curl "http://localhost:8080/debug/tmdb/tt0123456"
# Check if movie exists in TMDB and what release data is available
```
### Unexpected Date Selection
```bash
curl "http://localhost:8080/debug/movie/tt0123456/priority"
# Shows exactly which dates were found and why one was selected
```
### Priority Order Not Working
1. Check your `.env` configuration
2. Restart container: `docker-compose restart`
3. Verify with debug endpoint: shows `"release_date_priority"` array
## 🎉 Success Indicators
- ✅ Old movies use theatrical dates instead of file dates
- ✅ New movies use digital dates per your preference
- ✅ NFO files show source in comments
- ✅ Priority order is respected and configurable
- ✅ Smart fallbacks prevent unrealistic dates
Your NFOGuard now has **intelligent, configurable date selection** that adapts to each movie's era and available release data! 🎯
VERSION
+1 -1
View File
@@ -1 +1 @@
0.5.1
0.6.0
clients/external_clients.py
+56 -3
View File
@@ -287,8 +287,8 @@ class ExternalClientManager:
self.omdb = OMDbClient()
self.jellyseerr = JellyseerrClient()
def get_release_date_by_priority(self, imdb_id: str, priority_order: List[str]) -> Optional[Tuple[str, str]]:
"""Get release date using configurable priority order"""
def get_release_date_by_priority(self, imdb_id: str, priority_order: List[str], enable_smart_validation: bool = True) -> Optional[Tuple[str, str]]:
"""Get release date using configurable priority order with smart date validation"""
# Get all possible release dates
release_options = {}
@@ -326,13 +326,66 @@ class ExternalClientManager:
earliest_jellyseerr = min(jellyseerr_dates)
release_options["digital"] = (earliest_jellyseerr, "jellyseerr:digital")
# Return first available option according to priority
# Smart date validation: Check if priority order makes sense given the actual dates
if enable_smart_validation and len(release_options) > 1:
validated_choice = self._validate_date_choice(release_options, priority_order)
if validated_choice:
return validated_choice
# Return first available option according to priority (fallback behavior)
for priority in priority_order:
if priority in release_options:
return release_options[priority]
return None
def _validate_date_choice(self, release_options: Dict[str, Tuple[str, str]], priority_order: List[str]) -> Optional[Tuple[str, str]]:
"""Validate date choice and prefer theatrical if digital/physical are unreasonably late"""
from datetime import datetime, timezone
import os
# Get configuration for maximum gap (default: 10 years)
max_reasonable_gap_years = int(os.environ.get("MAX_RELEASE_DATE_GAP_YEARS", "10"))
# Parse all available dates
parsed_dates = {}
for release_type, (date_str, source) in release_options.items():
try:
parsed_dates[release_type] = (datetime.fromisoformat(date_str.replace('Z', '+00:00')), source)
except Exception:
continue
if not parsed_dates or "theatrical" not in parsed_dates:
return None # No smart validation possible without theatrical date
theatrical_date, theatrical_source = parsed_dates["theatrical"]
# Check each priority option against theatrical date
for priority in priority_order:
if priority == "theatrical":
continue # Skip theatrical in this validation
if priority in parsed_dates:
priority_date, priority_source = parsed_dates[priority]
# Calculate the gap in years
gap = (priority_date - theatrical_date).days / 365.25
# If the gap is too large, skip this priority and continue
if gap > max_reasonable_gap_years:
print(f"[SMART VALIDATION] {priority} date {priority_date.strftime('%Y-%m-%d')} is {gap:.1f} years after theatrical {theatrical_date.strftime('%Y-%m-%d')}, preferring theatrical")
continue
# This priority option is reasonable, use it
return (priority_date.isoformat(timespec="seconds"), f"{priority_source} (validated)")
# If all priority options are unreasonable, fall back to theatrical
if "theatrical" in release_options:
theatrical_date_str, theatrical_source = release_options["theatrical"]
return (theatrical_date_str, f"{theatrical_source} (smart fallback)")
return None
def get_digital_release_candidates(self, imdb_id: str) -> List[Tuple[str, str]]:
"""Get digital release date candidates from all sources (legacy method)"""
candidates = []
media_date_cache.py
-1270
View File
File diff suppressed because it is too large Load Diff
nfoguard.py
+7 -1
View File
@@ -148,6 +148,8 @@ class NFOGuardConfig:
self.prefer_release_dates_over_file_dates = _bool_env("PREFER_RELEASE_DATES_OVER_FILE_DATES", True)
self.allow_file_date_fallback = _bool_env("ALLOW_FILE_DATE_FALLBACK", False)
self.release_date_priority = [p.strip() for p in os.environ.get("RELEASE_DATE_PRIORITY", "digital,physical,theatrical").split(",")]
self.enable_smart_date_validation = _bool_env("ENABLE_SMART_DATE_VALIDATION", True)
self.max_release_date_gap_years = int(os.environ.get("MAX_RELEASE_DATE_GAP_YEARS", "10"))
self.movie_poll_mode = os.environ.get("MOVIE_POLL_MODE", "always").lower()
self.movie_update_mode = os.environ.get("MOVIE_DATE_UPDATE_MODE", "backfill_only").lower()
@@ -568,7 +570,11 @@ class MovieProcessor:
def _get_digital_release_date(self, imdb_id: str) -> Tuple[Optional[str], str]:
"""Get release date from external sources using configured priority"""
release_result = self.external_clients.get_release_date_by_priority(imdb_id, config.release_date_priority)
release_result = self.external_clients.get_release_date_by_priority(
imdb_id,
config.release_date_priority,
enable_smart_validation=config.enable_smart_date_validation
)
if release_result:
return release_result[0], release_result[1]
return None, "release:none"
test_db_performance.py
-214
View File
@@ -1,214 +0,0 @@
#!/usr/bin/env python3
"""
Test script for comparing Radarr API vs Database performance
Run this to validate the new database client functionality
"""
import os
import time
from pathlib import Path
from clients.radarr_client import RadarrClient
from clients.radarr_db_client import RadarrDbClient
def test_database_connection():
"""Test database connection and basic functionality"""
print("=== Testing Database Connection ===")
db_client = RadarrDbClient.from_env()
if not db_client:
print("❌ Database client not configured - check environment variables")
return False
print("✅ Database connection successful")
# Test basic stats
stats = db_client.get_database_stats()
print(f"Database stats: {stats}")
return True
def test_movie_lookup_performance():
"""Compare API vs Database performance for movie lookups"""
print("\n=== Testing Movie Lookup Performance ===")
# Test IMDb IDs - adjust these to movies in your library
test_imdb_ids = [
"tt1596343", # Fast & Furious 6
"tt0468569", # The Dark Knight
"tt0137523", # Fight Club
"tt0109830", # Forrest Gump
"tt0111161" # The Shawshank Redemption
]
# Initialize clients
radarr_url = os.environ.get("RADARR_URL")
radarr_api_key = os.environ.get("RADARR_API_KEY")
if not radarr_url or not radarr_api_key:
print("❌ RADARR_URL and RADARR_API_KEY required for testing")
return
client = RadarrClient(radarr_url, radarr_api_key)
# Test API performance
print("\n--- API Performance ---")
api_times = []
api_found = 0
for imdb_id in test_imdb_ids:
start_time = time.time()
# Temporarily disable database client for API-only test
original_db_client = client.db_client
client.db_client = None
movie = client.movie_by_imdb(imdb_id)
# Restore database client
client.db_client = original_db_client
elapsed = time.time() - start_time
api_times.append(elapsed)
if movie:
api_found += 1
print(f" {imdb_id}: {elapsed:.3f}s - Found: {movie.get('title')}")
else:
print(f" {imdb_id}: {elapsed:.3f}s - Not found")
# Test Database performance
print("\n--- Database Performance ---")
db_times = []
db_found = 0
if client.db_client:
for imdb_id in test_imdb_ids:
start_time = time.time()
movie = client.movie_by_imdb(imdb_id) # This will use database first
elapsed = time.time() - start_time
db_times.append(elapsed)
if movie:
db_found += 1
print(f" {imdb_id}: {elapsed:.3f}s - Found: {movie.get('title')}")
else:
print(f" {imdb_id}: {elapsed:.3f}s - Not found")
else:
print(" Database client not available")
return
# Performance comparison
print("\n--- Performance Summary ---")
avg_api_time = sum(api_times) / len(api_times)
avg_db_time = sum(db_times) / len(db_times)
print(f"API Average: {avg_api_time:.3f}s ({api_found}/{len(test_imdb_ids)} found)")
print(f"Database Average: {avg_db_time:.3f}s ({db_found}/{len(test_imdb_ids)} found)")
if avg_api_time > 0 and avg_db_time > 0:
speedup = avg_api_time / avg_db_time
print(f"Speedup: {speedup:.1f}x faster with database")
def test_import_date_performance():
"""Test import date retrieval performance"""
print("\n=== Testing Import Date Performance ===")
radarr_url = os.environ.get("RADARR_URL")
radarr_api_key = os.environ.get("RADARR_API_KEY")
if not radarr_url or not radarr_api_key:
print("❌ RADARR_URL and RADARR_API_KEY required for testing")
return
client = RadarrClient(radarr_url, radarr_api_key)
# Find a test movie
test_movie = client.movie_by_imdb("tt1596343") # Use database if available
if not test_movie:
print("❌ Test movie not found in library")
return
movie_id = test_movie.get('id')
movie_title = test_movie.get('title', 'Unknown')
print(f"Testing with: {movie_title} (ID: {movie_id})")
# Test API performance
print("\n--- API Import Date Performance ---")
start_time = time.time()
# Temporarily disable database client
original_db_client = client.db_client
client.db_client = None
api_date, api_source = client.get_movie_import_date(movie_id)
# Restore database client
client.db_client = original_db_client
api_time = time.time() - start_time
print(f"API Result: {api_date} ({api_source}) - {api_time:.3f}s")
# Test Database performance
if client.db_client:
print("\n--- Database Import Date Performance ---")
start_time = time.time()
db_date, db_source = client.get_movie_import_date(movie_id)
db_time = time.time() - start_time
print(f"Database Result: {db_date} ({db_source}) - {db_time:.3f}s")
if api_time > 0 and db_time > 0:
speedup = api_time / db_time
print(f"Import date speedup: {speedup:.1f}x faster with database")
else:
print("Database client not available for comparison")
def test_bulk_operations():
"""Test bulk import date operations"""
print("\n=== Testing Bulk Operations ===")
db_client = RadarrDbClient.from_env()
if not db_client:
print("❌ Database client required for bulk operations")
return
# Test bulk import dates
test_imdb_ids = ["tt1596343", "tt0468569", "tt0137523", "tt0109830", "tt0111161"]
start_time = time.time()
results = db_client.bulk_import_dates(test_imdb_ids)
elapsed = time.time() - start_time
print(f"Bulk operation completed in {elapsed:.3f}s")
print("Results:")
for imdb_id, (date_iso, source) in results.items():
print(f" {imdb_id}: {date_iso} ({source})")
if __name__ == "__main__":
print("NFOGuard Database Performance Test")
print("=" * 50)
# Test database connection
if not test_database_connection():
print("Exiting due to database connection failure")
exit(1)
# Test movie lookups
test_movie_lookup_performance()
# Test import dates
test_import_date_performance()
# Test bulk operations
test_bulk_operations()
print("\n✅ All tests completed!")
print("\nTo enable database access, configure these environment variables:")
print(" RADARR_DB_TYPE=postgresql")
print(" RADARR_DB_HOST=your_postgres_host")
print(" RADARR_DB_PORT=5432")
print(" RADARR_DB_NAME=radarr-main")
print(" RADARR_DB_USER=postgres")
print(" RADARR_DB_PASSWORD=your_password")
print("\nOr for SQLite:")
print(" RADARR_DB_TYPE=sqlite")
print(" RADARR_DB_PATH=/config/radarr.db")
webhook_server.py
-1977
View File
File diff suppressed because it is too large Load Diff