fix: web interface 422 errors

1. Enhanced Debugging: Will show exact episode dict structure and contents
  2. Safe Conversion: Try/catch around int(episode) with detailed error logging
  3. Format String Protection: All print statements now have fallback error handling
  4. Response Safety: Safe episode conversion for API response object

.env.template

@@ -93,6 +93,11 @@ UPDATE_MODE=always
 # File modification time behavior (update, leave_alone)
 MTIME_BEHAVIOR=update
 
+# Manual scan priority: use existing NFO dates for speed vs check APIs for accuracy
+# false (default) = Check external APIs first, use NFO as fallback (slower but accurate)
+# true = Use NFO dates immediately without API checks (faster but may use wrong dates)
+MANUAL_SCAN_PRIORITIZE_NFO=false
+
 # ===========================================
 # PERFORMANCE & BATCHING
 # ===========================================

Dockerfile

@@ -12,11 +12,12 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
     GIT_BRANCH=${GIT_BRANCH} \
     BUILD_SOURCE=${BUILD_SOURCE}
 
-# Install system dependencies including PostgreSQL client libraries
+# Install system dependencies including PostgreSQL client libraries and tini
 RUN apt-get update && apt-get install -y \
     curl \
     libpq-dev \
     gcc \
+    tini \
     && rm -rf /var/lib/apt/lists/*
 
 # Create app user and directory
@@ -51,7 +52,7 @@ RUN mkdir -p /app/emby-plugin && \
     echo '  echo "No Emby plugins directory found - skipping plugin deployment"' >> /app/deploy-plugin.sh && \
     echo '  echo "To enable plugin deployment, bind mount your Emby plugins directory to /emby-plugins"' >> /app/deploy-plugin.sh && \
     echo 'fi' >> /app/deploy-plugin.sh && \
-    echo 'exec python -u nfoguard.py' >> /app/deploy-plugin.sh && \
+    echo 'exec python -u main.py' >> /app/deploy-plugin.sh && \
     chmod +x /app/deploy-plugin.sh
 
 # Set ownership
@@ -67,5 +68,8 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
 # Expose port
 EXPOSE 8080
 
+# Use tini as init process to handle signals and zombie processes properly
+ENTRYPOINT ["tini", "--"]
+
 # Run the application with plugin deployment
 CMD ["/app/deploy-plugin.sh"]

README.md

@@ -26,14 +26,37 @@ NFOGuard automatically updates movie and TV show NFO files with proper release d
 
 ## ✨ Features
 
-- 🎬 **Movie & TV Support** - Works with both Radarr and Sonarr
+### **🎬 Core Media Management**
-- 📅 **Smart Date Handling** - Prioritizes digital, physical, and theatrical release dates
+- **Movie & TV Support** - Works with both Radarr and Sonarr
-- 🔄 **Webhook Integration** - Triggers automatically on import, upgrade, and rename
+- **Smart Date Handling** - Prioritizes digital, physical, and theatrical release dates
-- 🗄️ **Database Integration** - Direct PostgreSQL access for better performance
+- **Webhook Integration** - Triggers automatically on import, upgrade, and rename
-- 📝 **NFO Preservation** - Maintains existing metadata, adds fields cleanly at bottom
+- **NFO Preservation** - Maintains existing metadata, adds fields cleanly at bottom
-- 🔒 **Metadata Locking** - Prevents overwrites with lockdata tags
+- **Metadata Locking** - Prevents overwrites with lockdata tags
-- ⚡ **Batch Processing** - Efficient handling of multiple files
+
-- 🐳 **Docker Ready** - Easy deployment with Docker Compose
+### **⚡ Performance & Scalability**
+- **Async I/O Operations** - High-performance concurrent file processing
+- **Batch Processing** - Efficient handling of multiple files simultaneously
+- **Database Integration** - Direct PostgreSQL/SQLite access for better performance
+- **Smart Concurrency** - Controlled parallel processing with rate limiting
+
+### **🔧 Configuration & Validation**
+- **Comprehensive Config Validation** - Validates all settings before startup
+- **Runtime Health Checks** - Monitors system health and dependencies
+- **Path Validation** - Ensures media directories are accessible
+- **Database Connectivity Tests** - Validates database connections
+
+### **📊 Monitoring & Observability**
+- **Health Check APIs** - `/api/v1/health`, `/api/v1/health/ready`, `/api/v1/health/live`
+- **Prometheus Metrics** - `/api/v1/metrics` endpoint for monitoring integration
+- **Performance Monitoring** - Operation timing, error rates, and bottleneck detection
+- **Structured Logging** - JSON logs with correlation IDs for request tracing
+- **System Status APIs** - Real-time performance and error metrics
+
+### **🐳 Production Ready**
+- **Docker & Kubernetes** - Health checks for orchestration platforms
+- **Grafana Compatible** - Metrics endpoints for dashboard integration
+- **Configuration CLI** - Validation tools for troubleshooting
+- **Modular Architecture** - Clean separation of concerns for maintainability
 
 ## 🚀 Quick Start
 
@@ -125,9 +148,11 @@ image: sbcrumb/nfoguard:dev
 
 ### Specific Version  
 ```yaml
-image: sbcrumb/nfoguard:v1.5.5
+image: sbcrumb/nfoguard:v2.0.0  # Latest with monitoring & validation
 ```
 
+> **🚀 Version 2.0.0** includes major architecture improvements, async I/O performance enhancements, comprehensive monitoring, and configuration validation.
+
 ## 🔗 Webhook Setup
 
 Configure these webhook URLs in your applications:
@@ -162,13 +187,38 @@ curl -X POST "http://localhost:8080/bulk/update"
 
 ### API Endpoints
 
+#### **Core Operations**
 | Endpoint | Method | Purpose |
 |----------|--------|---------|
-| `/health` | GET | Health check |
 | `/webhook/radarr` | POST | Radarr webhook handler |
 | `/webhook/sonarr` | POST | Sonarr webhook handler |
 | `/manual/scan` | POST | Manual media scanning |
-| `/bulk/update` | POST | Bulk movie updates from Radarr DB |
+| `/bulk/update` | POST | Bulk update from Radarr database |
+
+#### **Health & Monitoring** 
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/health` | GET | Comprehensive health status |
+| `/api/v1/health/ready` | GET | Kubernetes readiness probe |
+| `/api/v1/health/live` | GET | Kubernetes liveness probe |
+| `/api/v1/status` | GET | Complete system status |
+| `/api/v1/status/brief` | GET | Quick system overview |
+
+#### **Metrics & Performance**
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/metrics` | GET | Prometheus metrics (text format) |
+| `/api/v1/metrics/json` | GET | Metrics in JSON format |
+| `/api/v1/metrics/processing` | GET | Processing-specific metrics |
+| `/api/v1/metrics/errors` | GET | Error metrics and recent failures |
+| `/api/v1/metrics/system` | GET | System resource metrics |
+
+#### **Configuration & Debugging**
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/debug/movie/{imdb_id}/priority` | GET | Debug movie priority logic |
+| `/debug/tmdb/{imdb_id}` | GET | Debug TMDB lookup |
+| `/ping` | GET | Simple connectivity test |
 
 ### Manual Scan Parameters
 
@@ -226,6 +276,79 @@ PATH_DEBUG=true
 curl http://localhost:8080/health
 ```
 
+## 📊 Monitoring & Observability
+
+NFOGuard provides comprehensive monitoring capabilities for production deployments:
+
+### **Health Checks**
+```bash
+# Basic health status
+curl http://localhost:8080/api/v1/health
+
+# Kubernetes readiness probe  
+curl http://localhost:8080/api/v1/health/ready
+
+# Kubernetes liveness probe
+curl http://localhost:8080/api/v1/health/live
+
+# Quick system overview
+curl http://localhost:8080/api/v1/status/brief
+```
+
+### **Metrics for Grafana/Prometheus**
+```bash
+# Prometheus metrics format
+curl http://localhost:8080/api/v1/metrics
+
+# JSON metrics for dashboards
+curl http://localhost:8080/api/v1/metrics/json
+
+# Processing performance metrics
+curl http://localhost:8080/api/v1/metrics/processing
+
+# Error rates and recent failures
+curl http://localhost:8080/api/v1/metrics/errors
+
+# System resource usage
+curl http://localhost:8080/api/v1/metrics/system
+```
+
+### **Configuration Validation**
+```bash
+# Validate configuration before deployment
+python config/validation_cli.py
+
+# Include runtime tests (database, APIs, file access)
+python config/validation_cli.py --runtime
+
+# JSON output for automation
+python config/validation_cli.py --runtime --json
+```
+
+### **Docker Health Checks**
+Add to your `docker-compose.yml`:
+```yaml
+healthcheck:
+  test: ["CMD", "curl", "-f", "http://localhost:8080/api/v1/health/live"]
+  interval: 30s
+  timeout: 10s
+  retries: 3
+  start_period: 40s
+```
+
+### **Structured Logging**
+NFOGuard uses structured JSON logging with correlation IDs for request tracing:
+```bash
+# Enable structured logging
+STRUCTURED_LOGGING=true
+
+# Log level configuration  
+LOG_LEVEL=INFO
+
+# Example log output with correlation ID
+{"timestamp": "2024-01-01T12:00:00Z", "level": "INFO", "correlation_id": "req_123", "message": "Webhook received", "context": {"webhook_type": "radarr", "media_type": "movie"}}
+```
+
 ### Common Issues
 
 1. **Permission Errors**: Ensure NFOGuard can write to mounted directories

VERSION

@@ -1 +1 @@
-2.0.0
+2.6.5

api/models.py

@@ -49,5 +49,62 @@ class TVSeasonRequest(BaseModel):
 class TVEpisodeRequest(BaseModel):
     """TV episode processing request model"""
     series_path: str
+    season: int
+    episode: int
+
+
+# Web interface models
+class MovieUpdateRequest(BaseModel):
+    """Request to update movie dateadded"""
+    dateadded: Optional[str]
+    source: str
+
+
+class EpisodeUpdateRequest(BaseModel):
+    """Request to update episode dateadded"""
+    dateadded: Optional[str]
+    source: str
+
+
+class BulkUpdateRequest(BaseModel):
+    """Request for bulk source updates"""
+    media_type: str  # "movies" or "episodes"
+    old_source: str
+    new_source: str
+
+
+class MovieResponse(BaseModel):
+    """Movie data response"""
+    imdb_id: str
+    title: str
+    path: str
+    released: Optional[str]
+    dateadded: Optional[str]
+    source: Optional[str]
+    has_video_file: bool
+    last_updated: str
+
+
+class SeriesResponse(BaseModel):
+    """TV series data response"""
+    imdb_id: str
+    title: str
+    path: str
+    last_updated: str
+    total_episodes: int
+    episodes_with_dates: int
+    episodes_with_video: int
+
+
+class EpisodeResponse(BaseModel):
+    """TV episode data response"""
+    season: int
+    episode: int
+    aired: Optional[str]
+    dateadded: Optional[str]
+    source: Optional[str]
+    has_video_file: bool
+    last_updated: str
+    series_path: str
     season_name: str
     episode_name: str

api/monitoring_routes.py

@@ -0,0 +1,351 @@
+"""
+Monitoring API Routes for NFOGuard
+Provides health checks, metrics, and system status endpoints
+"""
+from fastapi import APIRouter, Response, HTTPException
+from typing import Dict, Any, Optional
+import time
+
+from monitoring.health import health_checker, HealthStatus
+from monitoring.metrics import metrics
+try:
+    from config.validator import get_configuration_summary
+except ImportError:
+    def get_configuration_summary():
+        return {"status": "Configuration validator not available"}
+
+
+router = APIRouter(prefix="/api/v1", tags=["monitoring"])
+
+
+@router.get("/health")
+async def get_health_status():
+    """
+    Get comprehensive health status
+    
+    Returns detailed health information including:
+    - Overall system health
+    - Individual component health checks
+    - Performance metrics
+    - Error status
+    """
+    try:
+        health_status = await health_checker.get_full_health_status()
+        
+        # Set appropriate HTTP status code
+        if health_status.status == HealthStatus.HEALTHY:
+            status_code = 200
+        elif health_status.status == HealthStatus.DEGRADED:
+            status_code = 200  # Still operational
+        else:  # UNHEALTHY
+            status_code = 503  # Service unavailable
+        
+        return Response(
+            content=health_status.to_dict(),
+            status_code=status_code,
+            media_type="application/json"
+        )
+        
+    except Exception as e:
+        # Return unhealthy status if health check itself fails
+        return Response(
+            content={
+                "status": "unhealthy",
+                "message": f"Health check failed: {e}",
+                "timestamp": time.time()
+            },
+            status_code=500,
+            media_type="application/json"
+        )
+
+
+@router.get("/health/ready")
+async def get_readiness_status():
+    """
+    Kubernetes readiness probe endpoint
+    
+    Returns 200 if service is ready to accept traffic
+    Returns 503 if service is not ready
+    """
+    try:
+        readiness = await health_checker.get_readiness_status()
+        
+        status_code = 200 if readiness["ready"] else 503
+        
+        return Response(
+            content=readiness,
+            status_code=status_code,
+            media_type="application/json"
+        )
+        
+    except Exception as e:
+        return Response(
+            content={
+                "ready": False,
+                "message": f"Readiness check failed: {e}",
+                "timestamp": time.time()
+            },
+            status_code=503,
+            media_type="application/json"
+        )
+
+
+@router.get("/health/live")
+async def get_liveness_status():
+    """
+    Kubernetes liveness probe endpoint
+    
+    Returns 200 if service is alive and responsive
+    Returns 500 if service should be restarted
+    """
+    try:
+        liveness = await health_checker.get_liveness_status()
+        
+        status_code = 200 if liveness["alive"] else 500
+        
+        return Response(
+            content=liveness,
+            status_code=status_code,
+            media_type="application/json"
+        )
+        
+    except Exception as e:
+        return Response(
+            content={
+                "alive": False,
+                "message": f"Liveness check failed: {e}",
+                "timestamp": time.time()
+            },
+            status_code=500,
+            media_type="application/json"
+        )
+
+
+@router.get("/metrics")
+async def get_prometheus_metrics():
+    """
+    Prometheus-compatible metrics endpoint
+    
+    Returns metrics in Prometheus text format for scraping
+    """
+    try:
+        prometheus_metrics = metrics.get_prometheus_metrics()
+        
+        return Response(
+            content=prometheus_metrics,
+            media_type="text/plain; charset=utf-8"
+        )
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to generate metrics: {e}"
+        )
+
+
+@router.get("/metrics/json")
+async def get_metrics_json():
+    """
+    Get all metrics in JSON format
+    
+    Returns structured metrics data including:
+    - System metrics (CPU, memory, disk)
+    - Processing metrics (rates, durations)
+    - Error metrics (counts, recent errors)
+    """
+    try:
+        all_metrics = metrics.get_all_metrics()
+        return all_metrics
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to get metrics: {e}"
+        )
+
+
+@router.get("/status")
+async def get_system_status():
+    """
+    Get comprehensive system status
+    
+    Returns detailed system information including:
+    - Health status
+    - Configuration summary
+    - Performance metrics
+    - Recent activity
+    """
+    try:
+        # Get health status
+        health_status = await health_checker.get_full_health_status()
+        
+        # Get metrics
+        all_metrics = metrics.get_all_metrics()
+        
+        # Get configuration summary
+        config_summary = get_configuration_summary()
+        
+        # Combine into comprehensive status
+        status_response = {
+            "overall_status": health_status.status.value,
+            "timestamp": time.time(),
+            "uptime_seconds": health_status.uptime_seconds,
+            "version": health_status.version,
+            "health": health_status.to_dict(),
+            "metrics": all_metrics,
+            "configuration": config_summary,
+            "summary": {
+                "service_healthy": health_status.status in [HealthStatus.HEALTHY, HealthStatus.DEGRADED],
+                "total_webhooks_processed": all_metrics["processing"]["total_webhooks"],
+                "total_nfo_files_created": all_metrics["processing"]["total_nfo_created"],
+                "total_errors": all_metrics["processing"]["total_errors"],
+                "current_processing_rate": all_metrics["processing"]["webhooks_received_per_minute"],
+                "average_processing_time": all_metrics["processing"]["average_processing_time_seconds"]
+            }
+        }
+        
+        return status_response
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to get system status: {e}"
+        )
+
+
+@router.get("/status/brief")
+async def get_brief_status():
+    """
+    Get brief system status for quick monitoring
+    
+    Returns essential status information without detailed metrics
+    """
+    try:
+        # Get basic health
+        liveness = await health_checker.get_liveness_status()
+        readiness = await health_checker.get_readiness_status()
+        
+        # Get basic metrics
+        processing_metrics = metrics.get_processing_metrics()
+        system_metrics = metrics.get_system_metrics()
+        
+        return {
+            "status": "healthy" if liveness["alive"] and readiness["ready"] else "unhealthy",
+            "alive": liveness["alive"],
+            "ready": readiness["ready"],
+            "uptime_seconds": liveness["uptime_seconds"],
+            "webhooks_per_minute": processing_metrics["webhooks_received_per_minute"],
+            "active_operations": processing_metrics["active_operations"],
+            "total_errors": processing_metrics["total_errors"],
+            "cpu_percent": system_metrics.get("cpu_percent", 0),
+            "memory_percent": system_metrics.get("memory_percent", 0),
+            "timestamp": time.time()
+        }
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to get brief status: {e}"
+        )
+
+
+@router.get("/metrics/processing")
+async def get_processing_metrics():
+    """
+    Get processing-specific metrics
+    
+    Returns metrics focused on NFO processing performance
+    """
+    try:
+        processing_metrics = metrics.get_processing_metrics()
+        return processing_metrics
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to get processing metrics: {e}"
+        )
+
+
+@router.get("/metrics/errors")
+async def get_error_metrics():
+    """
+    Get error-specific metrics
+    
+    Returns error counts, types, and recent error information
+    """
+    try:
+        error_metrics = metrics.get_error_metrics()
+        return error_metrics
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to get error metrics: {e}"
+        )
+
+
+@router.get("/metrics/system")
+async def get_system_metrics():
+    """
+    Get system resource metrics
+    
+    Returns CPU, memory, disk, and process information
+    """
+    try:
+        system_metrics = metrics.get_system_metrics()
+        return system_metrics
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to get system metrics: {e}"
+        )
+
+
+@router.post("/metrics/reset")
+async def reset_metrics(metric_types: Optional[str] = None):
+    """
+    Reset specific metric types
+    
+    Parameters:
+    - metric_types: Comma-separated list of metric types to reset
+                   (counters, histograms, errors, timeseries)
+                   If not specified, resets all metrics
+    """
+    try:
+        reset_types = None
+        if metric_types:
+            reset_types = [t.strip() for t in metric_types.split(",")]
+        
+        metrics.reset_metrics(reset_types)
+        
+        return {
+            "message": "Metrics reset successfully",
+            "reset_types": reset_types or "all",
+            "timestamp": time.time()
+        }
+        
+    except Exception as e:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Failed to reset metrics: {e}"
+        )
+
+
+# Legacy endpoints for backwards compatibility
+@router.get("/health-check")
+async def legacy_health_check():
+    """Legacy health check endpoint (redirects to /health)"""
+    return await get_brief_status()
+
+
+@router.get("/ping")
+async def ping():
+    """Simple ping endpoint for basic connectivity testing"""
+    return {
+        "message": "pong",
+        "timestamp": time.time(),
+        "service": "nfoguard",
+        "version": "2.0.0"
+    }

api/routes.py

@@ -3,13 +3,23 @@ FastAPI routes for NFOGuard - extracted from main nfoguard.py for modular archit
 """
 import os
 import json
+import requests
+import asyncio
 from pathlib import Path
 from datetime import datetime, timezone
 from fastapi import HTTPException, BackgroundTasks, Request
 from typing import Optional
 
 # Import models
-from api.models import SonarrWebhook, RadarrWebhook, HealthResponse, TVSeasonRequest, TVEpisodeRequest
+from api.models import (
+    SonarrWebhook, RadarrWebhook, HealthResponse, TVSeasonRequest, TVEpisodeRequest,
+    MovieUpdateRequest, EpisodeUpdateRequest, BulkUpdateRequest
+)
+from api.web_routes import (
+    get_movies_list, get_tv_series_list, get_series_episodes, get_series_sources, get_missing_dates_report,
+    get_dashboard_stats, update_movie_date, update_episode_date, bulk_update_source,
+    get_movie_date_options, get_episode_date_options, debug_series_date_distribution
+)
 
 
 # ---------------------------
@@ -72,14 +82,144 @@ async def sonarr_webhook(request: Request, background_tasks: BackgroundTasks, de
             print(f"ERROR: Could not find series directory: {series_title} ({imdb_id})")
             return {"status": "error", "reason": "Series directory not found"}
         
+        # Extract episode data for targeted processing
+        episodes_data = webhook.episodes or []
+        print(f"DEBUG: Initial episodes_data from webhook.episodes: {len(episodes_data)} episodes")
+        
+        # For all webhook events, if no episodes in webhook.episodes, try to extract from episodeFile
+        # This ensures targeted processing for single episode operations (Download, Rename, Upgrade)
+        print(f"DEBUG: webhook.episodeFile present: {webhook.episodeFile is not None}")
+        if webhook.episodeFile:
+            print(f"DEBUG: episodeFile content: {webhook.episodeFile}")
+        
+        if not episodes_data and webhook.episodeFile:
+            episode_file = webhook.episodeFile
+            # Extract season and episode from episodeFile if available
+            season_num = episode_file.get("seasonNumber")
+            episode_num = episode_file.get("episodeNumber") 
+            print(f"DEBUG: episodeFile seasonNumber: {season_num}, episodeNumber: {episode_num}")
+            if season_num and episode_num:
+                # Create episode data structure that matches what process_webhook_episodes expects
+                episodes_data = [{
+                    "seasonNumber": season_num,
+                    "episodeNumber": episode_num,
+                    "id": episode_file.get("id"),
+                    "title": episode_file.get("title")
+                }]
+                print(f"INFO: Extracted episode info from episodeFile for {webhook.eventType}: S{season_num:02d}E{episode_num:02d}")
+            else:
+                print(f"DEBUG: Missing season/episode numbers in episodeFile for {webhook.eventType}")
+        
+        # Special handling for Rename events - Sonarr doesn't include episodeFile for renames
+        # Try to find recently renamed episodes using Sonarr history API
+        if not episodes_data and webhook.eventType == "Rename":
+            print(f"DEBUG: Attempting to find recently renamed episode for series {imdb_id}")
+            try:
+                # Get series info from Sonarr to find series ID
+                series_lookup_url = f"{config.sonarr_url}/api/v3/series/lookup?term=imdbid:{imdb_id}"
+                print(f"DEBUG: Sonarr lookup for rename: {series_lookup_url}")
+                
+                response = requests.get(series_lookup_url, headers={"X-Api-Key": os.environ.get("SONARR_API_KEY", "")}, timeout=10)
+                if response.status_code == 200:
+                    series_results = response.json()
+                    if series_results:
+                        series_id = series_results[0].get("id")
+                        print(f"DEBUG: Found series ID {series_id} for rename lookup")
+                        
+                        # Get recent history for the series and filter for rename events
+                        from datetime import datetime, timedelta
+                        since_date = (datetime.utcnow() - timedelta(hours=1)).strftime('%Y-%m-%dT%H:%M:%SZ')
+                        history_url = f"{config.sonarr_url}/api/v3/history?seriesId={series_id}&sortKey=date&sortDir=desc&page=1&pageSize=50"
+                        print(f"DEBUG: Checking recent rename history: {history_url}")
+                        
+                        history_response = requests.get(history_url, headers={"X-Api-Key": os.environ.get("SONARR_API_KEY", "")}, timeout=10)
+                        if history_response.status_code == 200:
+                            history_data = history_response.json()
+                            all_records = history_data.get("records", [])
+                            print(f"DEBUG: Got {len(all_records)} total history records")
+                            
+                            # Filter for recent rename events
+                            since_timestamp = datetime.utcnow() - timedelta(hours=1)
+                            recent_renames = []
+                            
+                            for record in all_records:
+                                event_type = record.get("eventType", "")
+                                date_str = record.get("date", "")
+                                
+                                if event_type == "episodeFileRenamed" and date_str:
+                                    try:
+                                        event_time = datetime.strptime(date_str.replace('Z', '+00:00'), '%Y-%m-%dT%H:%M:%S.%f%z')
+                                        event_time_utc = event_time.utctimetuple()
+                                        if datetime(*event_time_utc[:6]) > since_timestamp:
+                                            recent_renames.append(record)
+                                    except:
+                                        # If datetime parsing fails, include it anyway
+                                        recent_renames.append(record)
+                            
+                            print(f"DEBUG: Found {len(recent_renames)} recent rename events")
+                            
+                            if recent_renames:
+                                # Take the most recent rename event
+                                latest_rename = recent_renames[0]
+                                print(f"DEBUG: Processing latest rename event")
+                                
+                                # Extract episodeId directly from the rename event
+                                episode_id = latest_rename.get("episodeId")
+                                print(f"DEBUG: Found episodeId {episode_id} in rename event")
+                                
+                                if episode_id:
+                                    # Fetch episode details using the episodeId
+                                    episode_detail_url = f"{config.sonarr_url}/api/v3/episode/{episode_id}"
+                                    episode_response = requests.get(episode_detail_url, headers={"X-Api-Key": os.environ.get("SONARR_API_KEY", "")}, timeout=10)
+                                    
+                                    if episode_response.status_code == 200:
+                                        episode_detail = episode_response.json()
+                                        season_num = episode_detail.get("seasonNumber")
+                                        episode_num = episode_detail.get("episodeNumber")
+                                        episode_title = episode_detail.get("title")
+                                        
+                                        print(f"DEBUG: Episode details - Season: {season_num}, Episode: {episode_num}, Title: {episode_title}")
+                                        
+                                        if season_num is not None and episode_num is not None:
+                                            episodes_data = [{
+                                                "seasonNumber": season_num,
+                                                "episodeNumber": episode_num,
+                                                "id": episode_id,
+                                                "title": episode_title
+                                            }]
+                                            print(f"INFO: Successfully identified renamed episode: S{season_num:02d}E{episode_num:02d} - {episode_title}")
+                                        else:
+                                            print(f"DEBUG: Episode details missing season/episode numbers")
+                                    else:
+                                        print(f"DEBUG: Failed to fetch episode details: {episode_response.status_code}")
+                                else:
+                                    print(f"DEBUG: No episodeId found in rename event")
+                            else:
+                                print(f"DEBUG: No recent rename events found in last hour")
+                        else:
+                            print(f"DEBUG: Failed to get rename history: {history_response.status_code}")
+                    else:
+                        print(f"DEBUG: No series found for IMDb {imdb_id}")
+                else:
+                    print(f"DEBUG: Series lookup failed: {response.status_code}")
+            except Exception as e:
+                print(f"DEBUG: Error finding renamed episode: {e}")
+                # Continue with series processing as fallback
+        
+        # Force targeted mode for single-episode webhooks to prevent full series processing
+        processing_mode = config.tv_webhook_processing_mode
+        if episodes_data and len(episodes_data) <= 3:  # Single episode or small batch
+            processing_mode = "targeted"
+            print(f"INFO: Forcing targeted mode for {len(episodes_data)} episode(s)")
+        
         # Add to batch queue with TV-prefixed key to avoid movie conflicts
         tv_batch_key = f"tv:{imdb_id}"
         webhook_dict = {
             'path': str(series_path),
             'series_info': series_info,
             'event_type': webhook.eventType,
-            'episodes': webhook.episodes or [],  # Include episode data for targeted processing
+            'episodes': episodes_data,  # Include enhanced episode data for targeted processing
-            'processing_mode': config.tv_webhook_processing_mode
+            'processing_mode': processing_mode  # Use forced targeted mode when appropriate
         }
         batcher.add_webhook(tv_batch_key, webhook_dict, 'tv')
         
@@ -390,8 +530,8 @@ async def debug_movie_history(imdb_id: str, dependencies: dict):
         }
 
 
-async def manual_scan(background_tasks: BackgroundTasks, path: Optional[str] = None, scan_type: str = "both", dependencies: dict = None):
+async def manual_scan(background_tasks: BackgroundTasks, path: Optional[str] = None, scan_type: str = "both", scan_mode: str = "smart", dependencies: dict = None):
-    """Manual scan endpoint"""
+    """Manual scan endpoint with smart optimization modes"""
     config = dependencies["config"]
     nfo_manager = dependencies["nfo_manager"]
     tv_processor = dependencies["tv_processor"]
@@ -400,7 +540,43 @@ async def manual_scan(background_tasks: BackgroundTasks, path: Optional[str] = N
     if scan_type not in ["both", "tv", "movies"]:
         raise HTTPException(status_code=400, detail="scan_type must be 'both', 'tv', or 'movies'")
     
+    if scan_mode not in ["smart", "full", "incomplete"]:
+        raise HTTPException(status_code=400, detail="scan_mode must be 'smart', 'full', or 'incomplete'")
+    
     async def run_scan():
+        from datetime import datetime, timezone
+        import time
+        import os
+        start_time = datetime.now()
+        
+        # Handle timezone display - check if TZ is set in container
+        try:
+            tz_name = os.environ.get('TZ')
+            if tz_name:
+                # TZ is set, so datetime.now() already returns local time
+                local_start = start_time
+                tz_display = f" ({tz_name})"
+            else:
+                # No TZ set, assume container is UTC and convert to Eastern
+                import zoneinfo
+                local_tz = zoneinfo.ZoneInfo("America/New_York")
+                local_start = start_time.replace(tzinfo=timezone.utc).astimezone(local_tz)
+                tz_display = " (EDT/EST)"
+        except:
+            # Ultimate fallback - just show time as-is with note
+            local_start = start_time
+            tz_display = " (container time)"
+            
+        print(f"🚀 MANUAL SCAN STARTED: {scan_type} scan (mode: {scan_mode}) initiated at {local_start.strftime('%Y-%m-%d %H:%M:%S')}{tz_display}")
+        
+        # Initialize counters for scan statistics
+        tv_series_total = 0
+        tv_series_skipped = 0
+        tv_series_processed = 0
+        movie_total = 0
+        movie_skipped = 0
+        movie_processed = 0
+        
         paths_to_scan = []
         if path:
             paths_to_scan = [Path(path)]
@@ -439,37 +615,146 @@ async def manual_scan(background_tasks: BackgroundTasks, path: Optional[str] = N
                     # Check if this path itself is a series (has IMDb ID in the directory name)
                     if nfo_manager.parse_imdb_from_path(scan_path):
                         try:
-                            tv_processor.process_series(scan_path)
+                            # Determine force_scan based on scan mode
+                            force_scan = (scan_mode == "full")
+                            result = tv_processor.process_series(scan_path, force_scan=force_scan)
+                            tv_series_total += 1
+                            if result == "skipped":
+                                tv_series_skipped += 1
+                            elif result == "processed":
+                                tv_series_processed += 1
                         except Exception as e:
                             print(f"ERROR: Failed processing TV series {scan_path}: {e}")
+                            tv_series_total += 1
                     else:
                         # Full series processing - scan subdirectories
                         import re
+                        tv_count = 0
                         for item in scan_path.iterdir():
+                            # Check for shutdown signal at start of each item
+                            shutdown_event = dependencies.get("shutdown_event")
+                            if shutdown_event and shutdown_event.is_set():
+                                print("INFO: ⚠️ SHUTDOWN SIGNAL RECEIVED - Stopping scan gracefully")
+                                return
+                                
                             if (item.is_dir() and 
                                 not item.name.lower().startswith('season') and
                                 not re.match(r'^season\s+\d+$', item.name, re.IGNORECASE) and
                                 nfo_manager.parse_imdb_from_path(item)):
+                                tv_count += 1
                                 try:
-                                    tv_processor.process_series(item)
+                                    # Determine force_scan based on scan mode
+                                    force_scan = (scan_mode == "full")
+                                    result = tv_processor.process_series(item, force_scan=force_scan)
+                                    tv_series_total += 1
+                                    if result == "skipped":
+                                        tv_series_skipped += 1
+                                    elif result == "processed":
+                                        tv_series_processed += 1
                                 except Exception as e:
                                     print(f"ERROR: Failed processing TV series {item}: {e}")
+                                    tv_series_total += 1
+                                
+                                # Yield control every TV series to allow other requests  
+                                if tv_count % 1 == 0:
+                                    await asyncio.sleep(0.2)  # 200ms yield to process other requests
+                                    print(f"INFO: Processed {tv_count} TV series, yielding to other requests...")
+                                    
+                                    # Check for shutdown signal
+                                    shutdown_event = dependencies.get("shutdown_event")
+                                    if shutdown_event and shutdown_event.is_set():
+                                        print("INFO: ⚠️ SHUTDOWN SIGNAL RECEIVED - Stopping scan gracefully")
+                                        return
             
             if scan_type in ["both", "movies"] and scan_path in config.movie_paths:
                 print(f"INFO: Scanning movies in: {scan_path}")
                 movie_count = 0
                 for item in scan_path.iterdir():
+                    # Check for shutdown signal at start of each movie
+                    shutdown_event = dependencies.get("shutdown_event")
+                    if shutdown_event and shutdown_event.is_set():
+                        print("INFO: ⚠️ SHUTDOWN SIGNAL RECEIVED - Stopping scan gracefully")
+                        return
+                        
                     if item.is_dir() and nfo_manager.find_movie_imdb_id(item):
                         movie_count += 1
                         print(f"INFO: Processing movie: {item.name}")
                         try:
-                            movie_processor.process_movie(item)  
+                            # Determine force_scan based on scan mode
+                            force_scan = (scan_mode == "full")
+                            result = movie_processor.process_movie(item, webhook_mode=False, force_scan=force_scan)
+                            movie_total += 1
+                            if result == "skipped":
+                                movie_skipped += 1
+                            elif result == "processed":
+                                movie_processed += 1
                         except Exception as e:
                             print(f"ERROR: Failed processing movie {item}: {e}")
+                            movie_total += 1
+                        
+                        # Yield control every 2 movies to allow other requests (webhooks, web interface)
+                        if movie_count % 2 == 0:
+                            await asyncio.sleep(0.2)  # 200ms yield to process other requests
+                            print(f"INFO: Processed {movie_count} movies, yielding to other requests...")
+                            
+                            # Check for shutdown signal
+                            shutdown_event = dependencies.get("shutdown_event")
+                            if shutdown_event and shutdown_event.is_set():
+                                print("INFO: ⚠️ SHUTDOWN SIGNAL RECEIVED - Stopping scan gracefully")
+                                return
+                        
                 print(f"INFO: Completed movie scan: {movie_count} movies processed in {scan_path}")
         
+        # Log scan completion with duration
+        end_time = datetime.now()
+        duration = end_time - start_time
+        duration_str = str(duration).split('.')[0]  # Remove microseconds
+        
+        # Use same timezone logic as start
+        try:
+            tz_name = os.environ.get('TZ')
+            if tz_name:
+                # TZ is set, so datetime.now() already returns local time
+                local_end = end_time
+                tz_display = f" ({tz_name})"
+            else:
+                # No TZ set, assume container is UTC and convert to Eastern
+                import zoneinfo
+                local_tz = zoneinfo.ZoneInfo("America/New_York")
+                local_end = end_time.replace(tzinfo=timezone.utc).astimezone(local_tz)
+                tz_display = " (EDT/EST)"
+        except:
+            local_end = end_time
+            tz_display = " (container time)"
+            
+        print(f"✅ MANUAL SCAN COMPLETED: {scan_type} scan (mode: {scan_mode}) finished at {local_end.strftime('%Y-%m-%d %H:%M:%S')}{tz_display}")
+        print(f"⏱️ MANUAL SCAN DURATION: {duration_str} (total time: {duration.total_seconds():.1f} seconds)")
+        
+        # Print optimization statistics for TV scans
+        if scan_type in ["both", "tv"] and tv_series_total > 0:
+            print(f"📊 TV SCAN OPTIMIZATION: Total: {tv_series_total}, Processed: {tv_series_processed}, Skipped: {tv_series_skipped}")
+            if tv_series_skipped > 0:
+                skip_percentage = (tv_series_skipped / tv_series_total) * 100
+                print(f"⚡ TV PERFORMANCE BOOST: {tv_series_skipped}/{tv_series_total} series skipped ({skip_percentage:.1f}% time saved!)")
+        
+        # Print optimization statistics for movie scans
+        if scan_type in ["both", "movies"] and movie_total > 0:
+            print(f"📊 MOVIE SCAN OPTIMIZATION: Total: {movie_total}, Processed: {movie_processed}, Skipped: {movie_skipped}")
+            if movie_skipped > 0:
+                skip_percentage = (movie_skipped / movie_total) * 100
+                print(f"⚡ MOVIE PERFORMANCE BOOST: {movie_skipped}/{movie_total} movies skipped ({skip_percentage:.1f}% time saved!)")
+        
+        # Print combined optimization statistics for "both" scans
+        if scan_type == "both" and (tv_series_total > 0 or movie_total > 0):
+            total_items = tv_series_total + movie_total
+            total_skipped = tv_series_skipped + movie_skipped
+            total_processed = tv_series_processed + movie_processed
+            if total_skipped > 0:
+                overall_skip_percentage = (total_skipped / total_items) * 100
+                print(f"🎯 OVERALL OPTIMIZATION: {total_skipped}/{total_items} items skipped ({overall_skip_percentage:.1f}% total time saved!)")
+    
     background_tasks.add_task(run_scan)
-    return {"status": "started", "message": f"Manual {scan_type} scan started"}
+    return {"status": "started", "message": f"Manual {scan_type} scan started (mode: {scan_mode})"}
 
 
 async def scan_tv_season(background_tasks: BackgroundTasks, request: TVSeasonRequest, dependencies: dict):
@@ -787,6 +1072,88 @@ async def debug_tmdb_lookup(imdb_id: str, dependencies: dict):
         return {"error": str(e), "imdb_id": imdb_id, "traceback": str(e)}
 
 
+# ---------------------------
+# Database Cleanup Endpoints
+# ---------------------------
+
+async def delete_episode(imdb_id: str, season: int, episode: int, dependencies: dict):
+    """Delete a specific episode from the database"""
+    db = dependencies["db"]
+    
+    try:
+        deleted = db.delete_episode(imdb_id, season, episode)
+        
+        if deleted:
+            return {
+                "success": True,
+                "message": f"Deleted episode S{season:02d}E{episode:02d} from series {imdb_id}",
+                "imdb_id": imdb_id,
+                "season": season,
+                "episode": episode
+            }
+        else:
+            return {
+                "success": False,
+                "message": f"Episode S{season:02d}E{episode:02d} not found in series {imdb_id}",
+                "imdb_id": imdb_id,
+                "season": season,
+                "episode": episode
+            }
+            
+    except Exception as e:
+        return {
+            "success": False,
+            "error": str(e),
+            "imdb_id": imdb_id,
+            "season": season,
+            "episode": episode
+        }
+
+
+async def delete_series_episodes(imdb_id: str, dependencies: dict):
+    """Delete all episodes for a series from the database"""
+    db = dependencies["db"]
+    
+    try:
+        deleted_count = db.delete_series_episodes(imdb_id)
+        
+        return {
+            "success": True,
+            "message": f"Deleted {deleted_count} episodes from series {imdb_id}",
+            "imdb_id": imdb_id,
+            "deleted_count": deleted_count
+        }
+            
+    except Exception as e:
+        return {
+            "success": False,
+            "error": str(e),
+            "imdb_id": imdb_id
+        }
+
+
+async def cleanup_orphaned_episodes(dependencies: dict):
+    """Find and delete episodes that don't have corresponding video files"""
+    db = dependencies["db"]
+    
+    try:
+        deleted_episodes = db.delete_orphaned_episodes()
+        
+        return {
+            "success": True,
+            "message": f"Cleaned up {len(deleted_episodes)} orphaned episodes",
+            "deleted_count": len(deleted_episodes),
+            "deleted_episodes": deleted_episodes
+        }
+            
+    except Exception as e:
+        return {
+            "success": False,
+            "error": str(e),
+            "message": "Failed to cleanup orphaned episodes"
+        }
+
+
 # ---------------------------
 # Route Registration
 # ---------------------------
@@ -837,9 +1204,21 @@ def register_routes(app, dependencies: dict):
     async def _debug_movie_history(imdb_id: str):
         return await debug_movie_history(imdb_id, dependencies)
 
+    @app.delete("/database/episode/{imdb_id}/{season}/{episode}")
+    async def _delete_episode(imdb_id: str, season: int, episode: int):
+        return await delete_episode(imdb_id, season, episode, dependencies)
+
+    @app.delete("/database/series/{imdb_id}/episodes")
+    async def _delete_series_episodes(imdb_id: str):
+        return await delete_series_episodes(imdb_id, dependencies)
+
+    @app.post("/database/cleanup/orphaned-episodes")
+    async def _cleanup_orphaned_episodes():
+        return await cleanup_orphaned_episodes(dependencies)
+
     @app.post("/manual/scan")
-    async def _manual_scan(background_tasks: BackgroundTasks, path: Optional[str] = None, scan_type: str = "both"):
+    async def _manual_scan(background_tasks: BackgroundTasks, path: Optional[str] = None, scan_type: str = "both", scan_mode: str = "smart"):
-        return await manual_scan(background_tasks, path, scan_type, dependencies)
+        return await manual_scan(background_tasks, path, scan_type, scan_mode, dependencies)
 
     @app.post("/tv/scan-season")
     async def _scan_tv_season(background_tasks: BackgroundTasks, request: TVSeasonRequest):
@@ -868,3 +1247,105 @@ def register_routes(app, dependencies: dict):
     @app.get("/debug/tmdb/{imdb_id}")
     async def _debug_tmdb_lookup(imdb_id: str):
         return await debug_tmdb_lookup(imdb_id, dependencies)
+
+    # Include monitoring routes
+    from api.monitoring_routes import router as monitoring_router
+    app.include_router(monitoring_router)
+    
+    # ---------------------------
+    # Web Interface API Routes
+    # ---------------------------
+    
+    @app.get("/api/dashboard")
+    async def _dashboard_stats():
+        """Get dashboard statistics"""
+        return await get_dashboard_stats(dependencies)
+    
+    @app.get("/api/movies")
+    async def _movies_list(skip: int = 0, limit: int = 100, has_date: Optional[bool] = None, 
+                          source_filter: Optional[str] = None, search: Optional[str] = None,
+                          imdb_search: Optional[str] = None):
+        """Get paginated movies list with filtering"""
+        return await get_movies_list(dependencies, skip, limit, has_date, source_filter, search, imdb_search)
+    
+    @app.get("/api/series")
+    async def _series_list(skip: int = 0, limit: int = 50, search: Optional[str] = None, 
+                          imdb_search: Optional[str] = None, date_filter: Optional[str] = None, 
+                          source_filter: Optional[str] = None):
+        """Get paginated TV series list with filtering"""
+        return await get_tv_series_list(dependencies, skip, limit, search, imdb_search, date_filter, source_filter)
+    
+    @app.get("/api/series/{imdb_id}/episodes")
+    async def _series_episodes(imdb_id: str):
+        """Get episodes for a specific series"""
+        return await get_series_episodes(dependencies, imdb_id)
+    
+    @app.get("/api/series/sources")
+    async def _series_sources():
+        """Get list of available episode sources for filtering"""
+        return await get_series_sources(dependencies)
+    
+    @app.get("/api/debug/series-date-distribution")
+    async def _debug_series_dates():
+        """Debug endpoint showing TV series date distribution"""
+        return await debug_series_date_distribution(dependencies)
+    
+    @app.get("/api/reports/missing-dates")
+    async def _missing_dates_report():
+        """Get report of content missing dateadded"""
+        return await get_missing_dates_report(dependencies)
+    
+    @app.put("/api/movies/{imdb_id}")
+    async def _update_movie(imdb_id: str, request: MovieUpdateRequest):
+        """Update movie dateadded"""
+        return await update_movie_date(dependencies, imdb_id, request.dateadded, request.source)
+    
+    @app.put("/api/episodes/{imdb_id}/{season}/{episode}")
+    async def _update_episode(imdb_id: str, season: int, episode: int, request: EpisodeUpdateRequest):
+        """Update episode dateadded"""
+        return await update_episode_date(dependencies, imdb_id, season, episode, request.dateadded, request.source)
+    
+    @app.post("/api/bulk/update-source")
+    async def _bulk_update_source(request: BulkUpdateRequest):
+        """Bulk update source for movies or episodes"""
+        return await bulk_update_source(dependencies, request.media_type, request.old_source, request.new_source)
+    
+    @app.get("/api/movies/{imdb_id}/date-options")
+    async def _movie_date_options(imdb_id: str):
+        """Get available date options for a movie"""
+        return await get_movie_date_options(dependencies, imdb_id)
+    
+    @app.get("/api/episodes/{imdb_id}/{season}/{episode}/date-options")
+    async def _episode_date_options(imdb_id: str, season: int, episode: int):
+        """Get available date options for an episode"""
+        return await get_episode_date_options(dependencies, imdb_id, season, episode)
+    
+    @app.get("/api/debug/movie/{imdb_id}/raw")
+    async def _debug_movie_raw(imdb_id: str):
+        """Debug endpoint to see raw movie database data"""
+        db = dependencies["db"]
+        movie = db.get_movie_dates(imdb_id)
+        if not movie:
+            raise HTTPException(status_code=404, detail="Movie not found")
+        return {"raw_data": dict(movie), "imdb_id": imdb_id}
+    # ---------------------------
+    # Static Web Interface
+    # ---------------------------
+    
+    from fastapi.staticfiles import StaticFiles
+    from fastapi.responses import FileResponse
+    import os
+    
+    # Serve static files for web interface
+    static_dir = os.path.join(os.path.dirname(__file__), "..", "static")
+    if os.path.exists(static_dir):
+        app.mount("/static", StaticFiles(directory=static_dir), name="static")
+    
+    @app.get("/")
+    async def _serve_index():
+        """Serve web interface index page"""
+        index_path = os.path.join(static_dir, "index.html")
+        if os.path.exists(index_path):
+            return FileResponse(index_path)
+        else:
+            return {"message": "NFOGuard Web Interface - API endpoints available at /api/", "api_docs": "/docs"}

clients/external_clients.py

@@ -660,6 +660,36 @@ class ExternalClientManager:
         
         return self.tvdb.imdb_to_tvdb_series_id(imdb_id)
     
+    def get_episode_air_date(self, imdb_id: str, season: int, episode: int) -> Optional[str]:
+        """Get episode air date from external sources"""
+        _log("DEBUG", f"Looking for air date for {imdb_id} S{season:02d}E{episode:02d}")
+        
+        # Try TMDB first if available
+        if self.tmdb.enabled:
+            # Find TV show by IMDB ID
+            tv_find_result = self.tmdb._get(f"/find/{imdb_id}", {"external_source": "imdb_id"})
+            if tv_find_result and tv_find_result.get("tv_results"):
+                tv_show = tv_find_result["tv_results"][0]
+                tv_id = tv_show.get("id")
+                if tv_id:
+                    _log("DEBUG", f"Found TMDB TV ID {tv_id} for {imdb_id}")
+                    episodes = self.tmdb.get_tv_season_episodes(tv_id, season)
+                    if episode in episodes:
+                        air_date = episodes[episode]
+                        _log("INFO", f"Found TMDB air date for {imdb_id} S{season:02d}E{episode:02d}: {air_date}")
+                        return _parse_date_to_iso(air_date)
+        
+        # Try OMDb as fallback
+        if self.omdb.enabled:
+            episodes = self.omdb.get_tv_season_episodes(imdb_id, season)
+            if episode in episodes:
+                air_date = episodes[episode]
+                _log("INFO", f"Found OMDb air date for {imdb_id} S{season:02d}E{episode:02d}: {air_date}")
+                return _parse_date_to_iso(air_date)
+        
+        _log("WARNING", f"No air date found for {imdb_id} S{season:02d}E{episode:02d}")
+        return None
+
 
 if __name__ == "__main__":
     # Test the clients

clients/sonarr_client.py

@@ -248,7 +248,7 @@ class SonarrClient:
             _log("WARNING", f"No import events, using grab date: {earliest_grab['date']} for episode {episode_id}")
             return earliest_grab["date"]
         
-        _log("WARNING", f"No reliable import events found for episode {episode_id}")
+        _log("WARNING", f"No reliable import events found for episode {episode_id} - should use air date instead")
         return None
 
 

config/runtime_validator.py

@@ -0,0 +1,547 @@
+"""
+Runtime Configuration Validator for NFOGuard
+Provides validation of configuration at runtime with health checks
+"""
+import asyncio
+import time
+from pathlib import Path
+from typing import Dict, List, Any, Optional, Tuple
+from dataclasses import dataclass, field
+from contextlib import asynccontextmanager
+import aiohttp
+import sqlite3
+import psycopg2
+
+from config.validator import ValidationIssue, ValidationResult, ValidationSeverity
+from utils.exceptions import ConfigurationError, NetworkRetryableError, DatabaseError
+
+
+@dataclass
+class HealthCheckResult:
+    """Result of a runtime health check"""
+    component: str
+    is_healthy: bool
+    response_time_ms: Optional[float] = None
+    message: str = ""
+    details: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "component": self.component,
+            "is_healthy": self.is_healthy,
+            "response_time_ms": self.response_time_ms,
+            "message": self.message,
+            "details": self.details,
+            "timestamp": time.time()
+        }
+
+
+class RuntimeValidator:
+    """Runtime configuration and system health validator"""
+    
+    def __init__(self, config):
+        self.config = config
+        self._health_cache = {}
+        self._cache_ttl = 60  # Cache health results for 60 seconds
+    
+    async def validate_runtime_config(self) -> ValidationResult:
+        """Perform runtime validation of configuration"""
+        result = ValidationResult(is_valid=True)
+        
+        # Validate filesystem access
+        await self._validate_filesystem_access(result)
+        
+        # Validate database connectivity
+        await self._validate_database_connectivity(result)
+        
+        # Validate external API connectivity
+        await self._validate_api_connectivity(result)
+        
+        # Validate permissions
+        await self._validate_permissions(result)
+        
+        # Validate resource availability
+        await self._validate_resources(result)
+        
+        return result
+    
+    async def _validate_filesystem_access(self, result: ValidationResult) -> None:
+        """Validate filesystem access for media paths"""
+        all_paths = []
+        
+        # Collect all media paths
+        all_paths.extend(self.config.tv_paths)
+        all_paths.extend(self.config.movie_paths)
+        
+        for path in all_paths:
+            try:
+                # Check if path exists and is accessible
+                if not path.exists():
+                    result.add_issue(ValidationIssue(
+                        setting="media_paths",
+                        severity=ValidationSeverity.ERROR,
+                        message=f"Media path does not exist: {path}",
+                        current_value=str(path)
+                    ))
+                    continue
+                
+                # Check if path is readable
+                test_file = None
+                try:
+                    # Try to list directory contents
+                    list(path.iterdir())
+                except PermissionError:
+                    result.add_issue(ValidationIssue(
+                        setting="media_paths",
+                        severity=ValidationSeverity.ERROR,
+                        message=f"No read permission for media path: {path}",
+                        current_value=str(path)
+                    ))
+                    continue
+                except OSError as e:
+                    result.add_issue(ValidationIssue(
+                        setting="media_paths",
+                        severity=ValidationSeverity.WARNING,
+                        message=f"Error accessing media path: {e}",
+                        current_value=str(path)
+                    ))
+                    continue
+                
+                # Check write permissions (needed for NFO files)
+                try:
+                    test_file = path / ".nfoguard_write_test"
+                    test_file.write_text("test")
+                    test_file.unlink()
+                except PermissionError:
+                    result.add_issue(ValidationIssue(
+                        setting="media_paths",
+                        severity=ValidationSeverity.ERROR,
+                        message=f"No write permission for media path: {path}",
+                        current_value=str(path),
+                        details={"required_for": "NFO file creation"}
+                    ))
+                except OSError as e:
+                    result.add_issue(ValidationIssue(
+                        setting="media_paths",
+                        severity=ValidationSeverity.WARNING,
+                        message=f"Write test failed for media path: {e}",
+                        current_value=str(path)
+                    ))
+                finally:
+                    # Cleanup test file if it exists
+                    if test_file and test_file.exists():
+                        try:
+                            test_file.unlink()
+                        except:
+                            pass
+                            
+            except Exception as e:
+                result.add_issue(ValidationIssue(
+                    setting="media_paths",
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Unexpected error accessing path: {e}",
+                    current_value=str(path)
+                ))
+        
+        # Validate database directory
+        db_path = Path(self.config.db_path)
+        db_dir = db_path.parent
+        
+        if not db_dir.exists():
+            try:
+                db_dir.mkdir(parents=True, exist_ok=True)
+            except PermissionError:
+                result.add_issue(ValidationIssue(
+                    setting="DB_PATH",
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Cannot create database directory: {db_dir}",
+                    current_value=str(db_path)
+                ))
+            except OSError as e:
+                result.add_issue(ValidationIssue(
+                    setting="DB_PATH",
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Error creating database directory: {e}",
+                    current_value=str(db_path)
+                ))
+        
+        # Test database file access
+        if not db_path.exists():
+            try:
+                # Try to create the database
+                with sqlite3.connect(str(db_path)) as conn:
+                    conn.execute("SELECT 1")
+            except PermissionError:
+                result.add_issue(ValidationIssue(
+                    setting="DB_PATH",
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Cannot create database file: {db_path}",
+                    current_value=str(db_path)
+                ))
+            except sqlite3.Error as e:
+                result.add_issue(ValidationIssue(
+                    setting="DB_PATH",
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Database error: {e}",
+                    current_value=str(db_path)
+                ))
+    
+    async def _validate_database_connectivity(self, result: ValidationResult) -> None:
+        """Validate database connectivity"""
+        # Test Radarr database if configured
+        if hasattr(self.config, 'db_type') or 'RADARR_DB_TYPE' in os.environ:
+            import os
+            db_type = getattr(self.config, 'db_type', os.environ.get('RADARR_DB_TYPE', '')).lower()
+            
+            if db_type == 'postgresql':
+                await self._test_postgresql_connection(result)
+            elif db_type == 'sqlite':
+                await self._test_sqlite_connection(result)
+        
+        # Test local SQLite database
+        await self._test_local_database(result)
+    
+    async def _test_postgresql_connection(self, result: ValidationResult) -> None:
+        """Test PostgreSQL connection"""
+        import os
+        
+        try:
+            host = os.environ.get('RADARR_DB_HOST')
+            port = int(os.environ.get('RADARR_DB_PORT', 5432))
+            database = os.environ.get('RADARR_DB_NAME')
+            user = os.environ.get('RADARR_DB_USER')
+            password = os.environ.get('RADARR_DB_PASSWORD', '')
+            
+            start_time = time.time()
+            
+            # Use asyncio to run blocking DB call
+            def test_connection():
+                conn = psycopg2.connect(
+                    host=host, port=port, database=database,
+                    user=user, password=password,
+                    connect_timeout=10
+                )
+                with conn:
+                    with conn.cursor() as cur:
+                        cur.execute("SELECT 1")
+                return conn
+            
+            conn = await asyncio.get_event_loop().run_in_executor(None, test_connection)
+            conn.close()
+            
+            response_time = (time.time() - start_time) * 1000
+            
+            if response_time > 5000:  # 5 seconds
+                result.add_issue(ValidationIssue(
+                    setting="RADARR_DB",
+                    severity=ValidationSeverity.WARNING,
+                    message=f"Slow database connection ({response_time:.0f}ms)",
+                    details={"response_time_ms": response_time}
+                ))
+                
+        except psycopg2.OperationalError as e:
+            result.add_issue(ValidationIssue(
+                setting="RADARR_DB",
+                severity=ValidationSeverity.ERROR,
+                message=f"Cannot connect to PostgreSQL database: {e}",
+                details={"error_type": "connection_failed"}
+            ))
+        except Exception as e:
+            result.add_issue(ValidationIssue(
+                setting="RADARR_DB",
+                severity=ValidationSeverity.ERROR,
+                message=f"Database connection error: {e}",
+                details={"error_type": "unexpected_error"}
+            ))
+    
+    async def _test_local_database(self, result: ValidationResult) -> None:
+        """Test local SQLite database"""
+        try:
+            db_path = Path(self.config.db_path)
+            
+            start_time = time.time()
+            
+            def test_db():
+                with sqlite3.connect(str(db_path), timeout=10) as conn:
+                    conn.execute("SELECT 1")
+                    return True
+            
+            await asyncio.get_event_loop().run_in_executor(None, test_db)
+            
+            response_time = (time.time() - start_time) * 1000
+            
+            if response_time > 1000:  # 1 second is slow for SQLite
+                result.add_issue(ValidationIssue(
+                    setting="DB_PATH",
+                    severity=ValidationSeverity.WARNING,
+                    message=f"Slow local database access ({response_time:.0f}ms)",
+                    current_value=str(db_path),
+                    details={"response_time_ms": response_time}
+                ))
+                
+        except sqlite3.OperationalError as e:
+            result.add_issue(ValidationIssue(
+                setting="DB_PATH",
+                severity=ValidationSeverity.ERROR,
+                message=f"Local database error: {e}",
+                current_value=str(self.config.db_path)
+            ))
+    
+    async def _validate_api_connectivity(self, result: ValidationResult) -> None:
+        """Validate external API connectivity"""
+        apis = []
+        
+        if hasattr(self.config, 'radarr_url') and self.config.radarr_url:
+            apis.append(("Radarr", self.config.radarr_url))
+        
+        if hasattr(self.config, 'sonarr_url') and self.config.sonarr_url:
+            apis.append(("Sonarr", self.config.sonarr_url))
+        
+        for api_name, base_url in apis:
+            await self._test_api_connectivity(result, api_name, base_url)
+    
+    async def _test_api_connectivity(self, result: ValidationResult, api_name: str, base_url: str) -> None:
+        """Test connectivity to a specific API"""
+        try:
+            timeout = aiohttp.ClientTimeout(total=10)
+            
+            async with aiohttp.ClientSession(timeout=timeout) as session:
+                start_time = time.time()
+                
+                # Test basic connectivity
+                test_url = f"{base_url.rstrip('/')}/api/v1/health"
+                
+                async with session.get(test_url) as response:
+                    response_time = (time.time() - start_time) * 1000
+                    
+                    if response.status == 200:
+                        if response_time > 5000:  # 5 seconds
+                            result.add_issue(ValidationIssue(
+                                setting=f"{api_name.upper()}_URL",
+                                severity=ValidationSeverity.WARNING,
+                                message=f"Slow {api_name} API response ({response_time:.0f}ms)",
+                                current_value=base_url,
+                                details={"response_time_ms": response_time}
+                            ))
+                    else:
+                        result.add_issue(ValidationIssue(
+                            setting=f"{api_name.upper()}_URL",
+                            severity=ValidationSeverity.WARNING,
+                            message=f"{api_name} API returned HTTP {response.status}",
+                            current_value=base_url,
+                            details={"status_code": response.status}
+                        ))
+                        
+        except asyncio.TimeoutError:
+            result.add_issue(ValidationIssue(
+                setting=f"{api_name.upper()}_URL",
+                severity=ValidationSeverity.WARNING,
+                message=f"{api_name} API connection timeout",
+                current_value=base_url,
+                details={"error_type": "timeout"}
+            ))
+        except Exception as e:
+            result.add_issue(ValidationIssue(
+                setting=f"{api_name.upper()}_URL",
+                severity=ValidationSeverity.WARNING,
+                message=f"{api_name} API connection error: {e}",
+                current_value=base_url,
+                details={"error_type": "connection_error"}
+            ))
+    
+    async def _validate_permissions(self, result: ValidationResult) -> None:
+        """Validate file system permissions"""
+        # This is partially covered in filesystem validation
+        # Additional permission checks can be added here
+        pass
+    
+    async def _validate_resources(self, result: ValidationResult) -> None:
+        """Validate system resources"""
+        import shutil
+        
+        try:
+            # Check disk space for database directory
+            db_path = Path(self.config.db_path)
+            db_dir = db_path.parent if not db_path.is_dir() else db_path
+            
+            if db_dir.exists():
+                free_space = shutil.disk_usage(str(db_dir)).free
+                free_space_mb = free_space / (1024 * 1024)
+                
+                if free_space_mb < 100:  # Less than 100MB
+                    result.add_issue(ValidationIssue(
+                        setting="DB_PATH",
+                        severity=ValidationSeverity.ERROR,
+                        message=f"Low disk space in database directory ({free_space_mb:.1f}MB free)",
+                        current_value=str(db_path),
+                        details={"free_space_mb": free_space_mb}
+                    ))
+                elif free_space_mb < 1000:  # Less than 1GB
+                    result.add_issue(ValidationIssue(
+                        setting="DB_PATH",
+                        severity=ValidationSeverity.WARNING,
+                        message=f"Low disk space in database directory ({free_space_mb:.1f}MB free)",
+                        current_value=str(db_path),
+                        details={"free_space_mb": free_space_mb}
+                    ))
+                    
+        except Exception as e:
+            result.add_issue(ValidationIssue(
+                setting="system_resources",
+                severity=ValidationSeverity.WARNING,
+                message=f"Could not check disk space: {e}",
+                details={"error_type": "resource_check_failed"}
+            ))
+    
+    async def get_system_health(self) -> Dict[str, HealthCheckResult]:
+        """Get comprehensive system health status"""
+        health_checks = {}
+        
+        # Database health
+        health_checks["database"] = await self._check_database_health()
+        
+        # Filesystem health
+        health_checks["filesystem"] = await self._check_filesystem_health()
+        
+        # API health
+        health_checks["external_apis"] = await self._check_api_health()
+        
+        return health_checks
+    
+    async def _check_database_health(self) -> HealthCheckResult:
+        """Check database health"""
+        try:
+            start_time = time.time()
+            
+            def test_db():
+                with sqlite3.connect(str(self.config.db_path), timeout=5) as conn:
+                    conn.execute("SELECT COUNT(*) FROM sqlite_master")
+                    return True
+            
+            await asyncio.get_event_loop().run_in_executor(None, test_db)
+            
+            response_time = (time.time() - start_time) * 1000
+            
+            return HealthCheckResult(
+                component="database",
+                is_healthy=True,
+                response_time_ms=response_time,
+                message="Database accessible",
+                details={"db_path": str(self.config.db_path)}
+            )
+            
+        except Exception as e:
+            return HealthCheckResult(
+                component="database",
+                is_healthy=False,
+                message=f"Database error: {e}",
+                details={"db_path": str(self.config.db_path), "error": str(e)}
+            )
+    
+    async def _check_filesystem_health(self) -> HealthCheckResult:
+        """Check filesystem health"""
+        try:
+            accessible_paths = 0
+            total_paths = len(self.config.tv_paths) + len(self.config.movie_paths)
+            
+            for path in list(self.config.tv_paths) + list(self.config.movie_paths):
+                if path.exists() and path.is_dir():
+                    try:
+                        # Quick access test
+                        next(path.iterdir(), None)
+                        accessible_paths += 1
+                    except:
+                        pass
+            
+            is_healthy = accessible_paths == total_paths
+            health_percentage = (accessible_paths / total_paths * 100) if total_paths > 0 else 0
+            
+            return HealthCheckResult(
+                component="filesystem",
+                is_healthy=is_healthy,
+                message=f"{accessible_paths}/{total_paths} media paths accessible ({health_percentage:.1f}%)",
+                details={
+                    "accessible_paths": accessible_paths,
+                    "total_paths": total_paths,
+                    "health_percentage": health_percentage
+                }
+            )
+            
+        except Exception as e:
+            return HealthCheckResult(
+                component="filesystem",
+                is_healthy=False,
+                message=f"Filesystem check error: {e}",
+                details={"error": str(e)}
+            )
+    
+    async def _check_api_health(self) -> HealthCheckResult:
+        """Check external API health"""
+        apis_tested = 0
+        apis_healthy = 0
+        api_details = {}
+        
+        try:
+            # Test configured APIs
+            if hasattr(self.config, 'radarr_url') and self.config.radarr_url:
+                apis_tested += 1
+                healthy = await self._test_single_api("Radarr", self.config.radarr_url)
+                if healthy:
+                    apis_healthy += 1
+                api_details["radarr"] = {"healthy": healthy}
+            
+            if hasattr(self.config, 'sonarr_url') and self.config.sonarr_url:
+                apis_tested += 1
+                healthy = await self._test_single_api("Sonarr", self.config.sonarr_url)
+                if healthy:
+                    apis_healthy += 1
+                api_details["sonarr"] = {"healthy": healthy}
+            
+            if apis_tested == 0:
+                return HealthCheckResult(
+                    component="external_apis",
+                    is_healthy=True,
+                    message="No external APIs configured",
+                    details={"apis_configured": 0}
+                )
+            
+            is_healthy = apis_healthy == apis_tested
+            health_percentage = (apis_healthy / apis_tested * 100) if apis_tested > 0 else 0
+            
+            return HealthCheckResult(
+                component="external_apis",
+                is_healthy=is_healthy,
+                message=f"{apis_healthy}/{apis_tested} APIs healthy ({health_percentage:.1f}%)",
+                details={
+                    "healthy_apis": apis_healthy,
+                    "total_apis": apis_tested,
+                    "health_percentage": health_percentage,
+                    "api_status": api_details
+                }
+            )
+            
+        except Exception as e:
+            return HealthCheckResult(
+                component="external_apis",
+                is_healthy=False,
+                message=f"API health check error: {e}",
+                details={"error": str(e)}
+            )
+    
+    async def _test_single_api(self, api_name: str, base_url: str) -> bool:
+        """Test a single API for health"""
+        try:
+            timeout = aiohttp.ClientTimeout(total=5)
+            
+            async with aiohttp.ClientSession(timeout=timeout) as session:
+                test_url = f"{base_url.rstrip('/')}/api/v1/health"
+                
+                async with session.get(test_url) as response:
+                    return response.status == 200
+                    
+        except:
+            return False
+
+
+# Import needed for database validation
+import os

config/settings.py

@@ -1,10 +1,17 @@
 """
 NFOGuard Configuration Module
-Handles all configuration loading and validation
+Handles all configuration loading and validation with comprehensive error reporting
 """
 import os
+import sys
+import logging
 from pathlib import Path
-from typing import List
+from typing import List, Optional, Dict, Any
+
+from utils.exceptions import ConfigurationError
+
+
+logger = logging.getLogger(__name__)
 
 
 def _bool_env(name: str, default: bool) -> bool:
@@ -16,20 +23,30 @@ def _bool_env(name: str, default: bool) -> bool:
 
 
 class NFOGuardConfig:
-    """Configuration class for NFOGuard"""
+    """Configuration class for NFOGuard with integrated validation"""
     
-    def __init__(self):
+    def __init__(self, validate_on_init: bool = True, strict_validation: bool = False):
-        # Paths - No hardcoded defaults, must be configured via environment
+        """
-        tv_paths_env = os.environ.get("TV_PATHS", "")
+        Initialize NFOGuard configuration
-        movie_paths_env = os.environ.get("MOVIE_PATHS", "")
         
-        if not tv_paths_env:
+        Args:
-            raise ValueError("TV_PATHS environment variable is required but not set")
+            validate_on_init: Run validation during initialization
-        if not movie_paths_env:
+            strict_validation: Treat warnings as errors
-            raise ValueError("MOVIE_PATHS environment variable is required but not set")
+        """
+        self.strict_validation = strict_validation
+        self._validation_issues = []
         
-        self.tv_paths = [Path(p.strip()) for p in tv_paths_env.split(",") if p.strip()]
+        # Initialize configuration
-        self.movie_paths = [Path(p.strip()) for p in movie_paths_env.split(",") if p.strip()]
+        self._load_configuration()
+        
+        # Run validation if requested
+        if validate_on_init:
+            self._validate_configuration()
+    
+    def _load_configuration(self) -> None:
+        """Load all configuration from environment variables"""
+        # Core paths - Required
+        self._load_paths()
         
         # Core settings
         self.manage_nfo = _bool_env("MANAGE_NFO", True)
@@ -38,28 +55,248 @@ class NFOGuardConfig:
         self.debug = _bool_env("DEBUG", False)
         self.manager_brand = os.environ.get("MANAGER_BRAND", "NFOGuard")
         
-        # Batching
+        # Batching and performance
-        self.batch_delay = float(os.environ.get("BATCH_DELAY", "5.0"))
+        self.batch_delay = self._get_float_env("BATCH_DELAY", 5.0, 0.1, 300.0)
-        self.max_concurrent = int(os.environ.get("MAX_CONCURRENT_SERIES", "3"))
+        self.max_concurrent = self._get_int_env("MAX_CONCURRENT_SERIES", 3, 1, 10)
         
         # Database
+        self.db_type = os.environ.get("DB_TYPE", "sqlite").lower()
         self.db_path = Path(os.environ.get("DB_PATH", "/app/data/media_dates.db"))
         
+        # PostgreSQL database settings
+        if self.db_type == "postgresql":
+            self.db_host = os.environ.get("DB_HOST", "localhost")
+            self.db_port = self._get_int_env("DB_PORT", 5432, 1, 65535)
+            self.db_name = os.environ.get("DB_NAME", "nfoguard")
+            self.db_user = os.environ.get("DB_USER", "nfoguard")
+            self.db_password = os.environ.get("DB_PASSWORD", "")
+        
+        # External connections
+        self._load_external_connections()
+        
         # Movie processing
+        self._load_movie_settings()
+        
+        # TV processing
+        self._load_tv_settings()
+    
+    def _load_paths(self) -> None:
+        """Load and validate path configuration"""
+        tv_paths_env = os.environ.get("TV_PATHS", "")
+        movie_paths_env = os.environ.get("MOVIE_PATHS", "")
+        
+        if not tv_paths_env:
+            raise ConfigurationError(
+                setting="TV_PATHS",
+                reason="TV_PATHS environment variable is required but not set"
+            )
+        
+        if not movie_paths_env:
+            raise ConfigurationError(
+                setting="MOVIE_PATHS", 
+                reason="MOVIE_PATHS environment variable is required but not set"
+            )
+            
+        # Parse paths
+        self.tv_paths = [Path(p.strip()) for p in tv_paths_env.split(",") if p.strip()]
+        self.movie_paths = [Path(p.strip()) for p in movie_paths_env.split(",") if p.strip()]
+        
+        if not self.tv_paths:
+            raise ConfigurationError(
+                setting="TV_PATHS",
+                reason="No valid TV paths found after parsing",
+                current_value=tv_paths_env
+            )
+        
+        if not self.movie_paths:
+            raise ConfigurationError(
+                setting="MOVIE_PATHS",
+                reason="No valid movie paths found after parsing", 
+                current_value=movie_paths_env
+            )
+    
+    def _load_external_connections(self) -> None:
+        """Load external API and database connection settings"""
+        # API URLs
+        self.radarr_url = os.environ.get("RADARR_URL", "")
+        self.sonarr_url = os.environ.get("SONARR_URL", "")
+        self.jellyseerr_url = os.environ.get("JELLYSEERR_URL", "")
+        
+        # Radarr database settings
+        self.radarr_db_type = os.environ.get("RADARR_DB_TYPE", "").lower()
+        self.radarr_db_host = os.environ.get("RADARR_DB_HOST", "")
+        self.radarr_db_port = self._get_int_env("RADARR_DB_PORT", 5432, 1, 65535) 
+        self.radarr_db_name = os.environ.get("RADARR_DB_NAME", "")
+        self.radarr_db_user = os.environ.get("RADARR_DB_USER", "")
+        
+        # Timeout settings
+        self.timeout_seconds = self._get_int_env("TIMEOUT_SECONDS", 45, 10, 300)
+    
+    def _load_movie_settings(self) -> None:
+        """Load movie processing settings"""
         self.movie_priority = os.environ.get("MOVIE_PRIORITY", "import_then_digital").lower()
         self.prefer_release_dates_over_file_dates = _bool_env("PREFER_RELEASE_DATES_OVER_FILE_DATES", True)
         self.allow_file_date_fallback = _bool_env("ALLOW_FILE_DATE_FALLBACK", False)
-        self.release_date_priority = [p.strip() for p in os.environ.get("RELEASE_DATE_PRIORITY", "digital,physical,theatrical").split(",")]
+        
+        # Manual scan behavior
+        self.manual_scan_prioritize_nfo = _bool_env("MANUAL_SCAN_PRIORITIZE_NFO", False)
+        
+        # Release date settings
+        release_priority_env = os.environ.get("RELEASE_DATE_PRIORITY", "digital,physical,theatrical")
+        self.release_date_priority = [p.strip() for p in release_priority_env.split(",") if p.strip()]
+        
         self.enable_smart_date_validation = _bool_env("ENABLE_SMART_DATE_VALIDATION", True)
-        self.max_release_date_gap_years = int(os.environ.get("MAX_RELEASE_DATE_GAP_YEARS", "10"))
+        self.max_release_date_gap_years = self._get_int_env("MAX_RELEASE_DATE_GAP_YEARS", 10, 1, 50)
         self.movie_poll_mode = os.environ.get("MOVIE_POLL_MODE", "always").lower()
         self.movie_update_mode = os.environ.get("MOVIE_DATE_UPDATE_MODE", "backfill_only").lower()
     
-        # TV processing
+    def _load_tv_settings(self) -> None:
+        """Load TV processing settings"""
         self.tv_season_dir_format = os.environ.get("TV_SEASON_DIR_FORMAT", "Season {season:02d}")
         self.tv_season_dir_pattern = os.environ.get("TV_SEASON_DIR_PATTERN", "season ").lower()
         self.tv_webhook_processing_mode = os.environ.get("TV_WEBHOOK_PROCESSING_MODE", "targeted").lower()
     
+    def _get_int_env(self, name: str, default: int, min_val: int, max_val: int) -> int:
+        """Get integer environment variable with validation"""
+        value_str = os.environ.get(name)
+        if not value_str:
+            return default
         
-# Global config instance
+        try:
-config = NFOGuardConfig()
+            value = int(value_str)
+            if value < min_val or value > max_val:
+                raise ConfigurationError(
+                    setting=name,
+                    reason=f"Value must be between {min_val} and {max_val}",
+                    current_value=value_str
+                )
+            return value
+        except ValueError:
+            raise ConfigurationError(
+                setting=name,
+                reason=f"Invalid integer value",
+                current_value=value_str
+            )
+    
+    def _get_float_env(self, name: str, default: float, min_val: float, max_val: float) -> float:
+        """Get float environment variable with validation"""
+        value_str = os.environ.get(name)
+        if not value_str:
+            return default
+        
+        try:
+            value = float(value_str)
+            if value < min_val or value > max_val:
+                raise ConfigurationError(
+                    setting=name,
+                    reason=f"Value must be between {min_val} and {max_val}",
+                    current_value=value_str
+                )
+            return value
+        except ValueError:
+            raise ConfigurationError(
+                setting=name,
+                reason=f"Invalid float value",
+                current_value=value_str
+            )
+    
+    def _validate_configuration(self) -> None:
+        """Validate configuration using the validator"""
+        try:
+            # Import here to avoid circular imports
+            from config.validator import validate_configuration_and_raise
+            validate_configuration_and_raise()
+            
+        except ImportError:
+            # Fallback to basic validation if validator not available
+            logger.warning("Configuration validator not available, using basic validation")
+            self._basic_validation()
+        except ConfigurationError:
+            if self.strict_validation:
+                raise
+            else:
+                # Log warning but continue
+                logger.warning("Configuration validation found issues", exc_info=True)
+    
+    def _basic_validation(self) -> None:
+        """Basic fallback validation"""
+        # Validate that paths exist (basic check)
+        for path_list, path_type in [(self.tv_paths, "TV"), (self.movie_paths, "Movie")]:
+            for path in path_list:
+                if not path.is_absolute():
+                    logger.warning(f"{path_type} path should be absolute: {path}")
+    
+    def get_configuration_summary(self) -> Dict[str, Any]:
+        """Get a summary of current configuration"""
+        return {
+            "tv_paths": [str(p) for p in self.tv_paths],
+            "movie_paths": [str(p) for p in self.movie_paths],
+            "database": {
+                "type": self.db_type,
+                "path": str(self.db_path) if self.db_type == "sqlite" else None,
+                "host": getattr(self, 'db_host', None) if self.db_type == "postgresql" else None,
+                "port": getattr(self, 'db_port', None) if self.db_type == "postgresql" else None,
+                "name": getattr(self, 'db_name', None) if self.db_type == "postgresql" else None
+            },
+            "external_apis": {
+                "radarr": bool(self.radarr_url),
+                "sonarr": bool(self.sonarr_url),
+                "jellyseerr": bool(self.jellyseerr_url)
+            },
+            "radarr_database": {
+                "type": getattr(self, 'radarr_db_type', None),
+                "configured": bool(getattr(self, 'radarr_db_type', None) and getattr(self, 'radarr_db_host', None))
+            },
+            "performance": {
+                "batch_delay": self.batch_delay,
+                "max_concurrent": self.max_concurrent,
+                "timeout_seconds": self.timeout_seconds
+            },
+            "features": {
+                "manage_nfo": self.manage_nfo,
+                "fix_dir_mtimes": self.fix_dir_mtimes,
+                "lock_metadata": self.lock_metadata,
+                "debug": self.debug,
+                "manual_scan_prioritize_nfo": self.manual_scan_prioritize_nfo
+            }
+        }
+    
+    def validate_runtime_access(self) -> Dict[str, bool]:
+        """Quick runtime validation of critical paths"""
+        results = {
+            "tv_paths_accessible": True,
+            "movie_paths_accessible": True,
+            "database_writable": True
+        }
+        
+        # Test TV paths
+        for path in self.tv_paths:
+            if not path.exists() or not path.is_dir():
+                results["tv_paths_accessible"] = False
+                break
+        
+        # Test movie paths  
+        for path in self.movie_paths:
+            if not path.exists() or not path.is_dir():
+                results["movie_paths_accessible"] = False
+                break
+        
+        # Test database directory
+        db_dir = self.db_path.parent
+        try:
+            if not db_dir.exists():
+                db_dir.mkdir(parents=True, exist_ok=True)
+            
+            # Test write access
+            test_file = db_dir / ".nfoguard_write_test"
+            test_file.write_text("test")
+            test_file.unlink()
+        except (PermissionError, OSError):
+            results["database_writable"] = False
+        
+        return results
+
+
+# Global config instance - Initialize with validation disabled by default for backwards compatibility
+# Applications can enable validation by creating their own instance with validate_on_init=True
+config = NFOGuardConfig(validate_on_init=False)

config/validation_cli.py

@@ -0,0 +1,293 @@
+#!/usr/bin/env python3
+"""
+Configuration Validation CLI for NFOGuard
+Provides command-line validation and reporting of configuration issues
+"""
+import sys
+import json
+import argparse
+from pathlib import Path
+from typing import Dict, Any, List
+from datetime import datetime
+
+from config.validator import validate_configuration, ValidationSeverity
+from config.runtime_validator import RuntimeValidator
+from config.settings import NFOGuardConfig
+
+
+class ValidationReporter:
+    """Formats and displays validation results"""
+    
+    def __init__(self, verbose: bool = False, json_output: bool = False):
+        self.verbose = verbose
+        self.json_output = json_output
+        
+        # Color codes for terminal output
+        self.colors = {
+            'error': '\033[91m',    # Red
+            'warning': '\033[93m',  # Yellow
+            'info': '\033[94m',     # Blue
+            'success': '\033[92m',  # Green
+            'reset': '\033[0m',     # Reset
+            'bold': '\033[1m'       # Bold
+        }
+    
+    def report_validation_results(self, result, runtime_result=None) -> int:
+        """
+        Report validation results
+        
+        Returns:
+            Exit code (0 for success, 1 for warnings, 2 for errors)
+        """
+        if self.json_output:
+            return self._report_json(result, runtime_result)
+        else:
+            return self._report_human_readable(result, runtime_result)
+    
+    def _report_json(self, result, runtime_result=None) -> int:
+        """Report results in JSON format"""
+        output = {
+            "timestamp": datetime.now().isoformat(),
+            "validation": result.to_dict()
+        }
+        
+        if runtime_result:
+            output["runtime_validation"] = runtime_result.to_dict()
+        
+        print(json.dumps(output, indent=2))
+        
+        if result.errors_count > 0:
+            return 2
+        elif result.warnings_count > 0:
+            return 1
+        return 0
+    
+    def _report_human_readable(self, result, runtime_result=None) -> int:
+        """Report results in human-readable format"""
+        print(f"{self.colors['bold']}NFOGuard Configuration Validation Report{self.colors['reset']}")
+        print("=" * 50)
+        
+        # Overall status
+        if result.is_valid:
+            status_color = self.colors['success']
+            status_text = "✓ VALID"
+        else:
+            status_color = self.colors['error']
+            status_text = "✗ INVALID"
+        
+        print(f"Status: {status_color}{status_text}{self.colors['reset']}")
+        print(f"Errors: {result.errors_count}")
+        print(f"Warnings: {result.warnings_count}")
+        print(f"Total Issues: {len(result.issues)}")
+        print()
+        
+        # Report issues by severity
+        if result.issues:
+            self._report_issues_by_severity(result.issues)
+        
+        # Report runtime validation if available
+        if runtime_result:
+            print(f"\n{self.colors['bold']}Runtime Validation{self.colors['reset']}")
+            print("-" * 20)
+            if runtime_result.issues:
+                self._report_issues_by_severity(runtime_result.issues, "Runtime")
+            else:
+                print(f"{self.colors['success']}✓ All runtime checks passed{self.colors['reset']}")
+        
+        # Summary and recommendations
+        self._report_summary_and_recommendations(result)
+        
+        # Return appropriate exit code
+        if result.errors_count > 0 or (runtime_result and runtime_result.errors_count > 0):
+            return 2
+        elif result.warnings_count > 0 or (runtime_result and runtime_result.warnings_count > 0):
+            return 1
+        return 0
+    
+    def _report_issues_by_severity(self, issues: List, context: str = "Configuration") -> None:
+        """Report issues grouped by severity"""
+        errors = [issue for issue in issues if issue.severity == ValidationSeverity.ERROR]
+        warnings = [issue for issue in issues if issue.severity == ValidationSeverity.WARNING]
+        info_issues = [issue for issue in issues if issue.severity == ValidationSeverity.INFO]
+        
+        if errors:
+            print(f"{self.colors['error']}{self.colors['bold']}ERRORS ({len(errors)}):{self.colors['reset']}")
+            for issue in errors:
+                self._format_issue(issue)
+            print()
+        
+        if warnings:
+            print(f"{self.colors['warning']}{self.colors['bold']}WARNINGS ({len(warnings)}):{self.colors['reset']}")
+            for issue in warnings:
+                self._format_issue(issue)
+            print()
+        
+        if info_issues and self.verbose:
+            print(f"{self.colors['info']}{self.colors['bold']}INFO ({len(info_issues)}):{self.colors['reset']}")
+            for issue in info_issues:
+                self._format_issue(issue)
+            print()
+    
+    def _format_issue(self, issue) -> None:
+        """Format a single validation issue"""
+        severity_colors = {
+            ValidationSeverity.ERROR: self.colors['error'],
+            ValidationSeverity.WARNING: self.colors['warning'],
+            ValidationSeverity.INFO: self.colors['info']
+        }
+        
+        color = severity_colors.get(issue.severity, '')
+        
+        print(f"  {color}• {issue.setting}:{self.colors['reset']} {issue.message}")
+        
+        if issue.current_value is not None:
+            print(f"    Current: {issue.current_value}")
+        
+        if issue.suggested_value is not None:
+            print(f"    Suggested: {issue.suggested_value}")
+        
+        if self.verbose and issue.details:
+            print(f"    Details: {issue.details}")
+    
+    def _report_summary_and_recommendations(self, result) -> None:
+        """Report summary and general recommendations"""
+        print(f"\n{self.colors['bold']}Summary{self.colors['reset']}")
+        print("-" * 10)
+        
+        if result.is_valid:
+            print(f"{self.colors['success']}✓ Your configuration is valid and ready to use!{self.colors['reset']}")
+        else:
+            print(f"{self.colors['error']}✗ Your configuration has issues that need to be resolved.{self.colors['reset']}")
+        
+        # Provide specific recommendations based on issue types
+        recommendations = self._generate_recommendations(result)
+        if recommendations:
+            print(f"\n{self.colors['bold']}Recommendations{self.colors['reset']}")
+            print("-" * 15)
+            for rec in recommendations:
+                print(f"  {self.colors['info']}• {rec}{self.colors['reset']}")
+    
+    def _generate_recommendations(self, result) -> List[str]:
+        """Generate recommendations based on validation results"""
+        recommendations = []
+        
+        # Check for common patterns
+        path_issues = [issue for issue in result.issues if 'path' in issue.setting.lower()]
+        if path_issues:
+            recommendations.append("Verify all file paths are correct for your environment")
+            recommendations.append("Ensure paths are absolute when using Docker")
+        
+        db_issues = [issue for issue in result.issues if 'db' in issue.setting.lower()]
+        if db_issues:
+            recommendations.append("Check database connection settings and credentials")
+        
+        url_issues = [issue for issue in result.issues if 'url' in issue.setting.lower()]
+        if url_issues:
+            recommendations.append("Verify API URLs are reachable and include the correct port")
+        
+        perf_issues = [issue for issue in result.issues if any(perf in issue.setting.lower() 
+                      for perf in ['concurrent', 'delay', 'timeout'])]
+        if perf_issues:
+            recommendations.append("Consider adjusting performance settings based on your system resources")
+        
+        # General recommendations
+        if result.errors_count > 0:
+            recommendations.append("Fix all ERROR-level issues before starting NFOGuard")
+        
+        if result.warnings_count > 0:
+            recommendations.append("Review WARNING-level issues to optimize performance and reliability")
+        
+        return recommendations
+
+
+async def run_validation(args) -> int:
+    """Run configuration validation"""
+    reporter = ValidationReporter(verbose=args.verbose, json_output=args.json)
+    
+    try:
+        # Run static validation
+        print("Running configuration validation..." if not args.json else "", file=sys.stderr)
+        result = validate_configuration()
+        
+        runtime_result = None
+        
+        # Run runtime validation if requested
+        if args.runtime:
+            print("Running runtime validation..." if not args.json else "", file=sys.stderr)
+            try:
+                config = NFOGuardConfig()
+                runtime_validator = RuntimeValidator(config)
+                runtime_result = await runtime_validator.validate_runtime_config()
+            except Exception as e:
+                if not args.json:
+                    print(f"Runtime validation failed: {e}", file=sys.stderr)
+                # Continue with static validation results
+        
+        # Report results
+        return reporter.report_validation_results(result, runtime_result)
+        
+    except Exception as e:
+        if args.json:
+            error_output = {
+                "timestamp": datetime.now().isoformat(),
+                "error": {
+                    "message": str(e),
+                    "type": type(e).__name__
+                }
+            }
+            print(json.dumps(error_output, indent=2))
+        else:
+            print(f"Validation failed: {e}", file=sys.stderr)
+        return 2
+
+
+def main():
+    """Main CLI entry point"""
+    parser = argparse.ArgumentParser(
+        description="Validate NFOGuard configuration",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  %(prog)s                    # Basic validation
+  %(prog)s --runtime          # Include runtime checks
+  %(prog)s --verbose          # Show detailed information
+  %(prog)s --json             # Output JSON format
+  %(prog)s --runtime --json   # Runtime validation with JSON output
+        """
+    )
+    
+    parser.add_argument(
+        "--runtime",
+        action="store_true",
+        help="Perform runtime validation (tests actual connectivity and permissions)"
+    )
+    
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        help="Show verbose output including info-level messages"
+    )
+    
+    parser.add_argument(
+        "--json",
+        action="store_true",
+        help="Output results in JSON format"
+    )
+    
+    args = parser.parse_args()
+    
+    # Run validation
+    import asyncio
+    try:
+        exit_code = asyncio.run(run_validation(args))
+        sys.exit(exit_code)
+    except KeyboardInterrupt:
+        print("Validation interrupted", file=sys.stderr)
+        sys.exit(130)
+    except Exception as e:
+        print(f"Unexpected error: {e}", file=sys.stderr)
+        sys.exit(2)
+
+
+if __name__ == "__main__":
+    main()

config/validator.py

@@ -0,0 +1,498 @@
+"""
+Configuration Validator for NFOGuard
+Provides comprehensive validation of all configuration settings with detailed error reporting
+"""
+import os
+import re
+from pathlib import Path
+from typing import Dict, List, Any, Optional, Union, Type, Callable
+from dataclasses import dataclass, field
+from enum import Enum
+
+from utils.exceptions import ConfigurationError
+from utils.validation import validate_url_format
+
+
+class ValidationSeverity(Enum):
+    """Severity levels for validation issues"""
+    ERROR = "error"      # Configuration is invalid, will cause failures
+    WARNING = "warning"  # Configuration may cause issues but is workable
+    INFO = "info"        # Configuration could be improved
+
+
+@dataclass
+class ValidationIssue:
+    """Represents a configuration validation issue"""
+    setting: str
+    severity: ValidationSeverity
+    message: str
+    current_value: Any = None
+    suggested_value: Any = None
+    details: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for structured logging"""
+        return {
+            "setting": self.setting,
+            "severity": self.severity.value,
+            "message": self.message,
+            "current_value": str(self.current_value) if self.current_value is not None else None,
+            "suggested_value": str(self.suggested_value) if self.suggested_value is not None else None,
+            "details": self.details
+        }
+
+
+@dataclass
+class ValidationResult:
+    """Results of configuration validation"""
+    is_valid: bool
+    issues: List[ValidationIssue] = field(default_factory=list)
+    warnings_count: int = 0
+    errors_count: int = 0
+    
+    def add_issue(self, issue: ValidationIssue) -> None:
+        """Add a validation issue"""
+        self.issues.append(issue)
+        if issue.severity == ValidationSeverity.ERROR:
+            self.errors_count += 1
+            self.is_valid = False
+        elif issue.severity == ValidationSeverity.WARNING:
+            self.warnings_count += 1
+    
+    def get_errors(self) -> List[ValidationIssue]:
+        """Get only error-level issues"""
+        return [issue for issue in self.issues if issue.severity == ValidationSeverity.ERROR]
+    
+    def get_warnings(self) -> List[ValidationIssue]:
+        """Get only warning-level issues"""
+        return [issue for issue in self.issues if issue.severity == ValidationSeverity.WARNING]
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for structured logging"""
+        return {
+            "is_valid": self.is_valid,
+            "errors_count": self.errors_count,
+            "warnings_count": self.warnings_count,
+            "issues": [issue.to_dict() for issue in self.issues]
+        }
+
+
+class ConfigValidator:
+    """Comprehensive configuration validator for NFOGuard"""
+    
+    def __init__(self):
+        self.result = ValidationResult(is_valid=True)
+        
+        # Define validation rules
+        self._path_settings = {
+            "TV_PATHS", "MOVIE_PATHS", "RADARR_ROOT_FOLDERS", 
+            "SONARR_ROOT_FOLDERS", "DB_PATH", "LOG_DIR"
+        }
+        
+        self._url_settings = {
+            "RADARR_URL", "SONARR_URL", "JELLYSEERR_URL"
+        }
+        
+        self._required_settings = {
+            "TV_PATHS", "MOVIE_PATHS"
+        }
+        
+        self._numeric_settings = {
+            "BATCH_DELAY": (float, 0.1, 300.0),
+            "MAX_CONCURRENT_SERIES": (int, 1, 10),
+            "TIMEOUT_SECONDS": (int, 10, 300),
+            "PORT": (int, 1024, 65535),
+            "RADARR_DB_PORT": (int, 1, 65535),
+            "MAX_RELEASE_DATE_GAP_YEARS": (int, 1, 50)
+        }
+        
+        self._boolean_settings = {
+            "MANAGE_NFO", "FIX_DIR_MTIMES", "LOCK_METADATA", "DEBUG",
+            "PREFER_RELEASE_DATES_OVER_FILE_DATES", "ALLOW_FILE_DATE_FALLBACK",
+            "ENABLE_SMART_DATE_VALIDATION", "PATH_DEBUG", "SUPPRESS_TVDB_WARNINGS"
+        }
+        
+        self._choice_settings = {
+            "MOVIE_PRIORITY": ["import_then_digital", "digital_first", "file_date_only"],
+            "MOVIE_POLL_MODE": ["always", "missing_only", "never"],
+            "MOVIE_DATE_UPDATE_MODE": ["overwrite", "backfill_only", "preserve_existing"],
+            "TV_WEBHOOK_PROCESSING_MODE": ["targeted", "full_scan", "hybrid"],
+            "UPDATE_MODE": ["always", "missing_only", "never"],
+            "MTIME_BEHAVIOR": ["update", "leave_alone"],
+            "RADARR_DB_TYPE": ["postgresql", "sqlite"]
+        }
+    
+    def validate_all(self) -> ValidationResult:
+        """Validate all configuration settings"""
+        self.result = ValidationResult(is_valid=True)
+        
+        # Validate required settings
+        self._validate_required_settings()
+        
+        # Validate paths
+        self._validate_paths()
+        
+        # Validate URLs
+        self._validate_urls()
+        
+        # Validate numeric settings
+        self._validate_numeric_settings()
+        
+        # Validate boolean settings
+        self._validate_boolean_settings()
+        
+        # Validate choice settings
+        self._validate_choice_settings()
+        
+        # Validate database configuration
+        self._validate_database_config()
+        
+        # Validate release date configuration
+        self._validate_release_date_config()
+        
+        # Validate performance settings
+        self._validate_performance_settings()
+        
+        # Validate cross-setting dependencies
+        self._validate_dependencies()
+        
+        return self.result
+    
+    def _validate_required_settings(self) -> None:
+        """Validate required environment variables are set"""
+        for setting in self._required_settings:
+            value = os.environ.get(setting)
+            if not value:
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Required setting {setting} is not set",
+                    current_value=value
+                ))
+            elif not value.strip():
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Required setting {setting} is empty",
+                    current_value=value
+                ))
+    
+    def _validate_paths(self) -> None:
+        """Validate all path-related settings"""
+        for setting in self._path_settings:
+            value = os.environ.get(setting)
+            if not value:
+                if setting in self._required_settings:
+                    continue  # Already handled in required validation
+                else:
+                    # Optional path settings
+                    continue
+            
+            if setting in {"TV_PATHS", "MOVIE_PATHS", "RADARR_ROOT_FOLDERS", "SONARR_ROOT_FOLDERS"}:
+                # Multi-path settings
+                paths = [p.strip() for p in value.split(",") if p.strip()]
+                if not paths:
+                    self.result.add_issue(ValidationIssue(
+                        setting=setting,
+                        severity=ValidationSeverity.ERROR,
+                        message=f"No valid paths found in {setting}",
+                        current_value=value
+                    ))
+                    continue
+                
+                for path_str in paths:
+                    self._validate_single_path(setting, path_str)
+            else:
+                # Single path settings
+                self._validate_single_path(setting, value)
+    
+    def _validate_single_path(self, setting: str, path_str: str) -> None:
+        """Validate a single path"""
+        try:
+            path = Path(path_str)
+            
+            # Check if path is absolute (recommended for Docker)
+            if not path.is_absolute():
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.WARNING,
+                    message=f"Path should be absolute for reliable Docker operation",
+                    current_value=path_str,
+                    suggested_value=f"Use absolute path like /media/..."
+                ))
+            
+            # For media paths, check existence if not in container
+            if setting in {"TV_PATHS", "MOVIE_PATHS"} and not self._is_likely_container_path(path_str):
+                if not path.exists():
+                    self.result.add_issue(ValidationIssue(
+                        setting=setting,
+                        severity=ValidationSeverity.WARNING,
+                        message=f"Path does not exist (may be valid in container)",
+                        current_value=path_str,
+                        details={"path_type": "media"}
+                    ))
+                elif not path.is_dir():
+                    self.result.add_issue(ValidationIssue(
+                        setting=setting,
+                        severity=ValidationSeverity.ERROR,
+                        message=f"Path exists but is not a directory",
+                        current_value=path_str
+                    ))
+            
+            # Check for database directory
+            if setting == "DB_PATH":
+                parent_dir = path.parent
+                if not self._is_likely_container_path(str(parent_dir)) and not parent_dir.exists():
+                    self.result.add_issue(ValidationIssue(
+                        setting=setting,
+                        severity=ValidationSeverity.WARNING,
+                        message=f"Database directory does not exist: {parent_dir}",
+                        current_value=path_str
+                    ))
+                    
+        except (OSError, ValueError) as e:
+            self.result.add_issue(ValidationIssue(
+                setting=setting,
+                severity=ValidationSeverity.ERROR,
+                message=f"Invalid path format: {e}",
+                current_value=path_str
+            ))
+    
+    def _is_likely_container_path(self, path: str) -> bool:
+        """Check if path looks like a container path"""
+        container_indicators = ["/app/", "/media/", "/config/", "/data/"]
+        return any(indicator in path for indicator in container_indicators)
+    
+    def _validate_urls(self) -> None:
+        """Validate URL settings"""
+        for setting in self._url_settings:
+            value = os.environ.get(setting)
+            if value and not validate_url_format(value):
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Invalid URL format",
+                    current_value=value,
+                    suggested_value="Use format: http://hostname:port or https://hostname:port"
+                ))
+    
+    def _validate_numeric_settings(self) -> None:
+        """Validate numeric settings"""
+        for setting, (type_class, min_val, max_val) in self._numeric_settings.items():
+            value = os.environ.get(setting)
+            if not value:
+                continue
+                
+            try:
+                numeric_value = type_class(value)
+                if numeric_value < min_val or numeric_value > max_val:
+                    self.result.add_issue(ValidationIssue(
+                        setting=setting,
+                        severity=ValidationSeverity.ERROR,
+                        message=f"Value must be between {min_val} and {max_val}",
+                        current_value=value,
+                        suggested_value=f"Use value between {min_val}-{max_val}"
+                    ))
+            except (ValueError, TypeError):
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Invalid {type_class.__name__} value",
+                    current_value=value,
+                    suggested_value=f"Use a valid {type_class.__name__} between {min_val}-{max_val}"
+                ))
+    
+    def _validate_boolean_settings(self) -> None:
+        """Validate boolean settings"""
+        valid_true = {"1", "true", "yes", "y", "on"}
+        valid_false = {"0", "false", "no", "n", "off"}
+        valid_values = valid_true | valid_false
+        
+        for setting in self._boolean_settings:
+            value = os.environ.get(setting)
+            if value and value.lower() not in valid_values:
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Invalid boolean value",
+                    current_value=value,
+                    suggested_value="Use: true/false, 1/0, yes/no, y/n, on/off"
+                ))
+    
+    def _validate_choice_settings(self) -> None:
+        """Validate settings with predefined choices"""
+        for setting, valid_choices in self._choice_settings.items():
+            value = os.environ.get(setting)
+            if value and value.lower() not in [choice.lower() for choice in valid_choices]:
+                self.result.add_issue(ValidationIssue(
+                    setting=setting,
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Invalid choice",
+                    current_value=value,
+                    suggested_value=f"Use one of: {', '.join(valid_choices)}"
+                ))
+    
+    def _validate_database_config(self) -> None:
+        """Validate database configuration"""
+        db_type = os.environ.get("RADARR_DB_TYPE", "").lower()
+        
+        if db_type == "postgresql":
+            # Check required PostgreSQL settings
+            required_pg_settings = ["RADARR_DB_HOST", "RADARR_DB_NAME", "RADARR_DB_USER"]
+            for setting in required_pg_settings:
+                if not os.environ.get(setting):
+                    self.result.add_issue(ValidationIssue(
+                        setting=setting,
+                        severity=ValidationSeverity.ERROR,
+                        message=f"Required for PostgreSQL database connection",
+                        current_value=os.environ.get(setting)
+                    ))
+    
+    def _validate_release_date_config(self) -> None:
+        """Validate release date processing configuration"""
+        priority = os.environ.get("RELEASE_DATE_PRIORITY", "")
+        if priority:
+            priorities = [p.strip().lower() for p in priority.split(",")]
+            valid_priorities = {"digital", "physical", "theatrical"}
+            
+            invalid_priorities = [p for p in priorities if p not in valid_priorities]
+            if invalid_priorities:
+                self.result.add_issue(ValidationIssue(
+                    setting="RELEASE_DATE_PRIORITY",
+                    severity=ValidationSeverity.ERROR,
+                    message=f"Invalid release date priorities: {', '.join(invalid_priorities)}",
+                    current_value=priority,
+                    suggested_value="Use: digital, physical, theatrical"
+                ))
+            
+            if len(set(priorities)) != len(priorities):
+                self.result.add_issue(ValidationIssue(
+                    setting="RELEASE_DATE_PRIORITY",
+                    severity=ValidationSeverity.WARNING,
+                    message="Duplicate priorities found",
+                    current_value=priority
+                ))
+    
+    def _validate_performance_settings(self) -> None:
+        """Validate performance-related settings"""
+        batch_delay = os.environ.get("BATCH_DELAY")
+        max_concurrent = os.environ.get("MAX_CONCURRENT_SERIES")
+        
+        # Performance recommendations
+        if batch_delay:
+            try:
+                delay = float(batch_delay)
+                if delay < 1.0:
+                    self.result.add_issue(ValidationIssue(
+                        setting="BATCH_DELAY",
+                        severity=ValidationSeverity.WARNING,
+                        message="Very low batch delay may increase system load",
+                        current_value=batch_delay,
+                        suggested_value="Consider using 1.0 or higher for better stability"
+                    ))
+            except ValueError:
+                pass  # Already caught in numeric validation
+        
+        if max_concurrent:
+            try:
+                concurrent = int(max_concurrent)
+                if concurrent > 5:
+                    self.result.add_issue(ValidationIssue(
+                        setting="MAX_CONCURRENT_SERIES",
+                        severity=ValidationSeverity.WARNING,
+                        message="High concurrency may overload system resources",
+                        current_value=max_concurrent,
+                        suggested_value="Consider using 3-5 for optimal balance"
+                    ))
+            except ValueError:
+                pass  # Already caught in numeric validation
+    
+    def _validate_dependencies(self) -> None:
+        """Validate cross-setting dependencies"""
+        # If database is configured, recommend using database over API
+        db_type = os.environ.get("RADARR_DB_TYPE")
+        radarr_url = os.environ.get("RADARR_URL")
+        
+        if db_type and radarr_url:
+            self.result.add_issue(ValidationIssue(
+                setting="RADARR_DB_TYPE",
+                severity=ValidationSeverity.INFO,
+                message="Database connection preferred over API for better performance",
+                details={"recommendation": "Database access is faster and more reliable"}
+            ))
+        
+        # Check path mapping consistency
+        tv_paths = os.environ.get("TV_PATHS", "").split(",")
+        sonarr_paths = os.environ.get("SONARR_ROOT_FOLDERS", "").split(",")
+        
+        if len([p for p in tv_paths if p.strip()]) != len([p for p in sonarr_paths if p.strip()]):
+            self.result.add_issue(ValidationIssue(
+                setting="TV_PATHS",
+                severity=ValidationSeverity.WARNING,
+                message="TV_PATHS and SONARR_ROOT_FOLDERS should have matching number of paths",
+                details={
+                    "tv_paths_count": len([p for p in tv_paths if p.strip()]),
+                    "sonarr_paths_count": len([p for p in sonarr_paths if p.strip()])
+                }
+            ))
+
+
+def validate_configuration() -> ValidationResult:
+    """
+    Validate the complete NFOGuard configuration
+    
+    Returns:
+        ValidationResult with all validation issues found
+    """
+    validator = ConfigValidator()
+    return validator.validate_all()
+
+
+def validate_configuration_and_raise() -> None:
+    """
+    Validate configuration and raise ConfigurationError if invalid
+    
+    Raises:
+        ConfigurationError: If configuration validation fails
+    """
+    result = validate_configuration()
+    
+    if not result.is_valid:
+        error_messages = []
+        for error in result.get_errors():
+            error_messages.append(f"{error.setting}: {error.message}")
+        
+        raise ConfigurationError(
+            setting="configuration",
+            reason=f"Configuration validation failed with {result.errors_count} errors",
+            current_value={
+                "errors": error_messages,
+                "warnings_count": result.warnings_count,
+                "validation_details": result.to_dict()
+            }
+        )
+
+
+def get_configuration_summary() -> Dict[str, Any]:
+    """
+    Get a summary of current configuration status
+    
+    Returns:
+        Dictionary with configuration summary
+    """
+    result = validate_configuration()
+    
+    return {
+        "is_valid": result.is_valid,
+        "errors_count": result.errors_count,
+        "warnings_count": result.warnings_count,
+        "total_issues": len(result.issues),
+        "critical_issues": [
+            issue.to_dict() for issue in result.issues 
+            if issue.severity == ValidationSeverity.ERROR
+        ],
+        "recommendations": [
+            issue.to_dict() for issue in result.issues 
+            if issue.severity == ValidationSeverity.WARNING
+        ]
+    }

core/async_nfo_manager.py

@@ -4,7 +4,7 @@ High-performance async NFO file operations with concurrent processing
 """
 import asyncio
 from pathlib import Path
-from typing import Optional, List, Dict, Any
+from typing import Optional, List, Dict, Any, Tuple
 from datetime import datetime
 import xml.etree.ElementTree as ET
 

core/database.py

@@ -1,143 +1,152 @@
 #!/usr/bin/env python3
 """
-Database management for NFOGuard
+PostgreSQL database management for NFOGuard
-Handles SQLite database operations for tracking media dates and processing history
+Handles database operations for tracking media dates and processing history
 """
-import sqlite3
 import json
 import threading
-from pathlib import Path
 from datetime import datetime
 from typing import Optional, Dict, List, Any
 from contextlib import contextmanager
 
-class NFOGuardDatabase:
+import psycopg2
-    """Manages NFOGuard SQLite database operations"""
+import psycopg2.extras
+
+class NFOGuardDatabase:
+    """PostgreSQL database manager for NFOGuard media tracking and processing history"""
+    
+    def __init__(self, config):
+        """
+        Initialize PostgreSQL database connection
+        
+        Args:
+            config: Configuration object with database settings
+        """
+        if not config:
+            raise ValueError("PostgreSQL configuration is required")
+        self.db_host = config.db_host
+        self.db_port = config.db_port
+        self.db_name = config.db_name
+        self.db_user = config.db_user
+        self.db_password = config.db_password
         
-    def __init__(self, db_path: Path):
-        self.db_path = Path(db_path)
-        self.db_path.parent.mkdir(parents=True, exist_ok=True)
         self._local = threading.local()
         self._init_database()
     
-    def _get_connection(self) -> sqlite3.Connection:
+    
-        """Get thread-local database connection"""
+    def _get_connection(self) -> 'psycopg2.extensions.connection':
+        """Get thread-local PostgreSQL database connection"""
         if not hasattr(self._local, 'connection'):
-            self._local.connection = sqlite3.connect(
+            self._local.connection = psycopg2.connect(
-                self.db_path, 
+                host=self.db_host,
-                check_same_thread=False,
+                port=self.db_port,
-                timeout=30.0
+                database=self.db_name,
+                user=self.db_user,
+                password=self.db_password,
+                cursor_factory=psycopg2.extras.RealDictCursor
             )
-            self._local.connection.row_factory = sqlite3.Row
+            self._local.connection.autocommit = True
         return self._local.connection
     
+    def _get_first_value(self, row):
+        """Get first value from row from PostgreSQL RealDictCursor"""
+        # RealDictCursor returns dict-like objects
+        return list(row.values())[0] if row else None
+    
     @contextmanager
     def get_connection(self):
-        """Context manager for database connections"""
+        """Context manager for PostgreSQL database connections"""
         conn = self._get_connection()
         try:
             yield conn
-            conn.commit()
+            # PostgreSQL uses autocommit - no manual commit needed
         except Exception:
-            conn.rollback()
+            # PostgreSQL uses autocommit - no manual rollback needed
             raise
     
     def _init_database(self):
-        """Initialize database tables with migration support"""
+        """Initialize PostgreSQL database tables"""
         with self.get_connection() as conn:
             cursor = conn.cursor()
+            self._init_postgresql_tables(cursor)
             
-            # Series table
+            # Test the connection works and verify autocommit
-            cursor.execute("""
+            cursor.execute("SELECT 1")
-                CREATE TABLE IF NOT EXISTS series (
+            print(f"✅ PostgreSQL database initialized and connection verified")
-                    imdb_id TEXT PRIMARY KEY,
+            print(f"🔍 Autocommit status: {conn.autocommit}")
-                    path TEXT NOT NULL,
-                    last_updated TEXT NOT NULL,
-                    metadata TEXT
-                )
-            """)
     
-            # Episodes table
+    def _init_postgresql_tables(self, cursor):
-            cursor.execute("""
+        """Initialize database tables"""
-                CREATE TABLE IF NOT EXISTS episodes (
+        # Series table
-                    imdb_id TEXT NOT NULL,
+        cursor.execute("""
-                    season INTEGER NOT NULL,
+            CREATE TABLE IF NOT EXISTS series (
-                    episode INTEGER NOT NULL,
+                imdb_id VARCHAR(20) PRIMARY KEY,
-                    aired TEXT,
+                path TEXT NOT NULL,
-                    dateadded TEXT,
+                last_updated TIMESTAMP NOT NULL,
-                    source TEXT,
+                metadata JSONB
-                    last_updated TEXT NOT NULL,
+            )
-                    PRIMARY KEY (imdb_id, season, episode),
+        """)
-                    FOREIGN KEY (imdb_id) REFERENCES series(imdb_id)
-                )
-            """)
         
-            # Movies table
+        # Episodes table
-            cursor.execute("""
+        cursor.execute("""
-                CREATE TABLE IF NOT EXISTS movies (
+            CREATE TABLE IF NOT EXISTS episodes (
-                    imdb_id TEXT PRIMARY KEY,
+                imdb_id VARCHAR(20) NOT NULL,
-                    path TEXT NOT NULL,
+                season INTEGER NOT NULL,
-                    released TEXT,
+                episode INTEGER NOT NULL,
-                    dateadded TEXT,
+                aired DATE,
-                    source TEXT,
+                dateadded TIMESTAMP,
-                    last_updated TEXT NOT NULL
+                source VARCHAR(100),
-                )
+                last_updated TIMESTAMP NOT NULL,
-            """)
+                has_video_file BOOLEAN DEFAULT FALSE,
+                PRIMARY KEY (imdb_id, season, episode),
+                FOREIGN KEY (imdb_id) REFERENCES series(imdb_id)
+            )
+        """)
         
-            # Processing history table
+        # Movies table
-            cursor.execute("""
+        cursor.execute("""
-                CREATE TABLE IF NOT EXISTS processing_history (
+            CREATE TABLE IF NOT EXISTS movies (
-                    id INTEGER PRIMARY KEY AUTOINCREMENT,
+                imdb_id VARCHAR(20) PRIMARY KEY,
-                    imdb_id TEXT NOT NULL,
+                path TEXT NOT NULL,
-                    media_type TEXT NOT NULL,
+                released DATE,
-                    event_type TEXT NOT NULL,
+                dateadded TIMESTAMP,
-                    processed_at TEXT NOT NULL,
+                source VARCHAR(100),
-                    details TEXT
+                last_updated TIMESTAMP NOT NULL,
-                )
+                has_video_file BOOLEAN DEFAULT FALSE
-            """)
+            )
+        """)
         
-            # Add missing columns if they don't exist (migration)
+        # Processing history table
-            # Check current schema and add missing columns
+        cursor.execute("""
-            cursor.execute("PRAGMA table_info(movies)")
+            CREATE TABLE IF NOT EXISTS processing_history (
-            movie_columns = [row[1] for row in cursor.fetchall()]
+                id SERIAL PRIMARY KEY,
-            
+                imdb_id VARCHAR(20) NOT NULL,
-            cursor.execute("PRAGMA table_info(episodes)")
+                media_type VARCHAR(20) NOT NULL,
-            episode_columns = [row[1] for row in cursor.fetchall()]
+                event_type VARCHAR(50) NOT NULL,
-            
+                processed_at TIMESTAMP NOT NULL,
-            # Add missing columns to movies table
+                details TEXT
-            if 'path' not in movie_columns:
+            )
-                cursor.execute("ALTER TABLE movies ADD COLUMN path TEXT")
+        """)
-                cursor.execute("UPDATE movies SET path = '/unknown/path/' || imdb_id WHERE path IS NULL")
-            
-            if 'has_video_file' not in movie_columns:
-                cursor.execute("ALTER TABLE movies ADD COLUMN has_video_file BOOLEAN DEFAULT FALSE")
-            
-            if 'last_updated' not in movie_columns:
-                cursor.execute("ALTER TABLE movies ADD COLUMN last_updated TEXT")
-                cursor.execute("UPDATE movies SET last_updated = datetime('now') WHERE last_updated IS NULL")
-            
-            # Add missing columns to episodes table  
-            if 'has_video_file' not in episode_columns:
-                cursor.execute("ALTER TABLE episodes ADD COLUMN has_video_file BOOLEAN DEFAULT FALSE")
-            
-            if 'last_updated' not in episode_columns:
-                cursor.execute("ALTER TABLE episodes ADD COLUMN last_updated TEXT")
-                cursor.execute("UPDATE episodes SET last_updated = datetime('now') WHERE last_updated IS NULL")
-            
-            # Create indexes
-            cursor.execute("CREATE INDEX IF NOT EXISTS idx_episodes_imdb ON episodes(imdb_id)")
-            cursor.execute("CREATE INDEX IF NOT EXISTS idx_episodes_video ON episodes(has_video_file)")
-            cursor.execute("CREATE INDEX IF NOT EXISTS idx_movies_video ON movies(has_video_file)")
-            cursor.execute("CREATE INDEX IF NOT EXISTS idx_history_imdb ON processing_history(imdb_id)")
         
+        # Create indexes for PostgreSQL
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_episodes_imdb ON episodes(imdb_id)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_episodes_video ON episodes(has_video_file)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_movies_video ON movies(has_video_file)")
+        cursor.execute("CREATE INDEX IF NOT EXISTS idx_history_imdb ON processing_history(imdb_id)")
     def upsert_series(self, imdb_id: str, path: str, metadata: Optional[Dict] = None):
         """Insert or update series record"""
         with self.get_connection() as conn:
             cursor = conn.cursor()
+            timestamp = datetime.utcnow()
+            
             cursor.execute("""
-                INSERT OR REPLACE INTO series (imdb_id, path, last_updated, metadata)
+                INSERT INTO series (imdb_id, path, last_updated, metadata)
-                VALUES (?, ?, ?, ?)
+                VALUES (%s, %s, %s, %s)
-            """, (imdb_id, path, datetime.utcnow().isoformat(), json.dumps(metadata) if metadata else None))
+                ON CONFLICT (imdb_id) DO UPDATE SET
+                    path = EXCLUDED.path,
+                    last_updated = EXCLUDED.last_updated,
+                    metadata = EXCLUDED.metadata
+            """, (imdb_id, path, timestamp, json.dumps(metadata) if metadata else None))
     
     def upsert_episode_date(self, imdb_id: str, season: int, episode: int, 
                            aired: Optional[str], dateadded: Optional[str], 
@@ -145,30 +154,34 @@ class NFOGuardDatabase:
         """Insert or update episode date record"""
         with self.get_connection() as conn:
             cursor = conn.cursor()
+            timestamp = datetime.utcnow()
+            
             cursor.execute("""
-                INSERT OR REPLACE INTO episodes 
+                INSERT INTO episodes 
                 (imdb_id, season, episode, aired, dateadded, source, has_video_file, last_updated)
-                VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+                VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
-            """, (imdb_id, season, episode, aired, dateadded, source, has_video_file, datetime.utcnow().isoformat()))
+                ON CONFLICT (imdb_id, season, episode) DO UPDATE SET
+                    aired = EXCLUDED.aired,
+                    dateadded = EXCLUDED.dateadded,
+                    source = EXCLUDED.source,
+                    has_video_file = EXCLUDED.has_video_file,
+                    last_updated = EXCLUDED.last_updated
+            """, (imdb_id, season, episode, aired, dateadded, source, has_video_file, timestamp))
+            print(f"🔍 DEBUG: PostgreSQL upsert executed for {imdb_id} S{season:02d}E{episode:02d}, rows affected: {cursor.rowcount}")
     
     def upsert_movie(self, imdb_id: str, path: str):
         """Insert or update movie record"""
         with self.get_connection() as conn:
             cursor = conn.cursor()
-            try:
+            timestamp = datetime.utcnow()
-                cursor.execute("""
+            
-                    INSERT OR REPLACE INTO movies (imdb_id, path, last_updated)
+            cursor.execute("""
-                    VALUES (?, ?, ?)
+                INSERT INTO movies (imdb_id, path, last_updated)
-                """, (imdb_id, path, datetime.utcnow().isoformat()))
+                VALUES (%s, %s, %s)
-            except sqlite3.OperationalError as e:
+                ON CONFLICT (imdb_id) DO UPDATE SET
-                if "no column named path" in str(e):
+                    path = EXCLUDED.path,
-                    # Fallback for databases without path column - just insert imdb_id
+                    last_updated = EXCLUDED.last_updated
-                    cursor.execute("""
+            """, (imdb_id, path, timestamp))
-                        INSERT OR REPLACE INTO movies (imdb_id, last_updated)
-                        VALUES (?, ?)
-                    """, (imdb_id, datetime.utcnow().isoformat()))
-                else:
-                    raise
     
     def upsert_movie_dates(self, imdb_id: str, released: Optional[str], 
                           dateadded: Optional[str], source: str, has_video_file: bool = False):
@@ -176,18 +189,21 @@ class NFOGuardDatabase:
         print(f"🔍 DATABASE UPSERT: imdb_id={imdb_id}, dateadded={dateadded}, source={source}")
         with self.get_connection() as conn:
             cursor = conn.cursor()
-            # Use INSERT OR REPLACE to ensure we always update the dates properly
+            timestamp = datetime.utcnow()
+            
             cursor.execute("""
-                INSERT OR REPLACE INTO movies (imdb_id, path, released, dateadded, source, has_video_file, last_updated)
+                INSERT INTO movies (imdb_id, path, released, dateadded, source, has_video_file, last_updated)
-                VALUES (
+                VALUES (%s, COALESCE((SELECT path FROM movies WHERE imdb_id = %s), 'unknown'), %s, %s, %s, %s, %s)
-                    ?, 
+                ON CONFLICT (imdb_id) DO UPDATE SET
-                    COALESCE((SELECT path FROM movies WHERE imdb_id = ?), 'unknown'),
+                    released = EXCLUDED.released,
-                    ?, ?, ?, ?, ?
+                    dateadded = EXCLUDED.dateadded,
-                )
+                    source = EXCLUDED.source,
-            """, (imdb_id, imdb_id, released, dateadded, source, has_video_file, datetime.utcnow().isoformat()))
+                    has_video_file = EXCLUDED.has_video_file,
+                    last_updated = EXCLUDED.last_updated
+            """, (imdb_id, imdb_id, released, dateadded, source, has_video_file, timestamp))
             
             # Debug: Check what was actually saved
-            cursor.execute("SELECT dateadded, source FROM movies WHERE imdb_id = ?", (imdb_id,))
+            cursor.execute("SELECT dateadded, source FROM movies WHERE imdb_id = %s", (imdb_id,))
             result = cursor.fetchone()
             print(f"🔍 DATABASE VERIFY: After upsert, found dateadded={result[0] if result else 'NOT_FOUND'}, source={result[1] if result else 'NOT_FOUND'}")
     
@@ -196,9 +212,8 @@ class NFOGuardDatabase:
         with self.get_connection() as conn:
             cursor = conn.cursor()
             
-            query = "SELECT * FROM episodes WHERE imdb_id = ?"
+            query = "SELECT * FROM episodes WHERE imdb_id = %s"
             params = [imdb_id]
-            
             if has_video_file_only:
                 query += " AND has_video_file = TRUE"
             
@@ -211,7 +226,7 @@ class NFOGuardDatabase:
             cursor = conn.cursor()
             cursor.execute("""
                 SELECT * FROM episodes 
-                WHERE imdb_id = ? AND season = ? AND episode = ?
+                WHERE imdb_id = %s AND season = %s AND episode = %s
             """, (imdb_id, season, episode))
             
             row = cursor.fetchone()
@@ -221,7 +236,7 @@ class NFOGuardDatabase:
         """Get movie date record"""
         with self.get_connection() as conn:
             cursor = conn.cursor()
-            cursor.execute("SELECT * FROM movies WHERE imdb_id = ?", (imdb_id,))
+            cursor.execute("SELECT * FROM movies WHERE imdb_id = %s", (imdb_id,))
             
             row = cursor.fetchone()
             return dict(row) if row else None
@@ -232,36 +247,41 @@ class NFOGuardDatabase:
             cursor = conn.cursor()
             cursor.execute("""
                 INSERT INTO processing_history (imdb_id, media_type, event_type, processed_at, details)
-                VALUES (?, ?, ?, ?, ?)
+                VALUES (%s, %s, %s, %s, %s)
             """, (imdb_id, media_type, event_type, datetime.utcnow().isoformat(), 
                   json.dumps(details) if details else None))
     
     def get_stats(self) -> Dict[str, Any]:
         """Get database statistics"""
         with self.get_connection() as conn:
-            cursor = conn.cursor()
+            cursor = conn.cursor()  # Regular cursor for PostgreSQL
             
             # Series stats
             cursor.execute("SELECT COUNT(*) FROM series")
-            series_count = cursor.fetchone()[0]
+            series_count = self._get_first_value(cursor.fetchone())
             
             # Episode stats
             cursor.execute("SELECT COUNT(*) FROM episodes")
-            episodes_total = cursor.fetchone()[0]
+            episodes_total = self._get_first_value(cursor.fetchone())
             
             cursor.execute("SELECT COUNT(*) FROM episodes WHERE has_video_file = TRUE")
-            episodes_with_video = cursor.fetchone()[0]
+            episodes_with_video = self._get_first_value(cursor.fetchone())
             
             # Movie stats
             cursor.execute("SELECT COUNT(*) FROM movies")
-            movies_total = cursor.fetchone()[0]
+            movies_total = self._get_first_value(cursor.fetchone())
             
             cursor.execute("SELECT COUNT(*) FROM movies WHERE has_video_file = TRUE")
-            movies_with_video = cursor.fetchone()[0]
+            movies_with_video = self._get_first_value(cursor.fetchone())
             
             # Processing history
             cursor.execute("SELECT COUNT(*) FROM processing_history")
-            history_count = cursor.fetchone()[0]
+            history_count = self._get_first_value(cursor.fetchone())
+            
+            # Database size calculation for PostgreSQL
+            cursor.execute("SELECT pg_database_size(%s)", (self.db_name,))
+            db_size_bytes = self._get_first_value(cursor.fetchone())
+            db_size_mb = round(db_size_bytes / 1024 / 1024, 2) if db_size_bytes else 0
             
             return {
                 "series_count": series_count,
@@ -270,5 +290,132 @@ class NFOGuardDatabase:
                 "movies_total": movies_total,
                 "movies_with_video": movies_with_video,
                 "processing_history_count": history_count,
-                "database_size_mb": round(self.db_path.stat().st_size / 1024 / 1024, 2)
+                "database_size_mb": db_size_mb,
+                "database_type": "postgresql"
             }
+    
+    def delete_episode(self, imdb_id: str, season: int, episode: int) -> bool:
+        """
+        Delete a specific episode from the database
+        
+        Args:
+            imdb_id: Series IMDb ID
+            season: Season number
+            episode: Episode number
+            
+        Returns:
+            True if episode was deleted, False if not found
+        """
+        with self.get_connection() as conn:
+            cursor = conn.cursor()
+            
+            cursor.execute("""
+                DELETE FROM episodes 
+                WHERE imdb_id = %s AND season = %s AND episode = %s
+            """, (imdb_id, season, episode))
+            
+            deleted_count = cursor.rowcount
+            conn.commit()
+            
+            return deleted_count > 0
+    
+    def delete_series_episodes(self, imdb_id: str) -> int:
+        """
+        Delete all episodes for a series from the database
+        
+        Args:
+            imdb_id: Series IMDb ID
+            
+        Returns:
+            Number of episodes deleted
+        """
+        with self.get_connection() as conn:
+            cursor = conn.cursor()
+            
+            cursor.execute("""
+                DELETE FROM episodes 
+                WHERE imdb_id = %s
+            """, (imdb_id,))
+            
+            deleted_count = cursor.rowcount
+            conn.commit()
+            
+            return deleted_count
+    
+    def delete_orphaned_episodes(self) -> List[Dict]:
+        """
+        Find and delete episodes that don't have corresponding video files on disk
+        This requires checking filesystem for each episode, so use carefully
+        
+        Returns:
+            List of deleted episodes with their details
+        """
+        from utils.file_utils import find_episodes_on_disk
+        from pathlib import Path
+        
+        deleted_episodes = []
+        
+        with self.get_connection() as conn:
+            cursor = conn.cursor()
+            
+            # Get all series with their paths
+            cursor.execute("""
+                SELECT DISTINCT imdb_id, path FROM series
+            """)
+            
+            series_list = cursor.fetchall()
+            
+            for series in series_list:
+                imdb_id = series['imdb_id']
+                series_path = Path(series['path'])
+                
+                if not series_path.exists():
+                    continue
+                
+                # Get episodes on disk
+                disk_episodes = find_episodes_on_disk(series_path)
+                disk_episode_keys = set(disk_episodes.keys())
+                
+                # Get episodes in database
+                cursor.execute("""
+                    SELECT season, episode, dateadded, source 
+                    FROM episodes 
+                    WHERE imdb_id = %s
+                """, (imdb_id,))
+                
+                db_episodes = cursor.fetchall()
+                
+                # Find orphaned episodes (in DB but not on disk)
+                for db_episode in db_episodes:
+                    season = db_episode['season']
+                    episode = db_episode['episode']
+                    episode_key = (season, episode)
+                    
+                    if episode_key not in disk_episode_keys:
+                        # Episode is orphaned - delete it
+                        cursor.execute("""
+                            DELETE FROM episodes 
+                            WHERE imdb_id = %s AND season = %s AND episode = %s
+                        """, (imdb_id, season, episode))
+                        
+                        deleted_episodes.append({
+                            'imdb_id': imdb_id,
+                            'season': season,
+                            'episode': episode,
+                            'dateadded': db_episode['dateadded'],
+                            'source': db_episode['source'],
+                            'series_path': str(series_path)
+                        })
+            
+            conn.commit()
+            
+        return deleted_episodes
+    
+    def close(self):
+        """Close all database connections"""
+        if hasattr(self._local, 'connection'):
+            try:
+                self._local.connection.close()
+                delattr(self._local, 'connection')
+            except Exception:
+                pass  # Connection may already be closed

core/nfo_manager.py

@@ -97,12 +97,9 @@ class NFOManager:
     
     def find_movie_imdb_id(self, movie_dir: Path) -> Optional[str]:
         """Find IMDb ID from directory name, filenames, or NFO file"""
-        print(f"🔍 Searching for IMDb ID in: {movie_dir.name}")
-        
         # First try directory name
         imdb_id = self.parse_imdb_from_path(movie_dir)
         if imdb_id:
-            print(f"✅ Found IMDb ID in directory name: {imdb_id}")
             return imdb_id
             
         # Try all files in the directory for IMDb ID patterns
@@ -110,20 +107,36 @@ class NFOManager:
             if file_path.is_file():
                 imdb_id = self.parse_imdb_from_path(file_path)
                 if imdb_id:
-                    print(f"✅ Found IMDb ID in filename: {imdb_id} from {file_path.name}")
                     return imdb_id
                     
         # Finally, try NFO file content (including TMDB fallback)
         nfo_path = movie_dir / "movie.nfo"
         imdb_id = self.parse_imdb_from_nfo(nfo_path)
         if imdb_id:
-            if imdb_id.startswith("tmdb-"):
-                print(f"✅ Found TMDB ID in NFO file: {imdb_id} from {nfo_path} (fallback mode)")
-            else:
-                print(f"✅ Found IMDb ID in NFO file: {imdb_id} from {nfo_path}")
             return imdb_id
             
-        print(f"❌ No IMDb or TMDB ID found in directory, filenames, or NFO for: {movie_dir.name}")
+        return None
+    
+    def find_series_imdb_id(self, series_dir: Path) -> Optional[str]:
+        """Find IMDb ID from TV series directory name, filenames, or tvshow.nfo file"""
+        # First try directory name
+        imdb_id = self.parse_imdb_from_path(series_dir)
+        if imdb_id:
+            return imdb_id
+            
+        # Try all files in the directory for IMDb ID patterns
+        for file_path in series_dir.iterdir():
+            if file_path.is_file():
+                imdb_id = self.parse_imdb_from_path(file_path)
+                if imdb_id:
+                    return imdb_id
+                    
+        # Finally, try tvshow.nfo file content
+        nfo_path = series_dir / "tvshow.nfo"
+        imdb_id = self.parse_imdb_from_nfo(nfo_path)
+        if imdb_id:
+            return imdb_id
+            
         return None
     
     def extract_nfoguard_dates_from_nfo(self, nfo_path: Path) -> Optional[Dict[str, str]]:
@@ -138,21 +151,36 @@ class NFOManager:
             # Look for NFOGuard fields
             dateadded_elem = root.find('.//dateadded')
             premiered_elem = root.find('.//premiered')
+            aired_elem = root.find('.//aired')  # For TV episodes
             lockdata_elem = root.find('.//lockdata')
             
-            # Only consider it NFOGuard-managed if it has dateadded and lockdata
+            # Consider it NFOGuard-managed if it has lockdata=true (with or without dateadded)
-            if (dateadded_elem is not None and dateadded_elem.text and 
+            if lockdata_elem is not None and lockdata_elem.text == "true":
-                lockdata_elem is not None and lockdata_elem.text == "true"):
+                # Extract original source from NFOGuard comment, default to nfo_file_existing
+                source = "nfo_file_existing"
+                
+                # Parse XML content to find NFOGuard comment with source
+                nfo_content = nfo_path.read_text(encoding='utf-8')
+                import re
+                source_match = re.search(r'<!--\s*NFOGuard\s*-\s*Source:\s*([^-]+?)\s*-->', nfo_content)
+                if source_match:
+                    source = source_match.group(1).strip()
+                    print(f"🔍 Extracted original source from NFO comment: {source}")
                 
                 result = {
-                    "dateadded": dateadded_elem.text.strip(),
+                    "source": source
-                    "source": "nfo_file_existing"
                 }
                 
+                if dateadded_elem is not None and dateadded_elem.text:
+                    result["dateadded"] = dateadded_elem.text.strip()
+                
                 if premiered_elem is not None and premiered_elem.text:
                     result["released"] = premiered_elem.text.strip()
                 
-                print(f"✅ Found NFOGuard data in NFO: dateadded={result['dateadded']}, released={result.get('released', 'None')}")
+                if aired_elem is not None and aired_elem.text:
+                    result["aired"] = aired_elem.text.strip()
+                    
+                print(f"✅ Found NFOGuard data in NFO: dateadded={result.get('dateadded', 'None')}, source={source}, released={result.get('released', 'None')}, aired={result.get('aired', 'None')}")
                 return result
                 
         except (ET.ParseError, Exception) as e:
@@ -178,19 +206,30 @@ class NFOManager:
             aired_elem = root.find('.//aired')
             lockdata_elem = root.find('.//lockdata')
             
-            # Only consider it NFOGuard-managed if it has dateadded and lockdata
+            # Consider it NFOGuard-managed if it has lockdata=true (with or without dateadded)
-            if (dateadded_elem is not None and dateadded_elem.text and 
+            if lockdata_elem is not None and lockdata_elem.text == "true":
-                lockdata_elem is not None and lockdata_elem.text == "true"):
+                # Extract original source from NFOGuard comment, default to episode_nfo_existing
+                source = "episode_nfo_existing"
+                
+                # Parse XML content to find NFOGuard comment with source
+                nfo_content = nfo_path.read_text(encoding='utf-8')
+                import re
+                source_match = re.search(r'<!--\s*NFOGuard\s*-\s*Source:\s*([^-]+?)\s*-->', nfo_content)
+                if source_match:
+                    source = source_match.group(1).strip()
+                    print(f"🔍 Extracted original source from episode NFO comment: {source}")
                 
                 result = {
-                    "dateadded": dateadded_elem.text.strip(),
+                    "source": source
-                    "source": "episode_nfo_existing"
                 }
                 
+                if dateadded_elem is not None and dateadded_elem.text:
+                    result["dateadded"] = dateadded_elem.text.strip()
+                
                 if aired_elem is not None and aired_elem.text:
                     result["aired"] = aired_elem.text.strip()
                     
-                print(f"✅ Found NFOGuard data in episode NFO S{season_num:02d}E{episode_num:02d}: dateadded={result['dateadded']}, aired={result.get('aired', 'None')}")
+                print(f"✅ Found NFOGuard data in episode NFO S{season_num:02d}E{episode_num:02d}: dateadded={result.get('dateadded', 'None')}, source={source}, aired={result.get('aired', 'None')}")
                 return result
                 
         except (ET.ParseError, Exception) as e:
@@ -234,6 +273,8 @@ class NFOManager:
         nfo_path = movie_dir / "movie.nfo"
         
         print(f"🔍 create_movie_nfo called: imdb_id={imdb_id}, dateadded={dateadded}, released={released}, source={source}")
+        print(f"🔍 NFO path: {nfo_path}")
+        print(f"🔍 NFO exists: {nfo_path.exists()}")
         
         try:
             # Try to load existing NFO file
@@ -246,24 +287,45 @@ class NFOManager:
                     if movie.tag != "movie":
                         raise ValueError("Root element is not <movie>")
                     
-                    # Remove existing NFOGuard-managed elements to avoid duplicates
+                    # Only remove elements that are clearly NFOGuard-managed
-                    # These will be re-added at the very bottom
+                    # Look for elements that have NFOGuard characteristics
-                    nfoguard_fields = ["dateadded", "lockdata", "premiered", "year"]
+                    elements_to_remove = []
-                    for tag in nfoguard_fields:
-                        existing = movie.find(tag)
-                        if existing is not None:
-                            # Store the value before removing (for premiered/year)
-                            if tag == "premiered" and not released:
-                                print(f"🔍 Preserving existing premiered date: {existing.text} (released was None)")
-                                released = existing.text  # Preserve existing premiered date
-                            elif tag == "premiered":
-                                print(f"🔍 NOT preserving premiered date: existing={existing.text}, released={released}")
-                            movie.remove(existing)
                     
-                    # Remove ALL existing uniqueid with type="imdb" regardless of attributes
+                    # Remove lockdata=true (this is definitely NFOGuard)
-                    # We'll add a clean one at the bottom
+                    for lockdata in movie.findall("lockdata"):
+                        if lockdata.text == "true":
+                            elements_to_remove.append(lockdata)
+                    
+                    # Remove IMDb uniqueids that we manage
                     for uniqueid in movie.findall("uniqueid[@type='imdb']"):
-                        movie.remove(uniqueid)
+                        elements_to_remove.append(uniqueid)
+                    
+                    # For dateadded/premiered/year, only remove if they appear to be NFOGuard-managed
+                    # (i.e., if lockdata=true exists, these are likely ours)
+                    has_nfoguard_lockdata = any(ld.text == "true" for ld in movie.findall("lockdata"))
+                    
+                    if has_nfoguard_lockdata:
+                        # This NFO was managed by NFOGuard, safe to remove our fields
+                        for tag in ["dateadded", "premiered", "year"]:
+                            existing = movie.find(tag)
+                            if existing is not None:
+                                # Store the value before removing (for premiered/year)
+                                if tag == "premiered" and not released:
+                                    print(f"🔍 Preserving existing premiered date: {existing.text}")
+                                    released = existing.text
+                                elements_to_remove.append(existing)
+                    else:
+                        # No NFOGuard lockdata found, be more conservative
+                        # Only remove dateadded if it looks like NFOGuard format (ISO timestamp)
+                        dateadded_elem = movie.find("dateadded")
+                        if dateadded_elem is not None and dateadded_elem.text:
+                            # NFOGuard uses ISO format like "2025-10-12 16:26:02"
+                            if re.match(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}', dateadded_elem.text.strip()):
+                                elements_to_remove.append(dateadded_elem)
+                    
+                    # Remove all identified elements
+                    for elem in elements_to_remove:
+                        movie.remove(elem)
                         
                 except (ET.ParseError, ValueError) as e:
                     print(f"⚠️  Corrupted NFO detected: {nfo_path} - {str(e)[:100]}...")
@@ -273,23 +335,24 @@ class NFOManager:
                 # Create new NFO structure
                 movie = ET.Element("movie")
             
-            # Now append ALL NFOGuard fields at the VERY END of the file
+            # Create all NFOGuard elements first, then append them in correct order
-            # Create elements and explicitly append them to ensure they're at the bottom
+            # This ensures they appear as a group at the bottom of the file
+            nfoguard_elements = []
             
             # Add NFOGuard comment marker as the first of our additions
             nfoguard_comment = ET.Comment(f" NFOGuard - Source: {source} ")
-            movie.append(nfoguard_comment)
+            nfoguard_elements.append(nfoguard_comment)
             
-            # Add IMDb uniqueid at the end (after all existing content)
+            # Add IMDb uniqueid 
             uniqueid = ET.Element("uniqueid", type="imdb", default="true")
             uniqueid.text = imdb_id
-            movie.append(uniqueid)
+            nfoguard_elements.append(uniqueid)
             
-            # Add premiered date at the bottom if we have it
+            # Add premiered date if we have it
             if released:
                 premiered_elem = ET.Element("premiered")
                 premiered_elem.text = released[:10] if len(released) >= 10 else released
-                movie.append(premiered_elem)
+                nfoguard_elements.append(premiered_elem)
                 
                 # Extract year from premiered date for consistency
                 try:
@@ -297,22 +360,32 @@ class NFOManager:
                     if year_value and year_value.isdigit():
                         year_elem = ET.Element("year")
                         year_elem.text = year_value
-                        movie.append(year_elem)
+                        nfoguard_elements.append(year_elem)
                 except:
                     pass  # Skip year if we can't extract it
             
-            # Add dateadded at the end - THIS IS CRITICAL FOR EMBY PLUGIN
+            # Add dateadded - THIS IS CRITICAL FOR EMBY PLUGIN
+            print(f"🔍 About to add dateadded: {dateadded} (type: {type(dateadded)})")
             if dateadded:
                 dateadded_elem = ET.Element("dateadded")
                 dateadded_elem.text = dateadded
-                movie.append(dateadded_elem)
+                nfoguard_elements.append(dateadded_elem)
-                print(f"✅ Added dateadded to NFO: {dateadded}")
+                print(f"✅ Adding dateadded to NFO: {dateadded}")
+            else:
+                print(f"❌ dateadded is empty/None, not adding to NFO")
             
             # Add lockdata at the very end
             if lock_metadata:
                 lockdata = ET.Element("lockdata")
                 lockdata.text = "true"
-                movie.append(lockdata)
+                nfoguard_elements.append(lockdata)
+            
+            # Now append all NFOGuard elements to the movie in one batch
+            # This ensures they appear as a contiguous block at the bottom
+            for elem in nfoguard_elements:
+                movie.append(elem)
+            
+            print(f"✅ Added {len(nfoguard_elements)} NFOGuard elements to bottom of NFO")
             
             # Write file with proper formatting
             tree = ET.ElementTree(movie)
@@ -590,7 +663,7 @@ class NFOManager:
                         if preserved_fields:
                             print(f"   🔍 Content preserved after cleanup: {', '.join(preserved_fields)}")
                         else:
-                            print(f"   ⚠️  No content fields found after cleanup!")
+                            print(f"   ℹ️  NFO contains only NFOGuard metadata (no additional content fields)")
                         
                 except (ET.ParseError, ValueError) as e:
                     print(f"⚠️  Corrupted episode NFO detected: {nfo_path} - {str(e)[:100]}...")
@@ -626,11 +699,14 @@ class NFOManager:
             # Dates at the end
             if aired:
                 aired_elem = ET.SubElement(episode, "aired")
-                aired_elem.text = aired[:10] if len(aired) >= 10 else aired
+                # Convert datetime objects to strings
+                aired_str = str(aired)
+                aired_elem.text = aired_str[:10] if len(aired_str) >= 10 else aired_str
             
             if dateadded:
                 dateadded_elem = ET.SubElement(episode, "dateadded")
-                dateadded_elem.text = dateadded
+                # Convert datetime objects to strings
+                dateadded_elem.text = str(dateadded)
             
             # Add lockdata at the very end
             if lock_metadata:
@@ -662,16 +738,25 @@ class NFOManager:
         except Exception as e:
             print(f"❌ Error creating/updating episode NFO {nfo_path}: {e}")
     
-    def set_file_mtime(self, file_path: Path, iso_timestamp: str) -> None:
+    def set_file_mtime(self, file_path: Path, iso_timestamp) -> None:
         """Set file modification time to match import date"""
         try:
+            # Convert datetime objects to strings first
+            if hasattr(iso_timestamp, 'isoformat'):
+                iso_timestamp = iso_timestamp.isoformat()
+            elif not isinstance(iso_timestamp, str):
+                iso_timestamp = str(iso_timestamp)
+            
             # Parse ISO timestamp
             if iso_timestamp.endswith('Z'):
                 dt = datetime.fromisoformat(iso_timestamp.replace('Z', '+00:00'))
             elif '+' in iso_timestamp or 'T' in iso_timestamp:
                 dt = datetime.fromisoformat(iso_timestamp)
+            elif ' ' in iso_timestamp:
+                # Handle space-separated datetime format (e.g., "2025-10-16 20:31:22")
+                dt = datetime.fromisoformat(iso_timestamp.replace(' ', 'T'))
             else:
-                # Assume it's already a simple date
+                # Assume it's a simple date (e.g., "2025-10-16")
                 dt = datetime.fromisoformat(iso_timestamp + 'T00:00:00')
             
             # Convert to timestamp

debug_movie.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+"""
+Debug script to check specific movie data in NFOGuard database
+"""
+import os
+import sys
+from pathlib import Path
+
+# Add the project root to the path
+sys.path.insert(0, str(Path(__file__).parent))
+
+from core.database import NFOGuardDatabase
+
+def debug_movie(imdb_id: str):
+    """Debug a specific movie's data"""
+    print(f"🔍 DEBUG MOVIE: {imdb_id}")
+    print("=" * 50)
+    
+    # Initialize database
+    db = NFOGuardDatabase()
+    
+    # Get movie data
+    movie = db.get_movie_dates(imdb_id)
+    if not movie:
+        print(f"❌ Movie {imdb_id} not found in database")
+        return
+    
+    print("📊 RAW MOVIE DATA:")
+    for key, value in movie.items():
+        print(f"   {key}: {repr(value)}")
+    
+    print("\n🎬 FORMATTED MOVIE DATA:")
+    print(f"   Title/Path: {movie.get('path', 'Unknown')}")
+    print(f"   Released: {movie.get('released', 'None')}")
+    print(f"   Date Added: {movie.get('dateadded', 'None')}")
+    print(f"   Source: {movie.get('source', 'None')}")
+    print(f"   Has Video: {movie.get('has_video_file', False)}")
+    print(f"   Last Updated: {movie.get('last_updated', 'None')}")
+    
+    # Check if released date is valid
+    released = movie.get('released')
+    if released and released.strip():
+        try:
+            from datetime import datetime
+            test_date = f"{released}T00:00:00"
+            parsed = datetime.fromisoformat(test_date.replace('Z', '+00:00'))
+            print(f"\n✅ Released date is valid: {parsed}")
+        except Exception as e:
+            print(f"\n❌ Released date is INVALID: {e}")
+    else:
+        print(f"\n⚠️ Released date is empty or None")
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print("Usage: python debug_movie.py <imdb_id>")
+        sys.exit(1)
+    
+    imdb_id = sys.argv[1]
+    if not imdb_id.startswith('tt'):
+        imdb_id = f'tt{imdb_id}'
+    
+    debug_movie(imdb_id)

docker-compose.example

@@ -16,6 +16,10 @@ services:
     # Restart policy
     restart: unless-stopped
     
+    # Graceful shutdown configuration
+    stop_grace_period: 30s
+    stop_signal: SIGTERM
+    
     # Environment files
     env_file:
       - .env

main.py

@@ -6,6 +6,7 @@ Modular architecture with webhook processing and intelligent date handling
 import os
 import sys
 import signal
+import asyncio
 from pathlib import Path
 from datetime import datetime, timezone
 
@@ -34,6 +35,8 @@ from webhooks.webhook_batcher import WebhookBatcher
 # Import API routes
 from api.routes import register_routes
 
+# Global shutdown event for graceful shutdown coordination
+shutdown_event = asyncio.Event()
 
 def get_version() -> str:
     """Get application version"""
@@ -86,7 +89,7 @@ def initialize_components():
     start_time = datetime.now(timezone.utc)
     
     # Initialize core components
-    db = NFOGuardDatabase(config.db_path)
+    db = NFOGuardDatabase(config=config)
     nfo_manager = NFOManager(config.manager_brand, config.debug)
     path_mapper = PathMapper(config)
     
@@ -94,8 +97,8 @@ def initialize_components():
     tv_processor = TVProcessor(db, nfo_manager, path_mapper)
     movie_processor = MovieProcessor(db, nfo_manager, path_mapper)
     
-    # Initialize webhook batcher
+    # Initialize webhook batcher with nfo_manager for comprehensive IMDb detection
-    batcher = WebhookBatcher()
+    batcher = WebhookBatcher(nfo_manager)
     batcher.set_processors(tv_processor, movie_processor)
     
     return {
@@ -107,13 +110,39 @@ def initialize_components():
         "batcher": batcher,
         "start_time": start_time,
         "config": config,
-        "version": get_version()
+        "version": get_version(),
+        "shutdown_event": shutdown_event
     }
 
 
 def signal_handler(signum, frame):
     """Handle shutdown signals gracefully"""
     _log("INFO", f"Received signal {signum}, shutting down gracefully...")
+    
+    # Set shutdown event to notify background tasks
+    shutdown_event.set()
+    
+    # Get the global dependencies if they exist
+    if hasattr(signal_handler, 'dependencies') and signal_handler.dependencies:
+        deps = signal_handler.dependencies
+        
+        # Shutdown webhook batcher cleanly
+        if 'batcher' in deps:
+            try:
+                _log("INFO", "Shutting down webhook batcher...")
+                deps['batcher'].shutdown()
+            except Exception as e:
+                _log("WARNING", f"Error during batcher shutdown: {e}")
+        
+        # Close database connection
+        if 'db' in deps:
+            try:
+                _log("INFO", "Closing database connection...")
+                deps['db'].close()
+            except Exception as e:
+                _log("WARNING", f"Error closing database: {e}")
+    
+    _log("INFO", "Graceful shutdown complete")
     sys.exit(0)
 
 
@@ -129,7 +158,11 @@ def main():
     _log("INFO", f"Version: {version}")
     _log("INFO", f"TV paths: {[str(p) for p in config.tv_paths]}")
     _log("INFO", f"Movie paths: {[str(p) for p in config.movie_paths]}")
-    _log("INFO", f"Database: {config.db_path}")
+    if config.db_type == "postgresql":
+        _log("INFO", f"Database: PostgreSQL at {config.db_host}:{config.db_port}/{config.db_name}")
+        _log("INFO", f"Database user: {config.db_user}")
+    else:
+        _log("INFO", f"Database: {config.db_path}")
     _log("INFO", f"Config: manage_nfo={config.manage_nfo}, fix_mtimes={config.fix_dir_mtimes}")
     _log("INFO", f"Movie priority: {config.movie_priority}")
     
@@ -139,6 +172,9 @@ def main():
     # Initialize components
     dependencies = initialize_components()
     
+    # Store dependencies globally for signal handler access
+    signal_handler.dependencies = dependencies
+    
     # Register routes
     register_routes(app, dependencies)
     
@@ -147,13 +183,32 @@ def main():
             app,
             host="0.0.0.0", 
             port=int(os.environ.get("PORT", "8080")),
-            reload=False
+            reload=False,
+            access_log=False,  # Reduce logging overhead
+            server_header=False,  # Reduce response overhead
+            timeout_graceful_shutdown=15  # Give more time for graceful shutdown
         )
     except KeyboardInterrupt:
         _log("INFO", "NFOGuard stopped by user")
     except Exception as e:
         _log("ERROR", f"NFOGuard crashed: {e}")
         sys.exit(1)
+    finally:
+        # Ensure cleanup happens even if uvicorn doesn't trigger signal handler
+        if hasattr(signal_handler, 'dependencies') and signal_handler.dependencies:
+            deps = signal_handler.dependencies
+            
+            if 'batcher' in deps:
+                try:
+                    deps['batcher'].shutdown()
+                except Exception:
+                    pass
+            
+            if 'db' in deps:
+                try:
+                    deps['db'].close()
+                except Exception:
+                    pass
 
 
 if __name__ == "__main__":

monitoring/health.py

@@ -0,0 +1,474 @@
+"""
+Health Check System for NFOGuard
+Provides health and readiness endpoints for monitoring and orchestration
+"""
+import time
+import asyncio
+from typing import Dict, Any, List, Optional
+from dataclasses import dataclass
+from enum import Enum
+from pathlib import Path
+
+from config.runtime_validator import RuntimeValidator, HealthCheckResult
+from monitoring.metrics import metrics
+
+
+class HealthStatus(Enum):
+    """Health check status levels"""
+    HEALTHY = "healthy"
+    DEGRADED = "degraded"
+    UNHEALTHY = "unhealthy"
+
+
+@dataclass
+class HealthCheck:
+    """Individual health check result"""
+    name: str
+    status: HealthStatus
+    message: str
+    duration_ms: float
+    details: Dict[str, Any] = None
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "name": self.name,
+            "status": self.status.value,
+            "message": self.message,
+            "duration_ms": round(self.duration_ms, 2),
+            "details": self.details or {}
+        }
+
+
+@dataclass
+class OverallHealth:
+    """Overall system health status"""
+    status: HealthStatus
+    checks: List[HealthCheck]
+    timestamp: float
+    uptime_seconds: float
+    version: str = "2.0.0"
+    
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "status": self.status.value,
+            "timestamp": self.timestamp,
+            "uptime_seconds": round(self.uptime_seconds, 2),
+            "version": self.version,
+            "checks": [check.to_dict() for check in self.checks],
+            "summary": {
+                "total_checks": len(self.checks),
+                "healthy_checks": len([c for c in self.checks if c.status == HealthStatus.HEALTHY]),
+                "degraded_checks": len([c for c in self.checks if c.status == HealthStatus.DEGRADED]),
+                "unhealthy_checks": len([c for c in self.checks if c.status == HealthStatus.UNHEALTHY])
+            }
+        }
+
+
+class HealthChecker:
+    """Comprehensive health checking system"""
+    
+    def __init__(self):
+        self.start_time = time.time()
+        self._last_health_check = None
+        self._health_check_cache_ttl = 30  # Cache for 30 seconds
+        self._runtime_validator = None
+    
+    def _get_runtime_validator(self):
+        """Get runtime validator instance"""
+        if self._runtime_validator is None:
+            try:
+                from config.settings import config
+                self._runtime_validator = RuntimeValidator(config)
+            except Exception as e:
+                # Create a dummy validator if config fails
+                self._runtime_validator = None
+        return self._runtime_validator
+    
+    async def check_basic_health(self) -> HealthCheck:
+        """Basic health check - always succeeds if service is running"""
+        start_time = time.time()
+        
+        try:
+            # Basic service availability
+            uptime = time.time() - self.start_time
+            
+            if uptime < 30:
+                status = HealthStatus.DEGRADED
+                message = f"Service starting up (uptime: {uptime:.1f}s)"
+            else:
+                status = HealthStatus.HEALTHY
+                message = f"Service running normally (uptime: {uptime:.1f}s)"
+            
+            return HealthCheck(
+                name="basic",
+                status=status,
+                message=message,
+                duration_ms=(time.time() - start_time) * 1000,
+                details={"uptime_seconds": uptime}
+            )
+            
+        except Exception as e:
+            return HealthCheck(
+                name="basic",
+                status=HealthStatus.UNHEALTHY,
+                message=f"Basic health check failed: {e}",
+                duration_ms=(time.time() - start_time) * 1000
+            )
+    
+    async def check_filesystem_health(self) -> HealthCheck:
+        """Check filesystem access for media paths"""
+        start_time = time.time()
+        
+        try:
+            from config.settings import config
+            
+            accessible_paths = 0
+            total_paths = len(config.tv_paths) + len(config.movie_paths)
+            issues = []
+            
+            # Check TV paths
+            for path in config.tv_paths:
+                try:
+                    if path.exists() and path.is_dir():
+                        # Try to read directory
+                        list(path.iterdir())
+                        accessible_paths += 1
+                    else:
+                        issues.append(f"TV path not accessible: {path}")
+                except PermissionError:
+                    issues.append(f"TV path permission denied: {path}")
+                except Exception as e:
+                    issues.append(f"TV path error {path}: {e}")
+            
+            # Check movie paths
+            for path in config.movie_paths:
+                try:
+                    if path.exists() and path.is_dir():
+                        list(path.iterdir())
+                        accessible_paths += 1
+                    else:
+                        issues.append(f"Movie path not accessible: {path}")
+                except PermissionError:
+                    issues.append(f"Movie path permission denied: {path}")
+                except Exception as e:
+                    issues.append(f"Movie path error {path}: {e}")
+            
+            # Determine status
+            if accessible_paths == total_paths:
+                status = HealthStatus.HEALTHY
+                message = f"All {total_paths} media paths accessible"
+            elif accessible_paths > 0:
+                status = HealthStatus.DEGRADED
+                message = f"{accessible_paths}/{total_paths} media paths accessible"
+            else:
+                status = HealthStatus.UNHEALTHY
+                message = "No media paths accessible"
+            
+            return HealthCheck(
+                name="filesystem",
+                status=status,
+                message=message,
+                duration_ms=(time.time() - start_time) * 1000,
+                details={
+                    "accessible_paths": accessible_paths,
+                    "total_paths": total_paths,
+                    "issues": issues[:5]  # Limit to first 5 issues
+                }
+            )
+            
+        except Exception as e:
+            return HealthCheck(
+                name="filesystem",
+                status=HealthStatus.UNHEALTHY,
+                message=f"Filesystem check failed: {e}",
+                duration_ms=(time.time() - start_time) * 1000
+            )
+    
+    async def check_database_health(self) -> HealthCheck:
+        """Check database connectivity and performance"""
+        start_time = time.time()
+        
+        try:
+            import sqlite3
+            from config.settings import config
+            
+            # Test local database
+            db_path = config.db_path
+            
+            def test_db():
+                with sqlite3.connect(str(db_path), timeout=5) as conn:
+                    cursor = conn.cursor()
+                    cursor.execute("SELECT COUNT(*) FROM sqlite_master WHERE type='table'")
+                    table_count = cursor.fetchone()[0]
+                    return table_count
+            
+            # Run database test
+            table_count = await asyncio.get_event_loop().run_in_executor(None, test_db)
+            
+            duration = (time.time() - start_time) * 1000
+            
+            if duration < 100:  # < 100ms is good
+                status = HealthStatus.HEALTHY
+                message = f"Database responsive ({duration:.1f}ms, {table_count} tables)"
+            elif duration < 1000:  # < 1s is acceptable
+                status = HealthStatus.DEGRADED
+                message = f"Database slow ({duration:.1f}ms, {table_count} tables)"
+            else:
+                status = HealthStatus.UNHEALTHY
+                message = f"Database very slow ({duration:.1f}ms)"
+            
+            return HealthCheck(
+                name="database",
+                status=status,
+                message=message,
+                duration_ms=duration,
+                details={
+                    "db_path": str(db_path),
+                    "table_count": table_count,
+                    "response_time_category": "fast" if duration < 100 else "slow" if duration < 1000 else "very_slow"
+                }
+            )
+            
+        except Exception as e:
+            return HealthCheck(
+                name="database",
+                status=HealthStatus.UNHEALTHY,
+                message=f"Database check failed: {e}",
+                duration_ms=(time.time() - start_time) * 1000,
+                details={"error": str(e)}
+            )
+    
+    async def check_external_apis_health(self) -> HealthCheck:
+        """Check external API connectivity"""
+        start_time = time.time()
+        
+        try:
+            import aiohttp
+            from config.settings import config
+            
+            api_results = []
+            apis_tested = 0
+            apis_healthy = 0
+            
+            timeout = aiohttp.ClientTimeout(total=5)
+            
+            async with aiohttp.ClientSession(timeout=timeout) as session:
+                # Test Radarr API if configured
+                if hasattr(config, 'radarr_url') and config.radarr_url:
+                    apis_tested += 1
+                    try:
+                        test_url = f"{config.radarr_url.rstrip('/')}/api/v3/health"
+                        async with session.get(test_url) as response:
+                            if response.status == 200:
+                                apis_healthy += 1
+                                api_results.append({"api": "radarr", "status": "healthy"})
+                            else:
+                                api_results.append({"api": "radarr", "status": f"unhealthy (HTTP {response.status})"})
+                    except Exception as e:
+                        api_results.append({"api": "radarr", "status": f"error: {str(e)[:50]}"})
+                
+                # Test Sonarr API if configured
+                if hasattr(config, 'sonarr_url') and config.sonarr_url:
+                    apis_tested += 1
+                    try:
+                        test_url = f"{config.sonarr_url.rstrip('/')}/api/v3/health"
+                        async with session.get(test_url) as response:
+                            if response.status == 200:
+                                apis_healthy += 1
+                                api_results.append({"api": "sonarr", "status": "healthy"})
+                            else:
+                                api_results.append({"api": "sonarr", "status": f"unhealthy (HTTP {response.status})"})
+                    except Exception as e:
+                        api_results.append({"api": "sonarr", "status": f"error: {str(e)[:50]}"})
+            
+            # Determine overall API health
+            if apis_tested == 0:
+                status = HealthStatus.HEALTHY
+                message = "No external APIs configured"
+            elif apis_healthy == apis_tested:
+                status = HealthStatus.HEALTHY
+                message = f"All {apis_tested} external APIs healthy"
+            elif apis_healthy > 0:
+                status = HealthStatus.DEGRADED
+                message = f"{apis_healthy}/{apis_tested} external APIs healthy"
+            else:
+                status = HealthStatus.UNHEALTHY
+                message = "No external APIs responding"
+            
+            return HealthCheck(
+                name="external_apis",
+                status=status,
+                message=message,
+                duration_ms=(time.time() - start_time) * 1000,
+                details={
+                    "apis_tested": apis_tested,
+                    "apis_healthy": apis_healthy,
+                    "api_results": api_results
+                }
+            )
+            
+        except Exception as e:
+            return HealthCheck(
+                name="external_apis",
+                status=HealthStatus.UNHEALTHY,
+                message=f"API health check failed: {e}",
+                duration_ms=(time.time() - start_time) * 1000
+            )
+    
+    async def check_performance_health(self) -> HealthCheck:
+        """Check system performance metrics"""
+        start_time = time.time()
+        
+        try:
+            system_metrics = metrics.get_system_metrics()
+            processing_metrics = metrics.get_processing_metrics()
+            
+            issues = []
+            warnings = []
+            
+            # Check CPU usage
+            cpu_percent = system_metrics.get("cpu_percent", 0)
+            if cpu_percent > 90:
+                issues.append(f"High CPU usage: {cpu_percent:.1f}%")
+            elif cpu_percent > 70:
+                warnings.append(f"Elevated CPU usage: {cpu_percent:.1f}%")
+            
+            # Check memory usage
+            memory_percent = system_metrics.get("memory_percent", 0)
+            if memory_percent > 90:
+                issues.append(f"High memory usage: {memory_percent:.1f}%")
+            elif memory_percent > 80:
+                warnings.append(f"Elevated memory usage: {memory_percent:.1f}%")
+            
+            # Check disk space
+            if "db_disk_free" in system_metrics and system_metrics["db_disk_free"]:
+                free_space_gb = system_metrics["db_disk_free"] / (1024**3)
+                if free_space_gb < 1:
+                    issues.append(f"Low disk space: {free_space_gb:.1f}GB free")
+                elif free_space_gb < 5:
+                    warnings.append(f"Low disk space: {free_space_gb:.1f}GB free")
+            
+            # Check active operations
+            active_ops = system_metrics.get("active_operations", 0)
+            if active_ops > 10:
+                warnings.append(f"High concurrent operations: {active_ops}")
+            
+            # Determine status
+            if issues:
+                status = HealthStatus.UNHEALTHY
+                message = f"Performance issues detected: {', '.join(issues[:2])}"
+            elif warnings:
+                status = HealthStatus.DEGRADED
+                message = f"Performance warnings: {', '.join(warnings[:2])}"
+            else:
+                status = HealthStatus.HEALTHY
+                message = "System performance normal"
+            
+            return HealthCheck(
+                name="performance",
+                status=status,
+                message=message,
+                duration_ms=(time.time() - start_time) * 1000,
+                details={
+                    "cpu_percent": cpu_percent,
+                    "memory_percent": memory_percent,
+                    "active_operations": active_ops,
+                    "issues": issues,
+                    "warnings": warnings
+                }
+            )
+            
+        except Exception as e:
+            return HealthCheck(
+                name="performance",
+                status=HealthStatus.DEGRADED,
+                message=f"Performance check failed: {e}",
+                duration_ms=(time.time() - start_time) * 1000
+            )
+    
+    async def get_full_health_status(self) -> OverallHealth:
+        """Get comprehensive health status"""
+        start_time = time.time()
+        
+        # Run all health checks concurrently
+        checks = await asyncio.gather(
+            self.check_basic_health(),
+            self.check_filesystem_health(),
+            self.check_database_health(),
+            self.check_external_apis_health(),
+            self.check_performance_health(),
+            return_exceptions=True
+        )
+        
+        # Filter out any exceptions and convert to HealthCheck objects
+        valid_checks = []
+        for check in checks:
+            if isinstance(check, HealthCheck):
+                valid_checks.append(check)
+            elif isinstance(check, Exception):
+                valid_checks.append(HealthCheck(
+                    name="unknown",
+                    status=HealthStatus.UNHEALTHY,
+                    message=f"Health check exception: {check}",
+                    duration_ms=0
+                ))
+        
+        # Determine overall status
+        unhealthy_count = len([c for c in valid_checks if c.status == HealthStatus.UNHEALTHY])
+        degraded_count = len([c for c in valid_checks if c.status == HealthStatus.DEGRADED])
+        
+        if unhealthy_count > 0:
+            overall_status = HealthStatus.UNHEALTHY
+        elif degraded_count > 0:
+            overall_status = HealthStatus.DEGRADED
+        else:
+            overall_status = HealthStatus.HEALTHY
+        
+        return OverallHealth(
+            status=overall_status,
+            checks=valid_checks,
+            timestamp=start_time,
+            uptime_seconds=time.time() - self.start_time
+        )
+    
+    async def get_readiness_status(self) -> Dict[str, Any]:
+        """Get readiness status for Kubernetes readiness probes"""
+        # Readiness is simpler - just check critical components
+        checks = await asyncio.gather(
+            self.check_basic_health(),
+            self.check_filesystem_health(),
+            self.check_database_health(),
+            return_exceptions=True
+        )
+        
+        critical_failures = 0
+        for check in checks:
+            if isinstance(check, HealthCheck) and check.status == HealthStatus.UNHEALTHY:
+                critical_failures += 1
+        
+        is_ready = critical_failures == 0
+        
+        return {
+            "ready": is_ready,
+            "timestamp": time.time(),
+            "critical_failures": critical_failures,
+            "message": "Service ready" if is_ready else f"{critical_failures} critical failures"
+        }
+    
+    async def get_liveness_status(self) -> Dict[str, Any]:
+        """Get liveness status for Kubernetes liveness probes"""
+        # Liveness is even simpler - just check if service is responsive
+        basic_check = await self.check_basic_health()
+        
+        is_alive = basic_check.status != HealthStatus.UNHEALTHY
+        
+        return {
+            "alive": is_alive,
+            "timestamp": time.time(),
+            "uptime_seconds": time.time() - self.start_time,
+            "message": basic_check.message
+        }
+
+
+# Global health checker instance
+health_checker = HealthChecker()

monitoring/logging_enhanced.py

@@ -0,0 +1,404 @@
+"""
+Enhanced Logging System for NFOGuard
+Provides structured logging with correlation IDs, request tracing, and monitoring integration
+"""
+import logging
+import json
+import time
+import uuid
+import threading
+from typing import Dict, Any, Optional, List, Union
+from dataclasses import dataclass, field
+from contextlib import contextmanager
+from datetime import datetime
+import sys
+import traceback
+
+from monitoring.metrics import metrics
+
+
+# Thread-local storage for correlation context
+_context = threading.local()
+
+
+@dataclass
+class LogContext:
+    """Logging context with correlation and tracing information"""
+    correlation_id: str
+    request_id: Optional[str] = None
+    user_id: Optional[str] = None
+    operation: Optional[str] = None
+    media_type: Optional[str] = None
+    media_title: Optional[str] = None
+    webhook_type: Optional[str] = None
+    processing_stage: Optional[str] = None
+    additional_fields: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert context to dictionary for logging"""
+        context = {
+            "correlation_id": self.correlation_id,
+            "timestamp": datetime.utcnow().isoformat(),
+        }
+        
+        # Add non-None fields
+        for field_name in ["request_id", "user_id", "operation", "media_type", 
+                          "media_title", "webhook_type", "processing_stage"]:
+            value = getattr(self, field_name)
+            if value is not None:
+                context[field_name] = value
+        
+        # Add additional fields
+        context.update(self.additional_fields)
+        
+        return context
+
+
+class StructuredFormatter(logging.Formatter):
+    """JSON formatter for structured logging"""
+    
+    def __init__(self, include_context: bool = True):
+        super().__init__()
+        self.include_context = include_context
+    
+    def format(self, record: logging.LogRecord) -> str:
+        # Base log entry
+        log_entry = {
+            "timestamp": datetime.utcnow().isoformat(),
+            "level": record.levelname,
+            "logger": record.name,
+            "message": record.getMessage(),
+            "module": record.module,
+            "function": record.funcName,
+            "line": record.lineno,
+        }
+        
+        # Add thread information
+        log_entry["thread"] = {
+            "id": record.thread,
+            "name": record.threadName
+        }
+        
+        # Add correlation context if available
+        if self.include_context and hasattr(_context, 'log_context'):
+            log_entry["context"] = _context.log_context.to_dict()
+        
+        # Add exception information if present
+        if record.exc_info:
+            log_entry["exception"] = {
+                "type": record.exc_info[0].__name__,
+                "message": str(record.exc_info[1]),
+                "traceback": traceback.format_exception(*record.exc_info)
+            }
+        
+        # Add any extra fields passed to log call
+        if hasattr(record, 'extra_fields') and record.extra_fields:
+            log_entry["extra"] = record.extra_fields
+        
+        # Add performance metrics if available
+        if hasattr(record, 'performance_data') and record.performance_data:
+            log_entry["performance"] = record.performance_data
+        
+        return json.dumps(log_entry, default=str, ensure_ascii=False)
+
+
+class CorrelationIDFilter(logging.Filter):
+    """Filter to add correlation ID to log records"""
+    
+    def filter(self, record: logging.LogRecord) -> bool:
+        # Add correlation ID to record if available
+        if hasattr(_context, 'log_context'):
+            record.correlation_id = _context.log_context.correlation_id
+        else:
+            record.correlation_id = "no-correlation"
+        
+        return True
+
+
+class EnhancedLogger:
+    """Enhanced logger with correlation IDs and structured logging"""
+    
+    def __init__(self, name: str):
+        self.logger = logging.getLogger(name)
+        self.name = name
+        
+        # Track log events for metrics
+        self._log_counts = {"debug": 0, "info": 0, "warning": 0, "error": 0, "critical": 0}
+    
+    def _log_with_context(self, level: int, message: str, extra_fields: Optional[Dict[str, Any]] = None,
+                         performance_data: Optional[Dict[str, Any]] = None, **kwargs):
+        """Log with enhanced context and metrics tracking"""
+        
+        # Track log counts for metrics
+        level_name = logging.getLevelName(level).lower()
+        if level_name in self._log_counts:
+            self._log_counts[level_name] += 1
+            metrics.increment_counter(f"log_messages", 1, {"level": level_name, "logger": self.name})
+        
+        # Create log record with extra data
+        extra = {}
+        if extra_fields:
+            extra['extra_fields'] = extra_fields
+        if performance_data:
+            extra['performance_data'] = performance_data
+        
+        # Log the message
+        self.logger.log(level, message, extra=extra, **kwargs)
+        
+        # Track errors in metrics
+        if level >= logging.ERROR:
+            metrics.record_error("logging_error", message, self.name)
+    
+    def debug(self, message: str, **kwargs):
+        """Log debug message"""
+        self._log_with_context(logging.DEBUG, message, **kwargs)
+    
+    def info(self, message: str, **kwargs):
+        """Log info message"""
+        self._log_with_context(logging.INFO, message, **kwargs)
+    
+    def warning(self, message: str, **kwargs):
+        """Log warning message"""
+        self._log_with_context(logging.WARNING, message, **kwargs)
+    
+    def error(self, message: str, **kwargs):
+        """Log error message"""
+        self._log_with_context(logging.ERROR, message, **kwargs)
+    
+    def critical(self, message: str, **kwargs):
+        """Log critical message"""
+        self._log_with_context(logging.CRITICAL, message, **kwargs)
+    
+    def exception(self, message: str, **kwargs):
+        """Log exception with traceback"""
+        kwargs['exc_info'] = True
+        self._log_with_context(logging.ERROR, message, **kwargs)
+    
+    def log_operation_start(self, operation: str, **context_fields):
+        """Log the start of an operation"""
+        self.info(f"Starting operation: {operation}", 
+                 extra_fields={"operation_event": "start", "operation": operation, **context_fields})
+    
+    def log_operation_end(self, operation: str, success: bool = True, duration: Optional[float] = None, **context_fields):
+        """Log the end of an operation"""
+        outcome = "success" if success else "failure"
+        extra = {"operation_event": "end", "operation": operation, "outcome": outcome, **context_fields}
+        
+        if duration is not None:
+            extra["duration_seconds"] = duration
+        
+        level = logging.INFO if success else logging.ERROR
+        self._log_with_context(level, f"Operation {outcome}: {operation}", extra_fields=extra)
+    
+    def log_webhook_received(self, webhook_type: str, payload_size: int, **context_fields):
+        """Log webhook reception"""
+        self.info(f"Webhook received: {webhook_type}", 
+                 extra_fields={
+                     "event_type": "webhook_received",
+                     "webhook_type": webhook_type,
+                     "payload_size_bytes": payload_size,
+                     **context_fields
+                 })
+    
+    def log_nfo_operation(self, operation: str, file_path: str, success: bool = True, **context_fields):
+        """Log NFO file operations"""
+        outcome = "success" if success else "failure"
+        level = logging.INFO if success else logging.ERROR
+        
+        self._log_with_context(level, f"NFO {operation} {outcome}: {file_path}",
+                              extra_fields={
+                                  "event_type": "nfo_operation",
+                                  "nfo_operation": operation,
+                                  "file_path": file_path,
+                                  "outcome": outcome,
+                                  **context_fields
+                              })
+    
+    def log_performance_metrics(self, operation: str, duration: float, success: bool = True, **metrics_data):
+        """Log performance metrics"""
+        self.debug(f"Performance: {operation} took {duration:.3f}s",
+                  performance_data={
+                      "operation": operation,
+                      "duration_seconds": duration,
+                      "success": success,
+                      **metrics_data
+                  })
+    
+    def get_log_stats(self) -> Dict[str, int]:
+        """Get logging statistics"""
+        return self._log_counts.copy()
+
+
+def setup_enhanced_logging(
+    log_level: str = "INFO", 
+    structured: bool = True,
+    log_file: Optional[str] = None,
+    max_bytes: int = 10 * 1024 * 1024,  # 10MB
+    backup_count: int = 5
+) -> None:
+    """Setup enhanced logging configuration"""
+    
+    # Configure root logger
+    root_logger = logging.getLogger()
+    root_logger.setLevel(getattr(logging, log_level.upper()))
+    
+    # Clear existing handlers
+    root_logger.handlers.clear()
+    
+    # Create console handler
+    console_handler = logging.StreamHandler(sys.stdout)
+    
+    if structured:
+        # Use structured JSON formatter
+        formatter = StructuredFormatter(include_context=True)
+    else:
+        # Use simple text formatter with correlation ID
+        formatter = logging.Formatter(
+            '%(asctime)s [%(correlation_id)s] %(levelname)s %(name)s: %(message)s'
+        )
+    
+    console_handler.setFormatter(formatter)
+    console_handler.addFilter(CorrelationIDFilter())
+    root_logger.addHandler(console_handler)
+    
+    # Add file handler if specified
+    if log_file:
+        from logging.handlers import RotatingFileHandler
+        
+        file_handler = RotatingFileHandler(
+            log_file, maxBytes=max_bytes, backupCount=backup_count
+        )
+        file_handler.setFormatter(formatter)
+        file_handler.addFilter(CorrelationIDFilter())
+        root_logger.addHandler(file_handler)
+    
+    # Reduce noise from external libraries
+    logging.getLogger("urllib3").setLevel(logging.WARNING)
+    logging.getLogger("requests").setLevel(logging.WARNING)
+    logging.getLogger("aiohttp").setLevel(logging.WARNING)
+
+
+def get_enhanced_logger(name: str) -> EnhancedLogger:
+    """Get enhanced logger instance"""
+    return EnhancedLogger(name)
+
+
+def set_log_context(
+    correlation_id: Optional[str] = None,
+    request_id: Optional[str] = None,
+    operation: Optional[str] = None,
+    **kwargs
+) -> LogContext:
+    """Set logging context for current thread"""
+    
+    if correlation_id is None:
+        correlation_id = str(uuid.uuid4())
+    
+    context = LogContext(
+        correlation_id=correlation_id,
+        request_id=request_id,
+        operation=operation,
+        **kwargs
+    )
+    
+    _context.log_context = context
+    return context
+
+
+def get_log_context() -> Optional[LogContext]:
+    """Get current logging context"""
+    return getattr(_context, 'log_context', None)
+
+
+def clear_log_context():
+    """Clear logging context for current thread"""
+    if hasattr(_context, 'log_context'):
+        delattr(_context, 'log_context')
+
+
+@contextmanager
+def log_context(correlation_id: Optional[str] = None, **context_fields):
+    """Context manager for scoped logging context"""
+    original_context = get_log_context()
+    
+    try:
+        # Set new context
+        new_context = set_log_context(correlation_id=correlation_id, **context_fields)
+        yield new_context
+    finally:
+        # Restore original context
+        if original_context:
+            _context.log_context = original_context
+        else:
+            clear_log_context()
+
+
+@contextmanager
+def log_operation(operation: str, logger: Optional[EnhancedLogger] = None, **context_fields):
+    """Context manager for logging operation start/end with timing"""
+    if logger is None:
+        logger = get_enhanced_logger(__name__)
+    
+    start_time = time.time()
+    success = True
+    
+    # Update context with operation
+    current_context = get_log_context()
+    if current_context:
+        current_context.operation = operation
+        current_context.processing_stage = "executing"
+    
+    logger.log_operation_start(operation, **context_fields)
+    
+    try:
+        yield
+    except Exception as e:
+        success = False
+        logger.exception(f"Operation failed: {operation}", 
+                        extra_fields={"operation": operation, "error": str(e), **context_fields})
+        raise
+    finally:
+        duration = time.time() - start_time
+        logger.log_operation_end(operation, success, duration, **context_fields)
+        
+        # Update metrics
+        metrics.record_operation_duration(operation, duration, success)
+
+
+def trace_request(request_id: Optional[str] = None, **context_fields):
+    """Decorator/context manager for request tracing"""
+    def decorator(func):
+        def wrapper(*args, **kwargs):
+            correlation_id = str(uuid.uuid4())
+            req_id = request_id or f"req_{int(time.time())}"
+            
+            with log_context(correlation_id=correlation_id, request_id=req_id, **context_fields):
+                return func(*args, **kwargs)
+        return wrapper
+    
+    # Can be used as context manager or decorator
+    if request_id is None and len(context_fields) == 1 and callable(list(context_fields.values())[0]):
+        # Used as decorator without parentheses
+        func = list(context_fields.values())[0]
+        return decorator(func)
+    else:
+        # Used as decorator with parameters or context manager
+        return decorator
+
+
+# Module-level logger for this module
+logger = get_enhanced_logger(__name__)
+
+
+def get_logging_stats() -> Dict[str, Any]:
+    """Get comprehensive logging statistics"""
+    # Collect stats from all enhanced loggers
+    total_stats = {"debug": 0, "info": 0, "warning": 0, "error": 0, "critical": 0}
+    
+    # This is a simplified version - in practice you'd track all logger instances
+    return {
+        "total_log_messages": sum(total_stats.values()),
+        "by_level": total_stats,
+        "structured_logging_enabled": True,
+        "correlation_tracking_enabled": True
+    }

monitoring/metrics.py

@@ -0,0 +1,354 @@
+"""
+Metrics Collection System for NFOGuard
+Provides performance monitoring, counters, and operational metrics
+"""
+import time
+import psutil
+import threading
+from datetime import datetime, timedelta
+from typing import Dict, Any, List, Optional
+from dataclasses import dataclass, field
+from collections import defaultdict, deque
+from contextlib import contextmanager
+import asyncio
+
+
+@dataclass
+class MetricValue:
+    """Individual metric value with timestamp"""
+    value: float
+    timestamp: float = field(default_factory=time.time)
+    labels: Dict[str, str] = field(default_factory=dict)
+
+
+@dataclass 
+class TimeSeriesMetric:
+    """Time series metric with historical data"""
+    name: str
+    values: deque = field(default_factory=lambda: deque(maxlen=1000))
+    total: float = 0.0
+    count: int = 0
+    
+    def add_value(self, value: float, labels: Optional[Dict[str, str]] = None):
+        """Add a new metric value"""
+        metric_value = MetricValue(value, labels=labels or {})
+        self.values.append(metric_value)
+        self.total += value
+        self.count += 1
+    
+    def get_average(self, window_seconds: int = 300) -> float:
+        """Get average value over time window"""
+        cutoff_time = time.time() - window_seconds
+        recent_values = [v.value for v in self.values if v.timestamp > cutoff_time]
+        return sum(recent_values) / len(recent_values) if recent_values else 0.0
+    
+    def get_rate_per_minute(self, window_seconds: int = 300) -> float:
+        """Get rate per minute over time window"""
+        cutoff_time = time.time() - window_seconds
+        recent_count = len([v for v in self.values if v.timestamp > cutoff_time])
+        return (recent_count / window_seconds) * 60 if window_seconds > 0 else 0.0
+
+
+class MetricsCollector:
+    """Central metrics collection system"""
+    
+    def __init__(self):
+        self._metrics: Dict[str, TimeSeriesMetric] = {}
+        self._counters: Dict[str, int] = defaultdict(int)
+        self._gauges: Dict[str, float] = {}
+        self._histograms: Dict[str, List[float]] = defaultdict(list)
+        self._start_time = time.time()
+        self._lock = threading.RLock()
+        
+        # Processing metrics
+        self._active_operations = 0
+        self._operation_durations = deque(maxlen=1000)
+        
+        # Error tracking
+        self._error_counts = defaultdict(int)
+        self._last_errors = deque(maxlen=100)
+        
+        # System metrics
+        self._system_stats_cache = {}
+        self._system_stats_last_update = 0
+        self._system_stats_cache_ttl = 30  # 30 seconds
+    
+    def increment_counter(self, name: str, value: int = 1, labels: Optional[Dict[str, str]] = None):
+        """Increment a counter metric"""
+        with self._lock:
+            full_name = self._build_metric_name(name, labels)
+            self._counters[full_name] += value
+            
+            # Also track in time series for rate calculations
+            if name not in self._metrics:
+                self._metrics[name] = TimeSeriesMetric(name)
+            self._metrics[name].add_value(value, labels)
+    
+    def set_gauge(self, name: str, value: float, labels: Optional[Dict[str, str]] = None):
+        """Set a gauge metric value"""
+        with self._lock:
+            full_name = self._build_metric_name(name, labels)
+            self._gauges[full_name] = value
+    
+    def record_histogram(self, name: str, value: float, labels: Optional[Dict[str, str]] = None):
+        """Record a histogram value"""
+        with self._lock:
+            full_name = self._build_metric_name(name, labels)
+            self._histograms[full_name].append(value)
+            
+            # Keep only recent values (last 1000)
+            if len(self._histograms[full_name]) > 1000:
+                self._histograms[full_name] = self._histograms[full_name][-1000:]
+            
+            # Also track in time series
+            if name not in self._metrics:
+                self._metrics[name] = TimeSeriesMetric(name)
+            self._metrics[name].add_value(value, labels)
+    
+    def record_operation_duration(self, operation: str, duration: float, success: bool = True):
+        """Record operation duration and outcome"""
+        with self._lock:
+            # Record duration
+            self.record_histogram(f"operation_duration_{operation}", duration)
+            
+            # Record outcome
+            outcome = "success" if success else "error"
+            self.increment_counter(f"operation_total", 1, {"operation": operation, "outcome": outcome})
+            
+            # Track active operations
+            if operation.endswith("_start"):
+                self._active_operations += 1
+            elif operation.endswith("_end"):
+                self._active_operations = max(0, self._active_operations - 1)
+    
+    def record_error(self, error_type: str, error_message: str, operation: Optional[str] = None):
+        """Record an error occurrence"""
+        with self._lock:
+            self._error_counts[error_type] += 1
+            
+            error_info = {
+                "type": error_type,
+                "message": error_message,
+                "operation": operation,
+                "timestamp": time.time()
+            }
+            self._last_errors.append(error_info)
+            
+            # Increment error counter
+            labels = {"error_type": error_type}
+            if operation:
+                labels["operation"] = operation
+            self.increment_counter("errors_total", 1, labels)
+    
+    @contextmanager
+    def operation_timer(self, operation: str):
+        """Context manager for timing operations"""
+        start_time = time.time()
+        success = True
+        
+        try:
+            self.record_operation_duration(f"{operation}_start", 0)
+            yield
+        except Exception as e:
+            success = False
+            self.record_error("operation_error", str(e), operation)
+            raise
+        finally:
+            duration = time.time() - start_time
+            self.record_operation_duration(operation, duration, success)
+            self.record_operation_duration(f"{operation}_end", 0)
+    
+    def get_system_metrics(self) -> Dict[str, Any]:
+        """Get current system resource metrics"""
+        now = time.time()
+        
+        # Use cached values if recent
+        if (now - self._system_stats_last_update) < self._system_stats_cache_ttl:
+            return self._system_stats_cache
+        
+        try:
+            # CPU metrics
+            cpu_percent = psutil.cpu_percent(interval=0.1)
+            cpu_count = psutil.cpu_count()
+            
+            # Memory metrics
+            memory = psutil.virtual_memory()
+            
+            # Disk metrics for database path
+            try:
+                from config.settings import config
+                db_disk = psutil.disk_usage(str(config.db_path.parent))
+            except:
+                db_disk = None
+            
+            # Process metrics
+            process = psutil.Process()
+            process_memory = process.memory_info()
+            
+            self._system_stats_cache = {
+                "cpu_percent": cpu_percent,
+                "cpu_count": cpu_count,
+                "memory_total": memory.total,
+                "memory_available": memory.available,
+                "memory_percent": memory.percent,
+                "process_memory_rss": process_memory.rss,
+                "process_memory_vms": process_memory.vms,
+                "db_disk_free": db_disk.free if db_disk else None,
+                "db_disk_total": db_disk.total if db_disk else None,
+                "active_operations": self._active_operations,
+                "uptime_seconds": now - self._start_time
+            }
+            
+            self._system_stats_last_update = now
+            
+        except Exception as e:
+            # Return basic metrics if detailed collection fails
+            self._system_stats_cache = {
+                "uptime_seconds": now - self._start_time,
+                "active_operations": self._active_operations,
+                "error": str(e)
+            }
+        
+        return self._system_stats_cache
+    
+    def get_processing_metrics(self) -> Dict[str, Any]:
+        """Get processing-related metrics"""
+        with self._lock:
+            # Calculate rates and averages
+            webhook_rate = self._metrics.get("webhooks_received", TimeSeriesMetric("webhooks_received")).get_rate_per_minute()
+            nfo_rate = self._metrics.get("nfo_created", TimeSeriesMetric("nfo_created")).get_rate_per_minute()
+            
+            avg_processing_time = 0.0
+            if "processing_duration" in self._metrics:
+                avg_processing_time = self._metrics["processing_duration"].get_average()
+            
+            return {
+                "webhooks_received_per_minute": webhook_rate,
+                "nfo_files_created_per_minute": nfo_rate,
+                "average_processing_time_seconds": avg_processing_time,
+                "active_operations": self._active_operations,
+                "total_webhooks": self._counters.get("webhooks_received", 0),
+                "total_nfo_created": self._counters.get("nfo_created", 0),
+                "total_errors": sum(self._error_counts.values())
+            }
+    
+    def get_error_metrics(self) -> Dict[str, Any]:
+        """Get error-related metrics"""
+        with self._lock:
+            recent_errors = []
+            cutoff_time = time.time() - 3600  # Last hour
+            
+            for error in self._last_errors:
+                if error["timestamp"] > cutoff_time:
+                    recent_errors.append({
+                        "type": error["type"],
+                        "message": error["message"][:100],  # Truncate long messages
+                        "operation": error["operation"],
+                        "timestamp": error["timestamp"]
+                    })
+            
+            return {
+                "error_counts_by_type": dict(self._error_counts),
+                "recent_errors": recent_errors[-10:],  # Last 10 errors
+                "total_errors": sum(self._error_counts.values()),
+                "error_rate_per_minute": len([e for e in self._last_errors if e["timestamp"] > time.time() - 300]) / 5
+            }
+    
+    def get_prometheus_metrics(self) -> str:
+        """Generate Prometheus-compatible metrics format"""
+        lines = []
+        
+        # Add help and type information
+        lines.append("# HELP nfoguard_webhooks_total Total number of webhooks received")
+        lines.append("# TYPE nfoguard_webhooks_total counter")
+        
+        with self._lock:
+            # Counters
+            for name, value in self._counters.items():
+                metric_name = f"nfoguard_{name.replace('-', '_')}"
+                lines.append(f"{metric_name} {value}")
+            
+            # Gauges
+            lines.append("# HELP nfoguard_active_operations Current number of active operations")
+            lines.append("# TYPE nfoguard_active_operations gauge")
+            lines.append(f"nfoguard_active_operations {self._active_operations}")
+            
+            # System metrics
+            system_metrics = self.get_system_metrics()
+            for key, value in system_metrics.items():
+                if isinstance(value, (int, float)) and value is not None:
+                    metric_name = f"nfoguard_system_{key}"
+                    lines.append(f"{metric_name} {value}")
+        
+        return "\n".join(lines)
+    
+    def get_all_metrics(self) -> Dict[str, Any]:
+        """Get all metrics in a structured format"""
+        return {
+            "system": self.get_system_metrics(),
+            "processing": self.get_processing_metrics(),
+            "errors": self.get_error_metrics(),
+            "timestamp": time.time(),
+            "uptime_seconds": time.time() - self._start_time
+        }
+    
+    def reset_metrics(self, metric_types: Optional[List[str]] = None):
+        """Reset specific metric types or all metrics"""
+        with self._lock:
+            if not metric_types or "counters" in metric_types:
+                self._counters.clear()
+            
+            if not metric_types or "histograms" in metric_types:
+                self._histograms.clear()
+            
+            if not metric_types or "errors" in metric_types:
+                self._error_counts.clear()
+                self._last_errors.clear()
+            
+            if not metric_types or "timeseries" in metric_types:
+                self._metrics.clear()
+    
+    def _build_metric_name(self, name: str, labels: Optional[Dict[str, str]]) -> str:
+        """Build metric name with labels"""
+        if not labels:
+            return name
+        
+        label_str = ",".join(f"{k}={v}" for k, v in sorted(labels.items()))
+        return f"{name}{{{label_str}}}"
+
+
+# Global metrics collector instance
+metrics = MetricsCollector()
+
+
+# Convenience functions for common operations
+def track_webhook_received(webhook_type: str):
+    """Track webhook received"""
+    metrics.increment_counter("webhooks_received", 1, {"type": webhook_type})
+
+
+def track_nfo_created(media_type: str, success: bool = True):
+    """Track NFO file creation"""
+    outcome = "success" if success else "error"
+    metrics.increment_counter("nfo_created", 1, {"media_type": media_type, "outcome": outcome})
+
+
+def track_api_call(api_name: str, duration: float, success: bool = True):
+    """Track external API call"""
+    metrics.record_histogram(f"api_call_duration", duration, {"api": api_name})
+    outcome = "success" if success else "error" 
+    metrics.increment_counter("api_calls_total", 1, {"api": api_name, "outcome": outcome})
+
+
+def track_database_operation(operation: str, duration: float, success: bool = True):
+    """Track database operation"""
+    metrics.record_histogram("database_operation_duration", duration, {"operation": operation})
+    outcome = "success" if success else "error"
+    metrics.increment_counter("database_operations_total", 1, {"operation": operation, "outcome": outcome})
+
+
+def track_file_operation(operation: str, duration: float, success: bool = True):
+    """Track file system operation"""
+    metrics.record_histogram("file_operation_duration", duration, {"operation": operation})
+    outcome = "success" if success else "error"
+    metrics.increment_counter("file_operations_total", 1, {"operation": operation, "outcome": outcome})

monitoring/performance.py

@@ -0,0 +1,413 @@
+"""
+Performance Monitoring and Profiling for NFOGuard
+Provides detailed performance analysis and optimization insights
+"""
+import time
+import asyncio
+import threading
+import functools
+from typing import Dict, Any, List, Optional, Callable, TypeVar, Union
+from dataclasses import dataclass, field
+from collections import defaultdict, deque
+from contextlib import asynccontextmanager, contextmanager
+import traceback
+import sys
+
+from monitoring.metrics import metrics
+
+
+T = TypeVar('T')
+
+
+@dataclass
+class PerformanceProfile:
+    """Performance profile for an operation"""
+    operation_name: str
+    total_calls: int = 0
+    total_duration: float = 0.0
+    min_duration: float = float('inf')
+    max_duration: float = 0.0
+    recent_durations: deque = field(default_factory=lambda: deque(maxlen=100))
+    error_count: int = 0
+    concurrent_calls: int = 0
+    
+    def add_measurement(self, duration: float, success: bool = True):
+        """Add a performance measurement"""
+        self.total_calls += 1
+        self.total_duration += duration
+        self.min_duration = min(self.min_duration, duration)
+        self.max_duration = max(self.max_duration, duration)
+        self.recent_durations.append(duration)
+        
+        if not success:
+            self.error_count += 1
+    
+    def get_average_duration(self) -> float:
+        """Get average duration across all calls"""
+        return self.total_duration / self.total_calls if self.total_calls > 0 else 0.0
+    
+    def get_recent_average(self, window: int = 50) -> float:
+        """Get average of recent calls"""
+        recent = list(self.recent_durations)[-window:]
+        return sum(recent) / len(recent) if recent else 0.0
+    
+    def get_percentiles(self) -> Dict[str, float]:
+        """Get duration percentiles for recent calls"""
+        recent = sorted(list(self.recent_durations))
+        if not recent:
+            return {"p50": 0, "p95": 0, "p99": 0}
+        
+        length = len(recent)
+        return {
+            "p50": recent[int(length * 0.5)] if length > 0 else 0,
+            "p95": recent[int(length * 0.95)] if length > 0 else 0,
+            "p99": recent[int(length * 0.99)] if length > 0 else 0
+        }
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for API responses"""
+        percentiles = self.get_percentiles()
+        
+        return {
+            "operation_name": self.operation_name,
+            "total_calls": self.total_calls,
+            "error_count": self.error_count,
+            "error_rate": self.error_count / self.total_calls if self.total_calls > 0 else 0,
+            "concurrent_calls": self.concurrent_calls,
+            "duration_stats": {
+                "average": round(self.get_average_duration(), 4),
+                "recent_average": round(self.get_recent_average(), 4),
+                "min": round(self.min_duration if self.min_duration != float('inf') else 0, 4),
+                "max": round(self.max_duration, 4),
+                "p50": round(percentiles["p50"], 4),
+                "p95": round(percentiles["p95"], 4),
+                "p99": round(percentiles["p99"], 4)
+            },
+            "performance_rating": self._get_performance_rating()
+        }
+    
+    def _get_performance_rating(self) -> str:
+        """Get performance rating based on metrics"""
+        avg_duration = self.get_recent_average()
+        error_rate = self.error_count / self.total_calls if self.total_calls > 0 else 0
+        
+        if error_rate > 0.1:  # >10% error rate
+            return "poor"
+        elif avg_duration > 5.0:  # >5 seconds average
+            return "slow"
+        elif avg_duration > 1.0:  # >1 second average
+            return "acceptable"
+        else:
+            return "excellent"
+
+
+class PerformanceMonitor:
+    """Advanced performance monitoring system"""
+    
+    def __init__(self):
+        self._profiles: Dict[str, PerformanceProfile] = {}
+        self._active_operations: Dict[str, float] = {}  # operation_id -> start_time
+        self._lock = threading.RLock()
+        
+        # Slow operation tracking
+        self._slow_operation_threshold = 1.0  # 1 second
+        self._slow_operations = deque(maxlen=100)
+        
+        # Memory monitoring
+        self._memory_samples = deque(maxlen=1000)
+        self._memory_monitoring_enabled = True
+        
+        # Async operation tracking
+        self._async_tasks = {}
+        self._task_counter = 0
+    
+    def get_profile(self, operation_name: str) -> PerformanceProfile:
+        """Get or create performance profile for operation"""
+        with self._lock:
+            if operation_name not in self._profiles:
+                self._profiles[operation_name] = PerformanceProfile(operation_name)
+            return self._profiles[operation_name]
+    
+    @contextmanager
+    def monitor_operation(self, operation_name: str, **kwargs):
+        """Context manager for monitoring synchronous operations"""
+        start_time = time.time()
+        operation_id = f"{operation_name}_{id(threading.current_thread())}_{time.time()}"
+        success = True
+        
+        profile = self.get_profile(operation_name)
+        
+        with self._lock:
+            profile.concurrent_calls += 1
+            self._active_operations[operation_id] = start_time
+        
+        try:
+            yield
+        except Exception as e:
+            success = False
+            metrics.record_error("performance_monitor", str(e), operation_name)
+            raise
+        finally:
+            end_time = time.time()
+            duration = end_time - start_time
+            
+            with self._lock:
+                profile.concurrent_calls = max(0, profile.concurrent_calls - 1)
+                self._active_operations.pop(operation_id, None)
+                
+                # Record measurement
+                profile.add_measurement(duration, success)
+                
+                # Track slow operations
+                if duration > self._slow_operation_threshold:
+                    self._slow_operations.append({
+                        "operation": operation_name,
+                        "duration": duration,
+                        "timestamp": end_time,
+                        "success": success,
+                        "metadata": kwargs
+                    })
+                
+                # Update metrics
+                metrics.record_histogram(f"operation_duration", duration, {"operation": operation_name})
+                if not success:
+                    metrics.increment_counter("operation_errors", 1, {"operation": operation_name})
+    
+    @asynccontextmanager
+    async def monitor_async_operation(self, operation_name: str, **kwargs):
+        """Context manager for monitoring asynchronous operations"""
+        start_time = time.time()
+        task_id = f"{operation_name}_{self._task_counter}"
+        self._task_counter += 1
+        success = True
+        
+        profile = self.get_profile(operation_name)
+        
+        with self._lock:
+            profile.concurrent_calls += 1
+            self._async_tasks[task_id] = {
+                "operation": operation_name,
+                "start_time": start_time,
+                "metadata": kwargs
+            }
+        
+        try:
+            yield
+        except Exception as e:
+            success = False
+            metrics.record_error("async_performance_monitor", str(e), operation_name)
+            raise
+        finally:
+            end_time = time.time()
+            duration = end_time - start_time
+            
+            with self._lock:
+                profile.concurrent_calls = max(0, profile.concurrent_calls - 1)
+                self._async_tasks.pop(task_id, None)
+                
+                # Record measurement
+                profile.add_measurement(duration, success)
+                
+                # Track slow operations
+                if duration > self._slow_operation_threshold:
+                    self._slow_operations.append({
+                        "operation": operation_name,
+                        "duration": duration,
+                        "timestamp": end_time,
+                        "success": success,
+                        "async": True,
+                        "metadata": kwargs
+                    })
+                
+                # Update metrics
+                metrics.record_histogram(f"async_operation_duration", duration, {"operation": operation_name})
+                if not success:
+                    metrics.increment_counter("async_operation_errors", 1, {"operation": operation_name})
+    
+    def monitor_function(self, operation_name: Optional[str] = None):
+        """Decorator for monitoring function performance"""
+        def decorator(func: Callable[..., T]) -> Callable[..., T]:
+            name = operation_name or f"{func.__module__}.{func.__name__}"
+            
+            if asyncio.iscoroutinefunction(func):
+                @functools.wraps(func)
+                async def async_wrapper(*args, **kwargs):
+                    async with self.monitor_async_operation(name):
+                        return await func(*args, **kwargs)
+                return async_wrapper
+            else:
+                @functools.wraps(func)
+                def sync_wrapper(*args, **kwargs):
+                    with self.monitor_operation(name):
+                        return func(*args, **kwargs)
+                return sync_wrapper
+        
+        return decorator
+    
+    def get_performance_summary(self) -> Dict[str, Any]:
+        """Get comprehensive performance summary"""
+        with self._lock:
+            # Get top operations by various metrics
+            profiles = list(self._profiles.values())
+            
+            # Sort by total calls
+            most_called = sorted(profiles, key=lambda p: p.total_calls, reverse=True)[:10]
+            
+            # Sort by average duration
+            slowest_avg = sorted(profiles, key=lambda p: p.get_average_duration(), reverse=True)[:10]
+            
+            # Sort by recent average
+            slowest_recent = sorted(profiles, key=lambda p: p.get_recent_average(), reverse=True)[:10]
+            
+            # Sort by error rate
+            highest_errors = sorted(
+                [p for p in profiles if p.total_calls > 0],
+                key=lambda p: p.error_count / p.total_calls,
+                reverse=True
+            )[:10]
+            
+            # Get active operations count
+            total_active = sum(p.concurrent_calls for p in profiles)
+            
+            # Get slow operations
+            recent_slow = list(self._slow_operations)[-20:]  # Last 20 slow operations
+            
+            return {
+                "overview": {
+                    "total_operations_tracked": len(profiles),
+                    "total_active_operations": total_active,
+                    "slow_operation_threshold_seconds": self._slow_operation_threshold,
+                    "total_slow_operations": len(self._slow_operations)
+                },
+                "top_operations": {
+                    "most_called": [p.to_dict() for p in most_called],
+                    "slowest_average": [p.to_dict() for p in slowest_avg],
+                    "slowest_recent": [p.to_dict() for p in slowest_recent],
+                    "highest_error_rate": [p.to_dict() for p in highest_errors]
+                },
+                "recent_slow_operations": recent_slow,
+                "performance_insights": self._generate_performance_insights(profiles)
+            }
+    
+    def get_operation_detail(self, operation_name: str) -> Optional[Dict[str, Any]]:
+        """Get detailed performance data for specific operation"""
+        with self._lock:
+            if operation_name not in self._profiles:
+                return None
+            
+            profile = self._profiles[operation_name]
+            
+            # Get related slow operations
+            related_slow = [
+                op for op in self._slow_operations
+                if op["operation"] == operation_name
+            ]
+            
+            detail = profile.to_dict()
+            detail.update({
+                "detailed_stats": {
+                    "total_duration": round(profile.total_duration, 4),
+                    "recent_durations": list(profile.recent_durations)[-20:],  # Last 20 calls
+                    "slow_operations_count": len(related_slow),
+                    "recent_slow_operations": related_slow[-10:]  # Last 10 slow calls
+                },
+                "recommendations": self._get_operation_recommendations(profile)
+            })
+            
+            return detail
+    
+    def _generate_performance_insights(self, profiles: List[PerformanceProfile]) -> List[str]:
+        """Generate performance optimization insights"""
+        insights = []
+        
+        # Check for very slow operations
+        very_slow = [p for p in profiles if p.get_recent_average() > 5.0]
+        if very_slow:
+            insights.append(f"Found {len(very_slow)} operations with >5s average duration - consider optimization")
+        
+        # Check for high error rates
+        high_error_rate = [p for p in profiles if p.total_calls > 10 and (p.error_count / p.total_calls) > 0.1]
+        if high_error_rate:
+            insights.append(f"Found {len(high_error_rate)} operations with >10% error rate - investigate failures")
+        
+        # Check for high concurrency
+        high_concurrency = [p for p in profiles if p.concurrent_calls > 5]
+        if high_concurrency:
+            insights.append(f"Found {len(high_concurrency)} operations with high concurrency - may need rate limiting")
+        
+        # Check total active operations
+        total_active = sum(p.concurrent_calls for p in profiles)
+        if total_active > 20:
+            insights.append(f"High total concurrent operations ({total_active}) - system may be under load")
+        
+        # Performance trends
+        recent_slow_count = len([op for op in self._slow_operations if op["timestamp"] > time.time() - 300])
+        if recent_slow_count > 10:
+            insights.append(f"Many slow operations recently ({recent_slow_count} in last 5 minutes)")
+        
+        if not insights:
+            insights.append("No significant performance issues detected")
+        
+        return insights
+    
+    def _get_operation_recommendations(self, profile: PerformanceProfile) -> List[str]:
+        """Get recommendations for optimizing specific operation"""
+        recommendations = []
+        
+        avg_duration = profile.get_recent_average()
+        error_rate = profile.error_count / profile.total_calls if profile.total_calls > 0 else 0
+        
+        if avg_duration > 5.0:
+            recommendations.append("Consider breaking down this operation into smaller parts")
+            recommendations.append("Review database queries and file I/O for optimization opportunities")
+        elif avg_duration > 1.0:
+            recommendations.append("Monitor for potential optimization opportunities")
+        
+        if error_rate > 0.1:
+            recommendations.append("High error rate - investigate common failure causes")
+            recommendations.append("Consider adding retry logic or better error handling")
+        
+        if profile.concurrent_calls > 5:
+            recommendations.append("High concurrency - consider adding rate limiting")
+            recommendations.append("Review resource usage and potential bottlenecks")
+        
+        percentiles = profile.get_percentiles()
+        if percentiles["p99"] > percentiles["p50"] * 3:
+            recommendations.append("High latency variance - investigate outlier causes")
+        
+        if not recommendations:
+            recommendations.append("Performance appears optimal for this operation")
+        
+        return recommendations
+    
+    def set_slow_operation_threshold(self, threshold_seconds: float):
+        """Set threshold for what constitutes a slow operation"""
+        with self._lock:
+            self._slow_operation_threshold = threshold_seconds
+    
+    def clear_profiles(self, operation_names: Optional[List[str]] = None):
+        """Clear performance profiles for specific operations or all"""
+        with self._lock:
+            if operation_names:
+                for name in operation_names:
+                    self._profiles.pop(name, None)
+            else:
+                self._profiles.clear()
+                self._slow_operations.clear()
+
+
+# Global performance monitor instance
+performance_monitor = PerformanceMonitor()
+
+# Decorator shortcuts
+def monitor_performance(operation_name: Optional[str] = None):
+    """Shortcut decorator for performance monitoring"""
+    return performance_monitor.monitor_function(operation_name)
+
+def monitor_sync_operation(operation_name: str, **kwargs):
+    """Shortcut for synchronous operation monitoring"""
+    return performance_monitor.monitor_operation(operation_name, **kwargs)
+
+def monitor_async_operation(operation_name: str, **kwargs):
+    """Shortcut for asynchronous operation monitoring"""
+    return performance_monitor.monitor_async_operation(operation_name, **kwargs)

processors/movie_processor.py

@@ -88,12 +88,75 @@ class MovieProcessor:
             path_mapper=self.path_mapper
         )
     
-    def process_movie(self, movie_path: Path, webhook_mode: bool = False) -> None:
+    def should_skip_movie(self, imdb_id: str, movie_name: str = "") -> Tuple[bool, str]:
+        """
+        Determine if we should skip processing this movie based on completion status
+        
+        Args:
+            imdb_id: Movie IMDb ID  
+            movie_name: Movie name for logging
+            
+        Returns:
+            (should_skip: bool, reason: str)
+        """
+        try:
+            with self.db.get_connection() as conn:
+                cursor = conn.cursor()
+                
+                if self.db.db_type == "postgresql":
+                    cursor.execute("""
+                        SELECT dateadded, source, has_video_file
+                        FROM movies 
+                        WHERE imdb_id = %s
+                    """, (imdb_id,))
+                else:
+                    cursor.execute("""
+                        SELECT dateadded, source, has_video_file
+                        FROM movies 
+                        WHERE imdb_id = ?
+                    """, (imdb_id,))
+                
+                result = cursor.fetchone()
+                if not result:
+                    return False, "No database record found"
+                
+                if self.db.db_type == "postgresql":
+                    dateadded = result['dateadded']
+                    source = result['source']
+                    has_video_file = result['has_video_file']
+                else:
+                    dateadded = result[0] if result[0] else None
+                    source = result[1] if result[1] else None  
+                    has_video_file = result[2] if result[2] else False
+                
+                # Skip if:
+                # 1. Movie has a valid dateadded timestamp
+                # 2. Source is valid (not 'unknown' or 'no_valid_date_source')  
+                # 3. Has video file on disk
+                if (dateadded and 
+                    source and 
+                    source not in ['unknown', 'no_valid_date_source'] and
+                    has_video_file):
+                    return True, f"Complete: Has valid date '{dateadded}' from source '{source}'"
+                elif not dateadded:
+                    return False, "Missing dateadded"
+                elif not source or source in ['unknown', 'no_valid_date_source']:
+                    return False, f"Invalid source: '{source}'"
+                elif not has_video_file:
+                    return False, "No video file detected"
+                else:
+                    return False, "Incomplete movie data"
+                    
+        except Exception as e:
+            _log("ERROR", f"Error checking movie completion for {imdb_id}: {e}")
+            return False, f"Error checking completion: {e}"
+    
+    def process_movie(self, movie_path: Path, webhook_mode: bool = False, force_scan: bool = False) -> str:
         """Process a movie directory"""
         imdb_id = self.nfo_manager.find_movie_imdb_id(movie_path)
         if not imdb_id:
             _log("ERROR", f"No IMDb ID found in movie directory, filenames, or NFO file: {movie_path}")
-            return
+            return "error"
         
         # Handle TMDB ID fallback case
         is_tmdb_fallback = imdb_id.startswith("tmdb-")
@@ -102,6 +165,21 @@ class MovieProcessor:
         else:
             _log("INFO", f"Processing movie: {movie_path.name} (IMDb: {imdb_id})")
         
+        # Check if we should skip this movie (unless forced or webhook mode)
+        if not force_scan and not webhook_mode:
+            should_skip, reason = self.should_skip_movie(imdb_id, movie_path.name)
+            if should_skip:
+                _log("INFO", f"⏭️ SKIPPING MOVIE: {movie_path.name} [{imdb_id}] - {reason}")
+                # Still update the movie record to track that we've seen it
+                self.db.upsert_movie(imdb_id, str(movie_path))
+                return "skipped"
+            else:
+                _log("INFO", f"🎬 PROCESSING MOVIE: {movie_path.name} [{imdb_id}] - {reason}")
+        elif force_scan:
+            _log("INFO", f"🔄 FORCE PROCESSING MOVIE: {movie_path.name} [{imdb_id}] - Force scan enabled")
+        else:
+            _log("INFO", f"📥 WEBHOOK PROCESSING MOVIE: {movie_path.name} [{imdb_id}] - Webhook mode")
+        
         # Update database
         self.db.upsert_movie(imdb_id, str(movie_path))
         
@@ -112,23 +190,98 @@ class MovieProcessor:
         if not has_video:
             _log("WARNING", f"No video files found in: {movie_path}")
             self.db.upsert_movie_dates(imdb_id, None, None, None, False)
-            return
+            return "processed"
         
-        # TIER 1: Check if NFO file already has NFOGuard data (fastest - no DB or API calls)
+        # TIER 1: Check database first (fastest - local lookup)
+        existing = self.db.get_movie_dates(imdb_id)
+        _log("DEBUG", f"Database lookup for {imdb_id}: {existing}")
+        
+        # Enhanced debug for database state
+        if existing:
+            has_dateadded = bool(existing.get("dateadded"))
+            source_value = existing.get("source")
+            _log("INFO", f"🔍 TIER 1 DEBUG - {imdb_id}: has_dateadded={has_dateadded}, source='{source_value}', dateadded='{existing.get('dateadded')}'")
+        else:
+            _log("INFO", f"🔍 TIER 1 DEBUG - {imdb_id}: No database record found")
+        
+        # If we have complete data in database, use it and skip all other checks
+        if existing and existing.get("dateadded") and existing.get("source") != "no_valid_date_source":
+            _log("INFO", f"✅ TIER 1 - Using complete database data for {imdb_id}: {existing['dateadded']} (source: {existing['source']})")
+            dateadded, source, released = existing["dateadded"], existing["source"], existing.get("released")
+            
+            # Create NFO with existing data and update files
+            if config.manage_nfo:
+                self.nfo_manager.create_movie_nfo(
+                    movie_path, imdb_id, dateadded, released, source, config.lock_metadata
+                )
+            
+            if config.fix_dir_mtimes and dateadded:
+                self.nfo_manager.update_movie_files_mtime(movie_path, dateadded)
+            
+            _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source}) [database-cached]")
+            return "processed"
+        else:
+            _log("INFO", f"🔍 TIER 1 SKIP - {imdb_id}: Database incomplete, proceeding to Tier 2")
+        
+        # TIER 2: Check if NFO file has NFOGuard data and cache it in database
         nfo_path = movie_path / "movie.nfo"
+        _log("INFO", f"🔍 TIER 2 - Checking NFO file: {nfo_path}")
+        _log("INFO", f"🔍 TIER 2 - NFO exists: {nfo_path.exists()}")
+        
         nfo_data = self.nfo_manager.extract_nfoguard_dates_from_nfo(nfo_path)
+        _log("INFO", f"🔍 TIER 2 - NFOGuard data extracted: {nfo_data}")
+        
         if nfo_data:
-            _log("INFO", f"🚀 Using existing NFOGuard data from NFO file: {nfo_data['dateadded']} (source: {nfo_data['source']})")
+            _log("INFO", f"🚀 TIER 2 - Found NFOGuard data in NFO file: {nfo_data['dateadded']} (source: {nfo_data['source']})")
             dateadded = nfo_data["dateadded"]
             source = nfo_data["source"] 
             released = nfo_data.get("released")
             
+            # Cache NFO data in database for future lookups  
+            # Fixed parameter order: imdb_id, released, dateadded, source
+            self.db.upsert_movie_dates(imdb_id, released, dateadded, source, True)
+            _log("INFO", f"✅ Cached NFO data in database for {imdb_id}")
+            
             # Update file mtimes if enabled (NFO is already correct)
             if config.fix_dir_mtimes and dateadded:
                 self.nfo_manager.update_movie_files_mtime(movie_path, dateadded)
             
-            _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source}) [nfo-only]")
+            _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source}) [nfo-cached]")
-            return
+            return "processed"
+            
+        # TIER 2.5: Check for any existing valid date data in NFO (even without lockdata marker)
+        # Only use NFO dates if prioritize_nfo is enabled, otherwise check external APIs first
+        existing_nfo_data = self._extract_any_valid_dates_from_nfo(nfo_path)
+        if existing_nfo_data and config.manual_scan_prioritize_nfo:
+            _log("INFO", f"🔍 TIER 2.5 - Found existing date data in NFO (no lockdata): {existing_nfo_data['dateadded']} (source: {existing_nfo_data['source']})")
+            _log("INFO", f"⚡ MANUAL_SCAN_PRIORITIZE_NFO=True - Using NFO date for speed")
+            dateadded = existing_nfo_data["dateadded"]
+            source = existing_nfo_data["source"]
+            released = existing_nfo_data.get("released")
+            
+            # Cache existing data in database and add proper NFOGuard formatting
+            # Fixed parameter order: imdb_id, released, dateadded, source
+            self.db.upsert_movie_dates(imdb_id, released, dateadded, source, True)
+            _log("INFO", f"✅ Cached existing NFO data in database for {imdb_id}")
+            
+            # Update NFO file to add NFOGuard formatting (lockdata, comment)
+            if config.manage_nfo:
+                self.nfo_manager.create_movie_nfo(
+                    movie_path, imdb_id, dateadded, released, source, config.lock_metadata
+                )
+                _log("INFO", f"✅ Added NFOGuard formatting to existing NFO for {imdb_id}")
+            
+            # Update file mtimes if enabled
+            if config.fix_dir_mtimes and dateadded:
+                self.nfo_manager.update_movie_files_mtime(movie_path, dateadded)
+            
+            _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source}) [existing-nfo-enhanced]")
+            return "processed"
+        elif existing_nfo_data:
+            _log("INFO", f"🔍 TIER 2.5 - Found existing date data in NFO (no lockdata): {existing_nfo_data['dateadded']} (source: {existing_nfo_data['source']})")
+            _log("INFO", f"🎯 MANUAL_SCAN_PRIORITIZE_NFO=False - Will verify against external APIs first")
+            # Store NFO data as fallback but continue to TIER 3 to check external APIs
+            nfo_fallback_data = existing_nfo_data
             
         # TIER 1.5: Special handling for TMDB-only movies - extract dates from existing NFO
         if is_tmdb_fallback:
@@ -153,66 +306,30 @@ class MovieProcessor:
                 self.db.upsert_movie_dates(imdb_id, released, dateadded, source, True)
                 
                 _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source}) [tmdb-nfo]")
-                return
+                return "processed"
             
-        # TIER 2: Check database for existing data
+        # TIER 3: No cached data found - proceed with API lookups and verification
-        existing = self.db.get_movie_dates(imdb_id)
-        _log("DEBUG", f"Database lookup for {imdb_id}: {existing}")
         
-        # If we have complete data in database, use it and skip expensive API calls
+        # TIER 3: No cached data found - determine if we should query APIs
-        if existing and existing.get("dateadded") and existing.get("source") != "no_valid_date_source":
-            _log("INFO", f"✅ Using complete database data for {imdb_id}: {existing['dateadded']} (source: {existing['source']})")
-            # Still create NFO and update files but skip API queries
-            dateadded, source, released = existing["dateadded"], existing["source"], existing.get("released")
-            
-            # Create NFO with existing data
-            if config.manage_nfo:
-                self.nfo_manager.create_movie_nfo(
-                    movie_path, imdb_id, dateadded, released, source, config.lock_metadata
-                )
-            
-            # Update file mtimes if enabled
-            if config.fix_dir_mtimes and dateadded:
-                self.nfo_manager.update_movie_files_mtime(movie_path, dateadded)
-            
-            _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source}) [database-only]")
-            return
-        
-        # Handle webhook mode - prioritize database, then use proper date logic
         if webhook_mode:
-            _log("DEBUG", f"Webhook mode: existing={bool(existing)}, has_dateadded={bool(existing and existing.get('dateadded')) if existing else 'N/A'}")
+            _log("INFO", f"Webhook processing - no cached data found, using full date decision logic")
-            if existing and existing.get("dateadded"):
+            should_query = True  # Always query for webhooks when no cached data exists
-                _log("INFO", f"Webhook processing - using existing database entry: {existing['dateadded']} (source: {existing.get('source', 'unknown')})")
-                dateadded, source, released = existing["dateadded"], existing["source"], existing.get("released")
-            else:
-                if existing:
-                    _log("INFO", f"Webhook processing - database entry exists but no dateadded field: {existing}")
-                else:
-                    _log("INFO", f"Webhook processing - no database entry found for {imdb_id}")
-                _log("INFO", f"Using full date decision logic")
-                # Use same logic as manual scan to check Radarr import dates, release dates, etc.
-                should_query = True  # Always query for webhooks when no database entry exists
-                dateadded, source, released = self._decide_movie_dates(imdb_id, movie_path, should_query, existing)
-                
-                # Only if ALL date sources fail, fall back to current timestamp
-                if dateadded is None:
-                    local_tz = _get_local_timezone()
-                    current_time = datetime.now(local_tz).isoformat(timespec="seconds")
-                    _log("INFO", f"Webhook processing - all date sources failed, using current timestamp as last resort: {current_time}")
-                    dateadded, source = current_time, "webhook:fallback_timestamp"
         else:
             # Manual scan mode - determine if we should query APIs
-            should_query = (
+            should_query = config.movie_poll_mode == "always"
-                config.movie_poll_mode == "always" or
+            _log("DEBUG", f"Movie {imdb_id}: should_query={should_query}, poll_mode={config.movie_poll_mode}")
-                (config.movie_poll_mode == "if_missing" and not existing) or
-                (config.movie_poll_mode == "if_missing" and existing and existing.get("source") == "file:mtime") or
-                (config.movie_poll_mode == "if_missing" and existing and not existing.get("dateadded"))
-            )
         
-            _log("DEBUG", f"Movie {imdb_id}: should_query={should_query}, poll_mode={config.movie_poll_mode}, existing={bool(existing)}, has_dateadded={bool(existing and existing.get('dateadded')) if existing else False}")
+        # Use existing movie date decision logic
+        # Pass NFO fallback data if available for cases where external APIs don't have import history
+        nfo_fallback = locals().get('nfo_fallback_data', None)
+        dateadded, source, released = self._decide_movie_dates(imdb_id, movie_path, should_query, nfo_fallback)
         
-            # Use existing movie date decision logic
+        # Webhook fallback: if ALL date sources fail, use current timestamp
-            dateadded, source, released = self._decide_movie_dates(imdb_id, movie_path, should_query, existing)
+        if webhook_mode and dateadded is None:
+            local_tz = _get_local_timezone()
+            current_time = datetime.now(local_tz).isoformat(timespec="seconds")
+            _log("INFO", f"Webhook processing - all date sources failed, using current timestamp as last resort: {current_time}")
+            dateadded, source = current_time, "webhook:fallback_timestamp"
         
         # If we don't have an import/download date but we have a release date, use it as dateadded
         # This ensures we save digital release dates, theatrical dates, etc. to the database
@@ -225,16 +342,20 @@ class MovieProcessor:
             _log("INFO", f"Using release date as dateadded: {final_dateadded} (source: {final_source})")
         
         # Create NFO regardless of date availability (preserves existing metadata)
+        print(f"🔍 TIER3 - config.manage_nfo: {config.manage_nfo}")
         if config.manage_nfo:
+            print(f"🔍 TIER3 - Calling create_movie_nfo with final_dateadded: {final_dateadded}")
             self.nfo_manager.create_movie_nfo(
                 movie_path, imdb_id, final_dateadded, released, final_source, config.lock_metadata
             )
+        else:
+            print(f"❌ TIER3 - manage_nfo is disabled, skipping NFO creation")
         
         # Skip remaining processing if no valid date found and file dates disabled
         if final_dateadded is None:
             _log("WARNING", f"Movie {movie_path.name} - no valid date source available, but NFO was still processed")
             self.db.upsert_movie_dates(imdb_id, released, None, source, True)
-            return
+            return "processed"
             
         # Update dateadded and source for the rest of processing
         dateadded = final_dateadded
@@ -248,6 +369,10 @@ class MovieProcessor:
         
         _log("DEBUG", f"Movie processing reached file mtime section: fix_dir_mtimes={config.fix_dir_mtimes}, dateadded={dateadded}")
         
+        # Yield control briefly during movie processing to allow web interface requests
+        import time
+        time.sleep(0.005)  # 5ms yield per movie to improve responsiveness
+        
         # Save to database
         _log("DEBUG", f"About to save to database: imdb_id={imdb_id}, dateadded={dateadded}")
         try:
@@ -258,6 +383,7 @@ class MovieProcessor:
             raise
         
         _log("INFO", f"Completed processing movie: {movie_path.name} (source: {source})")
+        return "processed"
     
     def _extract_dates_from_tmdb_nfo(self, nfo_path: Path) -> Optional[Dict[str, str]]:
         """Extract date information from TMDB-based NFO file"""
@@ -284,6 +410,61 @@ class MovieProcessor:
             
         return None
     
+    def _extract_any_valid_dates_from_nfo(self, nfo_path: Path) -> Optional[Dict[str, str]]:
+        """Extract any valid date information from NFO file, even without NFOGuard markers"""
+        if not nfo_path.exists():
+            return None
+            
+        try:
+            import xml.etree.ElementTree as ET
+            tree = ET.parse(nfo_path)
+            root = tree.getroot()
+            
+            # Look for dateadded element (indicates previously processed by NFOGuard or similar)
+            dateadded_elem = root.find('.//dateadded')
+            premiered_elem = root.find('.//premiered')
+            
+            if dateadded_elem is not None and dateadded_elem.text:
+                dateadded = dateadded_elem.text.strip()
+                
+                # Try to determine source from NFOGuard comment if present
+                source = "existing_nfo_data"
+                try:
+                    nfo_content = nfo_path.read_text(encoding='utf-8')
+                    import re
+                    # Look for NFOGuard comment pattern
+                    source_match = re.search(r'<!--.*?NFOGuard.*?Source:\s*([^-\s]+).*?-->', nfo_content, re.DOTALL | re.IGNORECASE)
+                    if source_match:
+                        source = source_match.group(1).strip()
+                        _log("DEBUG", f"Found source in NFOGuard comment: {source}")
+                    else:
+                        # Try to infer source from dateadded format/content
+                        if "tmdb" in nfo_content.lower() or (premiered_elem and premiered_elem.text):
+                            source = "tmdb:digital"
+                        elif "radarr" in nfo_content.lower():
+                            source = "radarr:db.history.import"
+                        else:
+                            source = "existing_nfo_data"
+                        _log("DEBUG", f"Inferred source from NFO content: {source}")
+                except Exception as e:
+                    _log("DEBUG", f"Could not determine source from NFO content: {e}")
+                
+                result = {
+                    "dateadded": dateadded,
+                    "source": source
+                }
+                
+                if premiered_elem is not None and premiered_elem.text:
+                    result["released"] = premiered_elem.text.strip()
+                
+                _log("INFO", f"✅ Found existing date data in NFO: dateadded={dateadded}, source={source}")
+                return result
+                
+        except (ET.ParseError, Exception) as e:
+            _log("DEBUG", f"Error parsing NFO for existing date data: {e}")
+            
+        return None
+    
     def _decide_movie_dates(self, imdb_id: str, movie_path: Path, should_query: bool, existing: Optional[Dict]) -> Tuple[str, str, Optional[str]]:
         """Decide movie dates based on configuration and available data"""
         _log("DEBUG", f"_decide_movie_dates for {imdb_id}: should_query={should_query}, existing={existing}")
@@ -372,6 +553,11 @@ class MovieProcessor:
                         local_import_date = convert_utc_to_local(import_date)
                         return local_import_date, import_source, released
         
+        # Last resort: check if we have NFO fallback data (when external APIs don't have import history)
+        if existing and existing.get('dateadded'):
+            _log("INFO", f"✅ Movie {imdb_id}: External APIs don't have import history, using NFO fallback date: {existing['dateadded']} (source: {existing['source']})")
+            return existing["dateadded"], f"nfo_fallback:{existing['source']}", existing.get("released")
+        
         # Last resort: file mtime (if allowed)
         if config.allow_file_date_fallback:
             return self._get_file_mtime_date(movie_path)

processors/tv_processor.py

@@ -4,6 +4,7 @@ Handles TV series processing and episode management with async I/O support
 """
 import os
 import re
+import time
 import asyncio
 from pathlib import Path
 from typing import Optional, Dict, List, Set, Tuple, Any
@@ -52,28 +53,145 @@ class TVProcessor:
             path_mapper=self.path_mapper
         )
     
-    def process_series(self, series_path: Path) -> None:
+    def should_skip_series_fast(self, imdb_id: str, series_name: str = "") -> Tuple[bool, str, int]:
+        """
+        Fast preliminary check to skip series without filesystem scan
+        
+        Args:
+            imdb_id: Series IMDb ID
+            series_name: Series name for logging
+            
+        Returns:
+            (should_skip: bool, reason: str, episodes_in_db: int)
+        """
+        try:
+            with self.db.get_connection() as conn:
+                cursor = conn.cursor()
+                
+                # Check if we have complete episodes in database
+                cursor.execute("""
+                    SELECT 
+                        COUNT(*) as total_in_db,
+                        COUNT(CASE WHEN dateadded IS NOT NULL AND source IS NOT NULL AND source != 'unknown' AND source != 'no_valid_date_source' THEN 1 END) as complete_episodes
+                    FROM episodes 
+                    WHERE imdb_id = %s
+                """, (imdb_id,))
+                
+                result = cursor.fetchone()
+                if not result:
+                    return False, "No database records found", 0
+                
+                total_in_db = result['total_in_db']
+                complete_episodes = result['complete_episodes']
+                
+                # Skip if we have episodes and all are complete
+                # We'll verify disk count later if needed
+                if total_in_db > 0 and complete_episodes == total_in_db:
+                    return True, f"Likely complete: {complete_episodes} episodes in DB all have valid dates", total_in_db
+                else:
+                    return False, f"Needs checking: {complete_episodes}/{total_in_db} episodes complete in DB", total_in_db
+                    
+        except Exception as e:
+            _log("ERROR", f"Error in fast series check for {imdb_id}: {e}")
+            return False, f"Error in fast check: {e}", 0
+
+    def should_skip_series(self, imdb_id: str, episodes_on_disk: int, series_name: str = "") -> Tuple[bool, str]:
+        """
+        Determine if we should skip processing this series based on completion status
+        
+        Args:
+            imdb_id: Series IMDb ID
+            episodes_on_disk: Number of episodes found on disk
+            series_name: Series name for logging
+            
+        Returns:
+            (should_skip: bool, reason: str)
+        """
+        try:
+            with self.db.get_connection() as conn:
+                cursor = conn.cursor()
+                
+                # PostgreSQL-only query
+                cursor.execute("""
+                    SELECT 
+                        COUNT(*) as total_in_db,
+                        COUNT(CASE WHEN dateadded IS NOT NULL AND source IS NOT NULL AND source != 'unknown' AND source != 'no_valid_date_source' THEN 1 END) as complete_episodes
+                    FROM episodes 
+                    WHERE imdb_id = %s
+                """, (imdb_id,))
+                
+                result = cursor.fetchone()
+                if not result:
+                    return False, "No database records found"
+                
+                # PostgreSQL RealDictCursor returns dict-like objects
+                total_in_db = result['total_in_db']
+                complete_episodes = result['complete_episodes']
+                
+                # Skip if:
+                # 1. We have episodes in database
+                # 2. Database count matches disk count (no missing episodes)
+                # 3. All episodes have valid dates and sources
+                if total_in_db > 0 and total_in_db == episodes_on_disk and complete_episodes == episodes_on_disk:
+                    return True, f"Complete: {complete_episodes}/{episodes_on_disk} episodes have valid dates"
+                elif total_in_db == 0:
+                    return False, f"New series: No episodes in database"
+                elif total_in_db != episodes_on_disk:
+                    return False, f"Disk mismatch: {total_in_db} in DB vs {episodes_on_disk} on disk"
+                else:
+                    return False, f"Incomplete: {complete_episodes}/{episodes_on_disk} episodes have valid dates"
+                    
+        except Exception as e:
+            _log("ERROR", f"Error checking series completion for {imdb_id}: {e}")
+            return False, f"Error checking completion: {e}"
+    
+    def process_series(self, series_path: Path, force_scan: bool = False) -> str:
         """Process a TV series directory"""
         imdb_id = self.nfo_manager.parse_imdb_from_path(series_path)
         if not imdb_id:
             _log("ERROR", f"No IMDb ID found in series path: {series_path}")
-            return
+            return "error"
         
         _log("INFO", f"Processing TV series: {series_path.name}")
         
+        # Fast check first - avoid expensive filesystem scan if possible
+        if not force_scan:
+            should_skip_fast, reason_fast, episodes_in_db = self.should_skip_series_fast(imdb_id, series_path.name)
+            if should_skip_fast:
+                _log("INFO", f"⚡ FAST SKIP: {series_path.name} [{imdb_id}] - {reason_fast}")
+                # Still update the series record to track that we've seen it
+                self.db.upsert_series(imdb_id, str(series_path))
+                return "skipped"
+        
+        # Need filesystem scan - either force_scan=True or series not complete in DB
+        disk_episodes = find_episodes_on_disk(series_path)
+        _log("INFO", f"Found {len(disk_episodes)} episodes on disk")
+        
+        # Final skip check with actual episode count (unless forced)
+        if not force_scan:
+            should_skip, reason = self.should_skip_series(imdb_id, len(disk_episodes), series_path.name)
+            if should_skip:
+                _log("INFO", f"⏭️ SKIPPING SERIES: {series_path.name} [{imdb_id}] - {reason}")
+                # Still update the series record to track that we've seen it
+                self.db.upsert_series(imdb_id, str(series_path))
+                return "skipped"
+            else:
+                _log("INFO", f"📺 PROCESSING SERIES: {series_path.name} [{imdb_id}] - {reason}")
+        else:
+            _log("INFO", f"🔄 FORCE PROCESSING SERIES: {series_path.name} [{imdb_id}] - Force scan enabled")
+        
         # Update database
         self.db.upsert_series(imdb_id, str(series_path))
         
-        # Find video files
-        disk_episodes = find_episodes_on_disk(series_path)
-        _log("INFO", f"Found {len(disk_episodes)} episodes on disk")
-        
         # Get episode dates
         episode_dates = self._gather_episode_dates(series_path, imdb_id, disk_episodes)
         
-        # Process episodes
+        # Process episodes with periodic yielding for non-blocking operation
+        episode_count = 0
         for (season, episode), (aired, dateadded, source) in episode_dates.items():
             if (season, episode) in disk_episodes:
+                episode_count += 1
+                
                 # Create NFO
                 if config.manage_nfo:
                     season_dir = series_path / config.tv_season_dir_format.format(season=season)
@@ -89,12 +207,24 @@ class TVProcessor:
                         self.nfo_manager.set_file_mtime(video_file, dateadded)
                 
                 # Save to database
-                self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                try:
+                    self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                    _log("DEBUG", f"S{season:02d}E{episode:02d}: Database record saved successfully")
+                except Exception as e:
+                    _log("ERROR", f"S{season:02d}E{episode:02d}: Database write failed: {e}")
+                    # Continue processing other episodes
+                
+                # Yield control every 3 episodes to allow other requests (webhooks, web interface)
+                if episode_count % 3 == 0:
+                    import time
+                    time.sleep(0.1)  # 100ms yield to improve responsiveness during episode processing
+                    _log("DEBUG", f"Processed {episode_count} episodes, yielding to allow other requests...")
         
         # Skip season.nfo and tvshow.nfo creation - focus only on episode NFOs
         pass
         
         _log("INFO", f"Completed processing TV series: {series_path.name}")
+        return "processed"
     
     def _extract_series_title_from_path(self, series_path: Path) -> Optional[str]:
         """Extract series title from directory path using unified file utilities"""
@@ -102,67 +232,251 @@ class TVProcessor:
     
     
     def _gather_episode_dates(self, series_path: Path, imdb_id: str, disk_episodes: Dict[Tuple[int, int], List[Path]]) -> Dict[Tuple[int, int], Tuple[Optional[str], Optional[str], str]]:
-        """Gather episode air dates and date added information"""
+        """Gather episode air dates and date added information with database-first optimization"""
+        _log("INFO", f"🎯 GATHERING EPISODE DATES for {imdb_id}: {len(disk_episodes)} episodes on disk")
         episode_dates = {}
+        episodes_needing_lookup = []
         
-        # Get data from Sonarr first if available
+        # TIER 1: Check database first for existing dates (fastest)
-        sonarr_episodes = self._get_sonarr_episodes(imdb_id)
+        _log("DEBUG", f"TIER 1 - Checking database for existing episode dates for {len(disk_episodes)} episodes")
+        db_cache_hits = 0
+        episodes_needing_nfo_check = []
         
-        # Get data from external sources for missing information
         for (season, episode) in disk_episodes:
-            aired = None
+            # Try database first - this is much faster than API calls
-            dateadded = None
+            db_result = self.db.get_episode_date(imdb_id, season, episode)
-            source = "unknown"
             
-            # Try Sonarr first
+            if db_result and db_result.get('dateadded'):
-            if (season, episode) in sonarr_episodes:
+                # Found in database - use cached data
-                sonarr_data = sonarr_episodes[(season, episode)]
+                aired = db_result.get('aired')
-                aired = sonarr_data.get('airDate')
+                dateadded = db_result.get('dateadded')
-                dateadded = sonarr_data.get('dateAdded')
+                source = db_result.get('source', 'database_cache')
-                if dateadded:
+                episode_dates[(season, episode)] = (aired, dateadded, source)
-                    source = "sonarr"
+                db_cache_hits += 1
+                _log("DEBUG", f"Database cache hit for S{season:02d}E{episode:02d}: {dateadded}")
+            else:
+                # Not in database or incomplete - needs NFO check
+                episodes_needing_nfo_check.append((season, episode))
         
-            # Fallback to external sources if needed
+        _log("INFO", f"Database cache hits: {db_cache_hits}/{len(disk_episodes)} episodes. Need NFO check: {len(episodes_needing_nfo_check)}")
-            if not aired:
-                external_aired = self.external_clients.get_episode_air_date(imdb_id, season, episode)
-                if external_aired:
-                    aired = external_aired
-                    if not dateadded:
-                        source = "external"
         
-            # Use air date as fallback for dateadded if available
+        # TIER 2: Check NFO files for NFOGuard dates and cache them in database
-            if not dateadded and aired:
+        nfo_cache_hits = 0
-                dateadded = aired
+        episodes_needing_lookup = []
-                source = f"{source}_fallback" if source != "unknown" else "aired_fallback"
         
-            episode_dates[(season, episode)] = (aired, dateadded, source)
+        if episodes_needing_nfo_check:
+            _log("DEBUG", f"TIER 2 - Checking NFO files for NFOGuard dates for {len(episodes_needing_nfo_check)} episodes")
+            
+            for (season, episode) in episodes_needing_nfo_check:
+                # Look for existing NFO files for this episode
+                season_dir = series_path / config.tv_season_dir_format.format(season=season)
+                episode_files = disk_episodes[(season, episode)]
+                
+                nfo_found = False
+                for episode_file in episode_files:
+                    # Try to find matching NFO file
+                    nfo_path = episode_file.with_suffix('.nfo')
+                    if nfo_path.exists():
+                        # Extract NFOGuard data from episode NFO
+                        nfo_data = self.nfo_manager.extract_nfoguard_dates_from_nfo(nfo_path)
+                        if nfo_data:
+                            aired = nfo_data.get('aired') 
+                            dateadded = nfo_data.get('dateadded')
+                            source = nfo_data.get('source', 'nfo_cache')
+                            
+                            _log("DEBUG", f"S{season:02d}E{episode:02d}: NFO data found - aired={aired}, dateadded={dateadded}, source={source}")
+                            
+                            # Skip incomplete NFO files with "unknown" source or no useful dates
+                            if source == "unknown" and not dateadded and not aired:
+                                _log("INFO", f"S{season:02d}E{episode:02d}: Ignoring incomplete NFO file with source 'unknown' and no dates")
+                                break  # Break out of NFO search to mark episode as needing API lookup
+                            
+                            # Apply fallback logic if NFO has aired but no dateadded
+                            if not dateadded and aired:
+                                dateadded = aired
+                                source = f"{source}_aired_fallback" if source != 'nfo_cache' else 'nfo_aired_fallback'
+                                _log("DEBUG", f"S{season:02d}E{episode:02d}: NFO has aired but no dateadded, using aired as fallback: {dateadded}")
+                            
+                            if dateadded:
+                                episode_dates[(season, episode)] = (aired, dateadded, source)
+                                nfo_cache_hits += 1
+                                nfo_found = True
+                                
+                                # Cache NFO data in database for future lookups
+                                self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                                _log("DEBUG", f"NFO cache hit for S{season:02d}E{episode:02d}: {dateadded} - cached in DB")
+                                break
+                
+                if not nfo_found:
+                    # No NFO data found - needs API lookup
+                    episodes_needing_lookup.append((season, episode))
+            
+            _log("INFO", f"NFO cache hits: {nfo_cache_hits}/{len(episodes_needing_nfo_check)} episodes. Need API lookup: {len(episodes_needing_lookup)}")
+        
+        # TIER 3: Only call Sonarr API for episodes not in database or NFO files
+        if episodes_needing_lookup:
+            _log("DEBUG", f"TIER 3 - Querying Sonarr for {len(episodes_needing_lookup)} episodes missing from database and NFO files")
+            sonarr_episodes = self._get_sonarr_episodes(imdb_id, episodes_needing_lookup)
+            
+            # Process episodes that needed lookup
+            for (season, episode) in episodes_needing_lookup:
+                aired = None
+                dateadded = None
+                source = "unknown"
+                
+                # Try Sonarr first
+                if (season, episode) in sonarr_episodes:
+                    sonarr_data = sonarr_episodes[(season, episode)]
+                    aired = sonarr_data.get('airDate')
+                    dateadded = sonarr_data.get('dateAdded')
+                    if dateadded:
+                        source = "sonarr:history.import"
+                        _log("DEBUG", f"S{season:02d}E{episode:02d}: Got Sonarr import date: {dateadded}")
+                    else:
+                        # Sonarr has episode data but no import date - update source for better fallback handling
+                        source = "sonarr:no_import_date"
+                        _log("DEBUG", f"S{season:02d}E{episode:02d}: Sonarr has data but no dateAdded (aired: {aired})")
+                
+                # Fallback to external sources if needed
+                if not aired:
+                    _log("DEBUG", f"S{season:02d}E{episode:02d}: No aired date from Sonarr, trying external APIs for {imdb_id}")
+                    external_aired = self.external_clients.get_episode_air_date(imdb_id, season, episode)
+                    if external_aired:
+                        aired = external_aired
+                        if not dateadded:
+                            source = "external"
+                        _log("INFO", f"S{season:02d}E{episode:02d}: Found aired date from external APIs: {aired}")
+                    else:
+                        _log("WARNING", f"S{season:02d}E{episode:02d}: No aired date found from external APIs for {imdb_id}")
+                
+                # Use air date as fallback for dateadded if no import date found
+                if not dateadded and aired:
+                    # Always use air date as fallback when no import date is available
+                    dateadded = aired
+                    if source == "sonarr:no_import_date":
+                        source = "sonarr:aired_fallback"
+                    elif source == "sonarr:history.import":
+                        # This shouldn't happen but handle it gracefully
+                        source = "sonarr:aired_fallback"
+                    else:
+                        source = f"{source}_fallback" if source != "unknown" else "aired_fallback"
+                    _log("DEBUG", f"S{season:02d}E{episode:02d}: Using aired date as fallback: {dateadded} (source: {source})")
+                
+                # Ensure air date is saved to database even if used as dateadded fallback
+                if aired and not dateadded:
+                    # This is a fallback for cases where we have air date but absolutely no dateadded
+                    dateadded = aired
+                    source = "aired_only_fallback"
+                    _log("INFO", f"S{season:02d}E{episode:02d}: No import date found, using air date as both aired and dateadded: {dateadded}")
+                
+                episode_dates[(season, episode)] = (aired, dateadded, source)
+        
+        _log("INFO", f"🎯 EPISODE DATES GATHERED: {len(episode_dates)} episodes with dates")
+        for (s, e), (aired, dateadded, source) in episode_dates.items():
+            _log("INFO", f"   S{s:02d}E{e:02d}: aired={aired}, dateadded={dateadded}, source={source}")
         
         return episode_dates
     
-    def _get_sonarr_episodes(self, imdb_id: str) -> Dict[Tuple[int, int], Dict[str, Any]]:
+    def _get_sonarr_episodes(self, imdb_id: str, episodes_filter: List[Tuple[int, int]] = None) -> Dict[Tuple[int, int], Dict[str, Any]]:
-        """Get episode information from Sonarr"""
+        """Get episode information from Sonarr including import history - optimized to only fetch needed episodes"""
         try:
-            series_data = self.sonarr.get_series_by_imdb(imdb_id)
+            series_data = self.sonarr.series_by_imdb(imdb_id)
             if not series_data:
-                return {}
+                # Try fuzzy matching if exact IMDb lookup fails
+                _log("DEBUG", f"Exact IMDb lookup failed for {imdb_id}, trying fuzzy matching")
+                
+                # Get all series and try fuzzy matching
+                all_series = self.sonarr.get_all_series()
+                if all_series:
+                    _log("DEBUG", f"Found {len(all_series)} total series in Sonarr")
+                    
+                    for series in all_series:
+                        series_imdb = series.get('imdbId', '')
+                        if series_imdb and series_imdb.startswith('tt'):
+                            # Try fuzzy matching for IMDb numbers
+                            try:
+                                target_imdb_num = imdb_id.replace('tt', '').lower()
+                                series_imdb_num = series_imdb.replace('tt', '').lower()
+                                
+                                target_num = int(target_imdb_num)
+                                series_num = int(series_imdb_num)
+                                diff = abs(target_num - series_num)
+                                
+                                if diff <= 10:  # Allow small IMDb ID differences
+                                    _log("INFO", f"✅ Found fuzzy IMDb match: {series_imdb} vs {imdb_id} (diff: {diff})")
+                                    _log("DEBUG", f"Series data found: True")
+                                    _log("DEBUG", f"Found series '{series.get('title', 'Unknown')}' with ID {series.get('id')}")
+                                    series_data = series
+                                    break
+                            except (ValueError, TypeError):
+                                continue
+                
+                if not series_data:
+                    return {}
             
             series_id = series_data.get('id')
             if not series_id:
                 return {}
             
-            episodes = self.sonarr.get_episodes(series_id)
+            episodes = self.sonarr.episodes_for_series(series_id)
+            
+            # Convert episodes_filter to set for faster lookup
+            filter_set = set(episodes_filter) if episodes_filter else None
             
             episode_map = {}
+            api_calls_made = 0
+            episodes_processed = 0
+            
             for episode in episodes:
                 season = episode.get('seasonNumber', 0)
                 episode_num = episode.get('episodeNumber', 0)
                 
-                if season > 0 and episode_num > 0:
+                # Skip episodes not in filter if filter is provided
-                    episode_map[(season, episode_num)] = {
+                if filter_set and (season, episode_num) not in filter_set:
+                    continue
+                
+                if season >= 0 and episode_num > 0:
+                    episodes_processed += 1
+                    
+                    # Get basic episode info
+                    episode_data = {
                         'airDate': episode.get('airDate'),
-                        'dateAdded': episode.get('episodeFile', {}).get('dateAdded') if episode.get('hasFile') else None
+                        'dateAdded': None
                     }
                     
+                    # First try to get import date from history (more accurate)
+                    episode_id = episode.get('id')
+                    if episode_id and episode.get('hasFile'):
+                        import_date = self.sonarr.get_episode_import_history(episode_id)
+                        api_calls_made += 1
+                        if import_date:
+                            episode_data['dateAdded'] = import_date
+                            _log("DEBUG", f"Got import date from history for S{season:02d}E{episode_num:02d}: {import_date}")
+                        
+                        # Yield control every 5 API calls to allow other requests
+                        if api_calls_made % 5 == 0:
+                            import time
+                            time.sleep(0.01)  # 10ms yield to other processes
+                            _log("DEBUG", f"Yielded after {api_calls_made} Sonarr API calls to allow other requests...")
+                    
+                    # Fallback to episodeFile.dateAdded if history didn't work
+                    if not episode_data['dateAdded'] and episode.get('hasFile'):
+                        file_date = episode.get('episodeFile', {}).get('dateAdded')
+                        if file_date:
+                            episode_data['dateAdded'] = file_date
+                            _log("DEBUG", f"Got file date for S{season:02d}E{episode_num:02d}: {file_date}")
+                    
+                    episode_map[(season, episode_num)] = episode_data
+                    
+                    # Also yield control every 20 episodes processed to prevent blocking
+                    if episodes_processed % 20 == 0:
+                        import time
+                        time.sleep(0.01)  # 10ms yield for large episode lists
+                        _log("DEBUG", f"Processed {episodes_processed} episodes, yielding to allow other requests...")
+            
+            if filter_set:
+                _log("DEBUG", f"Made {api_calls_made} Sonarr history API calls for filtered episodes (instead of all episodes)")
+            
             return episode_map
             
         except Exception as e:
@@ -221,7 +535,12 @@ class TVProcessor:
                         self.nfo_manager.set_file_mtime(video_file, dateadded)
                 
                 # Save to database
-                self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                try:
+                    self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                    _log("DEBUG", f"S{season:02d}E{episode:02d}: Database record saved successfully")
+                except Exception as e:
+                    _log("ERROR", f"S{season:02d}E{episode:02d}: Database write failed: {e}")
+                    # Continue processing other episodes
                 processed_count += 1
         
         _log("INFO", f"Processed {processed_count} episodes in season {season_num}")
@@ -362,7 +681,12 @@ class TVProcessor:
                         mtime_operations.append((video_file, dateadded))
                 
                 # Save to database
-                self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                try:
+                    self.db.upsert_episode_date(imdb_id, season, episode, aired, dateadded, source, True)
+                    _log("DEBUG", f"S{season:02d}E{episode:02d}: Database record saved successfully")
+                except Exception as e:
+                    _log("ERROR", f"S{season:02d}E{episode:02d}: Database write failed: {e}")
+                    # Continue processing other episodes
         
         # Process NFOs and mtimes concurrently
         results = {}
@@ -439,6 +763,271 @@ class TVProcessor:
         
         return processed_results
     
+    def process_webhook_episodes(self, series_path: Path, webhook_episodes: List[Dict[str, Any]]) -> None:
+        """Process only the specific episodes mentioned in a webhook (targeted mode)"""
+        imdb_id = self.nfo_manager.parse_imdb_from_path(series_path)
+        if not imdb_id:
+            _log("ERROR", f"No IMDb ID found in series path: {series_path}")
+            return
+        
+        if not webhook_episodes:
+            _log("WARNING", f"No episodes in webhook, falling back to series processing: {series_path}")
+            self.process_series(series_path)
+            return
+        
+        _log("INFO", f"Processing {len(webhook_episodes)} webhook episodes for: {series_path.name} (IMDb: {imdb_id})")
+        
+        # Update database
+        self.db.upsert_series(imdb_id, str(series_path))
+        
+        # Get enhanced metadata from Sonarr
+        series_metadata = self._get_sonarr_series_metadata(imdb_id)
+        
+        episodes_processed = 0
+        for webhook_episode in webhook_episodes:
+            season_num = webhook_episode.get("seasonNumber")
+            episode_num = webhook_episode.get("episodeNumber")
+            
+            if not season_num or not episode_num:
+                _log("WARNING", f"Invalid episode data in webhook: {webhook_episode}")
+                continue
+            
+            # Check if episode file exists on disk
+            season_dir = series_path / config.tv_season_dir_format.format(season=season_num)
+            if not season_dir.exists():
+                _log("WARNING", f"Season directory not found: {season_dir}")
+                continue
+                
+            # Find matching episode files
+            episode_files = []
+            for file_path in season_dir.iterdir():
+                if file_path.is_file() and file_path.suffix.lower() in ('.mkv', '.mp4', '.avi', '.mov', '.m4v'):
+                    parsed = self._parse_episode_from_filename(file_path.name)
+                    if parsed and parsed == (season_num, episode_num):
+                        episode_files.append(file_path)
+            
+            if not episode_files:
+                _log("WARNING", f"No video files found for S{season_num:02d}E{episode_num:02d}")
+                continue
+                
+            # Get episode date information - webhook processing prioritizes existing DB entries
+            _log("DEBUG", f"Processing webhook episode: IMDb={imdb_id}, S{season_num:02d}E{episode_num:02d}")
+            aired, dateadded, source = self._get_webhook_episode_date(imdb_id, season_num, episode_num, series_metadata)
+            enhanced_metadata = self._get_episode_metadata(series_metadata, season_num, episode_num, season_dir) if series_metadata else self._get_episode_metadata(None, season_num, episode_num, season_dir)
+            
+            # Create NFO
+            if config.manage_nfo:
+                self.nfo_manager.create_episode_nfo(
+                    season_dir,
+                    season_num, episode_num, aired, dateadded, source, config.lock_metadata,
+                    enhanced_metadata
+                )
+            
+            # Update file mtimes
+            if config.fix_dir_mtimes and dateadded:
+                for episode_file in episode_files:
+                    self.nfo_manager.set_file_mtime(episode_file, dateadded)
+            
+            # Save to database
+            self.db.upsert_episode_date(imdb_id, season_num, episode_num, aired, dateadded, source, True)
+            
+            # Verify database entry was saved (debug)
+            verification = self.db.get_episode_date(imdb_id, season_num, episode_num)
+            if verification:
+                _log("DEBUG", f"Verified database entry saved: S{season_num:02d}E{episode_num:02d} -> {verification['dateadded']}")
+            else:
+                _log("ERROR", f"Failed to save episode to database: S{season_num:02d}E{episode_num:02d}")
+            
+            episodes_processed += 1
+        
+        # Create season/tvshow NFOs if any episodes were processed
+        if episodes_processed > 0 and config.manage_nfo:
+            seasons_processed = set()
+            for webhook_episode in webhook_episodes:
+                season_num = webhook_episode.get("seasonNumber")
+                if season_num and season_num not in seasons_processed:
+                    season_dir = series_path / config.tv_season_dir_format.format(season=season_num)
+                    if season_dir.exists():
+                        self.nfo_manager.create_season_nfo(season_dir, season_num)
+                        seasons_processed.add(season_num)
+            
+            # Get TVDB ID for better Emby compatibility
+            tvdb_id = self.external_clients.get_tvdb_series_id(imdb_id)
+            self.nfo_manager.create_tvshow_nfo(series_path, imdb_id, tvdb_id)
+        
+        _log("INFO", f"Completed targeted processing: {episodes_processed}/{len(webhook_episodes)} episodes processed")
+    
+    def _parse_episode_from_filename(self, filename: str) -> Optional[Tuple[int, int]]:
+        """Parse season and episode numbers from filename"""
+        # Try SxxExx format
+        match = re.search(r'S(\d{1,2})E(\d{1,2})', filename, re.IGNORECASE)
+        if match:
+            return int(match.group(1)), int(match.group(2))
+        
+        # Try season.episode format
+        match = re.search(r'(\d{1,2})\.(\d{1,2})', filename)
+        if match:
+            return int(match.group(1)), int(match.group(2))
+        
+        return None
+    
+    def _get_sonarr_series_metadata(self, imdb_id: str) -> Optional[Dict[str, Any]]:
+        """Get enhanced series metadata from Sonarr API"""
+        try:
+            if not self.sonarr.enabled:
+                return None
+            
+            series_data = self.sonarr.series_by_imdb(imdb_id)
+            if not series_data:
+                return None
+            
+            series_id = series_data.get('id')
+            if not series_id:
+                return None
+            
+            # Get all episodes for this series
+            episodes = self.sonarr.episodes_for_series(series_id)
+            
+            # Organize episodes by season/episode
+            episode_map = {}
+            for episode in episodes:
+                season = episode.get('seasonNumber', 0)
+                episode_num = episode.get('episodeNumber', 0)
+                
+                if season >= 0 and episode_num > 0:
+                    episode_map[(season, episode_num)] = episode
+            
+            return {
+                'series': series_data,
+                'episodes': episode_map
+            }
+            
+        except Exception as e:
+            _log("ERROR", f"Failed to get Sonarr series metadata for {imdb_id}: {e}")
+            return None
+    
+    def _get_episode_metadata(self, series_metadata: Optional[Dict[str, Any]], season_num: int, episode_num: int, season_dir: Optional[Path] = None) -> Optional[Dict[str, Any]]:
+        """Get enhanced episode metadata including title extraction from filename"""
+        _log("DEBUG", f"Getting episode metadata for S{season_num:02d}E{episode_num:02d}, season_dir: {season_dir}")
+        
+        metadata = {}
+        
+        # Try to get title from Sonarr first
+        if series_metadata and 'episodes' in series_metadata:
+            episode_data = series_metadata['episodes'].get((season_num, episode_num))
+            if episode_data:
+                title = episode_data.get('title')
+                if title and title != 'TBA':
+                    metadata['title'] = title
+                    _log("DEBUG", f"Got title from Sonarr for S{season_num:02d}E{episode_num:02d}: {title}")
+        
+        # If no title from Sonarr, try to extract from filename
+        if 'title' not in metadata and season_dir:
+            title = self._extract_title_from_filename(season_num, episode_num, season_dir)
+            if title:
+                metadata['title'] = title
+                _log("DEBUG", f"Extracted title from filename for S{season_num:02d}E{episode_num:02d}: {title}")
+        
+        return metadata if metadata else None
+    
+    def _extract_title_from_filename(self, season_num: int, episode_num: int, season_dir: Path) -> Optional[str]:
+        """Extract episode title from video filename using regex pattern"""
+        season_pattern = f"S{season_num:02d}E{episode_num:02d}"
+        
+        try:
+            # Look for video files in the season directory
+            for file_path in season_dir.iterdir():
+                if file_path.is_file() and file_path.suffix.lower() in ('.mkv', '.mp4', '.avi', '.mov', '.m4v'):
+                    filename = file_path.name
+                    
+                    # Check if this file matches our season/episode
+                    if season_pattern in filename.upper():
+                        # Extract title using regex pattern: S01E01-Title[WEBDL-1080p]
+                        match = re.search(rf'{season_pattern}-(.*?)\[', filename, re.IGNORECASE)
+                        if match:
+                            title = match.group(1)
+                            # Clean up the title
+                            title = title.replace('-', ' ').strip()
+                            if title:
+                                _log("DEBUG", f"Extracted title '{title}' from filename: {filename}")
+                                return title
+                            
+        except Exception as e:
+            _log("ERROR", f"Error extracting title from filename for S{season_num:02d}E{episode_num:02d}: {e}")
+        
+        return None
+    
+    def _get_webhook_episode_date(self, imdb_id: str, season_num: int, episode_num: int, series_metadata: Optional[Dict[str, Any]] = None) -> Tuple[Optional[str], Optional[str], str]:
+        """
+        Get episode date for webhook processing - avoid treating webhook as gospel.
+        
+        Logic:
+        1. Check Sonarr import history FIRST (any history = show has existed for a while)
+        2. If ANY import history exists → Use air date (webhook is likely upgrade/rename)
+        3. If NO import history → Use webhook date (truly new show)
+        
+        This prevents upgrades from overriding dates for shows you've had for months/years.
+        """
+        # Get aired date and episode ID from Sonarr
+        aired = None
+        episode_id = None
+        if series_metadata and 'episodes' in series_metadata:
+            episode_data = series_metadata['episodes'].get((season_num, episode_num))
+            if episode_data:
+                aired = episode_data.get('airDate')
+                episode_id = episode_data.get('id')
+                _log("DEBUG", f"Got aired date from Sonarr for S{season_num:02d}E{episode_num:02d}: {aired}")
+        
+        # STEP 1: Check Sonarr import history FIRST (this is the key check)
+        _log("INFO", f"Checking Sonarr import history for S{season_num:02d}E{episode_num:02d} to detect import vs rename events")
+        
+        if episode_id and hasattr(self, 'sonarr'):
+            try:
+                _log("DEBUG", f"Calling get_episode_import_history for episode_id: {episode_id}")
+                import_history = self.sonarr.get_episode_import_history(episode_id)
+                _log("DEBUG", f"Import history result: {import_history}")
+                
+                if import_history:
+                    # Found actual import event - use this date
+                    _log("INFO", f"Found real import event for S{season_num:02d}E{episode_num:02d}: {import_history}")
+                    _log("INFO", f"Using first import date (not webhook): {import_history}")
+                    return aired, import_history, "sonarr:import_history"
+                else:
+                    # No import events found - this means only renames/moves exist in history
+                    # The episode was already in Sonarr, just being managed/renamed
+                    _log("INFO", f"No import events found for S{season_num:02d}E{episode_num:02d} - only renames/moves in history")
+                    
+                    if aired:
+                        _log("INFO", f"Using air date for existing episode (rename-only history): {aired}")
+                        return aired, aired + "T20:00:00", "airdate"
+                    else:
+                        _log("DEBUG", f"No air date available for rename-only episode S{season_num:02d}E{episode_num:02d}")
+                    
+            except Exception as e:
+                _log("ERROR", f"Error checking Sonarr import history for S{season_num:02d}E{episode_num:02d}: {e}")
+                import traceback
+                _log("ERROR", traceback.format_exc())
+        else:
+            if not episode_id:
+                _log("DEBUG", f"No episode_id found for S{season_num:02d}E{episode_num:02d}")
+            if not hasattr(self, 'sonarr'):
+                _log("DEBUG", f"No sonarr client available")
+        
+        # STEP 2: No import history found - this is likely a genuinely new show
+        # Check our database to avoid duplicates
+        existing = self.db.get_episode_date(imdb_id, season_num, episode_num)
+        if existing and existing.get('dateadded'):
+            _log("INFO", f"Episode S{season_num:02d}E{episode_num:02d} already exists in our database: {existing['dateadded']}")
+            return existing.get('aired'), existing.get('dateadded'), existing.get('source', 'nfoguard:database')
+        
+        # STEP 3: Truly new episode - use webhook date
+        dateadded = datetime.now().isoformat()
+        source = "sonarr:webhook"
+        
+        _log("INFO", f"No import history and not in database - using webhook date for genuinely new episode S{season_num:02d}E{episode_num:02d}: {dateadded}")
+        
+        return aired, dateadded, source
+    
     async def async_batch_episode_processing(
         self,
         episodes_data: List[Dict[str, Any]],

processors/tv_series_processor.py

@@ -192,10 +192,19 @@ class TVSeriesProcessor:
             aired, dateadded, source = self._get_episode_dates_from_sonarr(imdb_id, season_num, episode_num)
         
         # Step 4: Create/update NFO and database
-        if dateadded:
+        if dateadded or aired:
             # Get episode metadata for title/plot
             title, plot = self._get_episode_metadata_from_sonarr(imdb_id, season_num, episode_num)
             
+            # Use aired date as dateadded if no import date found (user requirement)
+            if not dateadded and aired:
+                dateadded = aired
+                if source == "no_data_found":
+                    source = "tmdb:air_date_fallback"
+                else:
+                    source = f"{source}_used_as_dateadded"
+                _log("INFO", f"Using aired date as dateadded for S{season_num:02d}E{episode_num:02d}: {dateadded}")
+            
             # Create/update NFO with video filename
             success = self.episode_nfo_manager.create_episode_nfo(
                 season_dir, season_num, episode_num, aired, dateadded, source, title, plot

requirements.txt

@@ -5,4 +5,6 @@ psycopg2-binary==2.9.7
 requests==2.31.0
 python-multipart==0.0.6
 aiofiles==23.2.1
+aiohttp==3.8.6
+psutil==5.9.6
 python-dotenv==1.0.0

static/css/styles.css

@@ -0,0 +1,805 @@
+/* NFOGuard Web Interface Styles */
+:root {
+    --primary-color: #007bff;
+    --secondary-color: #6c757d;
+    --success-color: #28a745;
+    --warning-color: #ffc107;
+    --danger-color: #dc3545;
+    --dark-color: #343a40;
+    --light-color: #f8f9fa;
+    --border-color: #dee2e6;
+    --text-muted: #6c757d;
+    --shadow: 0 0.125rem 0.25rem rgba(0, 0, 0, 0.075);
+    --shadow-lg: 0 0.5rem 1rem rgba(0, 0, 0, 0.15);
+}
+
+* {
+    margin: 0;
+    padding: 0;
+    box-sizing: border-box;
+}
+
+body {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+    font-size: 14px;
+    line-height: 1.5;
+    color: var(--dark-color);
+    background-color: #f5f5f5;
+}
+
+.app-container {
+    min-height: 100vh;
+    display: flex;
+    flex-direction: column;
+}
+
+/* Header */
+.app-header {
+    background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+    color: white;
+    padding: 1rem 0;
+    box-shadow: var(--shadow-lg);
+}
+
+.header-content {
+    max-width: 1200px;
+    margin: 0 auto;
+    padding: 0 1rem;
+    text-align: center;
+}
+
+.header-content h1 {
+    font-size: 2rem;
+    font-weight: 300;
+    margin-bottom: 0.5rem;
+}
+
+.header-content h1 i {
+    margin-right: 0.5rem;
+}
+
+.header-content p {
+    opacity: 0.9;
+    font-size: 1rem;
+}
+
+.nav-tabs {
+    max-width: 1200px;
+    margin: 1rem auto 0;
+    padding: 0 1rem;
+    display: flex;
+    gap: 0.5rem;
+    flex-wrap: wrap;
+    justify-content: center;
+}
+
+.nav-tab {
+    background: rgba(255, 255, 255, 0.1);
+    border: none;
+    color: white;
+    padding: 0.75rem 1.5rem;
+    border-radius: 0.5rem;
+    cursor: pointer;
+    transition: all 0.3s ease;
+    font-size: 0.9rem;
+    display: flex;
+    align-items: center;
+    gap: 0.5rem;
+}
+
+.nav-tab:hover {
+    background: rgba(255, 255, 255, 0.2);
+    transform: translateY(-1px);
+}
+
+.nav-tab.active {
+    background: rgba(255, 255, 255, 0.9);
+    color: var(--dark-color);
+}
+
+/* Main Content */
+.main-content {
+    flex: 1;
+    max-width: 1200px;
+    margin: 0 auto;
+    padding: 2rem 1rem;
+    width: 100%;
+}
+
+.tab-content {
+    display: none;
+}
+
+.tab-content.active {
+    display: block;
+}
+
+/* Dashboard */
+.dashboard-grid {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
+    gap: 1.5rem;
+    margin-bottom: 2rem;
+}
+
+.stat-card {
+    background: white;
+    border-radius: 0.5rem;
+    padding: 1.5rem;
+    box-shadow: var(--shadow);
+    display: flex;
+    align-items: center;
+    gap: 1rem;
+}
+
+.stat-icon {
+    width: 60px;
+    height: 60px;
+    border-radius: 50%;
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    font-size: 1.5rem;
+    color: white;
+}
+
+.stat-icon.movies { background: linear-gradient(135deg, #667eea, #764ba2); }
+.stat-icon.tv { background: linear-gradient(135deg, #f093fb, #f5576c); }
+.stat-icon.missing { background: linear-gradient(135deg, #ffecd2, #fcb69f); }
+.stat-icon.activity { background: linear-gradient(135deg, #a8edea, #fed6e3); }
+
+.stat-info h3 {
+    font-size: 2rem;
+    font-weight: 700;
+    margin-bottom: 0.25rem;
+}
+
+.stat-info p {
+    font-weight: 500;
+    margin-bottom: 0.25rem;
+}
+
+.stat-info small {
+    color: var(--text-muted);
+    font-size: 0.85rem;
+}
+
+.dashboard-charts {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
+    gap: 1.5rem;
+}
+
+.chart-card {
+    background: white;
+    border-radius: 0.5rem;
+    padding: 1.5rem;
+    box-shadow: var(--shadow);
+}
+
+.chart-card h3 {
+    margin-bottom: 1rem;
+    color: var(--dark-color);
+}
+
+.chart-container {
+    height: 200px;
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    background: var(--light-color);
+    border-radius: 0.25rem;
+    color: var(--text-muted);
+}
+
+/* Content Header */
+.content-header {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    margin-bottom: 1.5rem;
+    flex-wrap: wrap;
+    gap: 1rem;
+}
+
+.content-header h2 {
+    color: var(--dark-color);
+    font-weight: 600;
+}
+
+.content-controls {
+    display: flex;
+    gap: 1rem;
+    align-items: center;
+    flex-wrap: wrap;
+}
+
+.search-controls {
+    display: flex;
+    gap: 0.5rem;
+    flex-wrap: wrap;
+}
+
+.filter-controls {
+    display: flex;
+    gap: 0.5rem;
+    align-items: center;
+    flex-wrap: wrap;
+}
+
+.search-box {
+    position: relative;
+    display: flex;
+    align-items: center;
+}
+
+.search-box i {
+    position: absolute;
+    left: 0.75rem;
+    color: var(--text-muted);
+}
+
+.search-box input {
+    padding: 0.5rem 0.75rem 0.5rem 2.5rem;
+    border: 1px solid var(--border-color);
+    border-radius: 0.25rem;
+    font-size: 0.9rem;
+    width: 250px;
+}
+
+.search-box input:focus {
+    outline: none;
+    border-color: var(--primary-color);
+    box-shadow: 0 0 0 0.2rem rgba(0, 123, 255, 0.25);
+}
+
+/* Buttons */
+.btn {
+    padding: 0.5rem 1rem;
+    border: none;
+    border-radius: 0.25rem;
+    cursor: pointer;
+    font-size: 0.9rem;
+    transition: all 0.2s ease;
+    display: inline-flex;
+    align-items: center;
+    gap: 0.5rem;
+    text-decoration: none;
+}
+
+.btn-primary {
+    background-color: var(--primary-color);
+    color: white;
+}
+
+.btn-primary:hover {
+    background-color: #0056b3;
+    transform: translateY(-1px);
+}
+
+.btn-secondary {
+    background-color: var(--secondary-color);
+    color: white;
+}
+
+.btn-secondary:hover {
+    background-color: #545b62;
+}
+
+.btn-success {
+    background-color: var(--success-color);
+    color: white;
+}
+
+.btn-success:hover {
+    background-color: #1e7e34;
+}
+
+.btn-warning {
+    background-color: var(--warning-color);
+    color: var(--dark-color);
+}
+
+.btn-warning:hover {
+    background-color: #e0a800;
+}
+
+.btn-danger {
+    background-color: var(--danger-color);
+    color: white;
+}
+
+.btn-danger:hover {
+    background-color: #c82333;
+}
+
+.btn-sm {
+    padding: 0.25rem 0.5rem;
+    font-size: 0.8rem;
+}
+
+/* Tables */
+.table-container {
+    background: white;
+    border-radius: 0.5rem;
+    box-shadow: var(--shadow);
+    overflow: hidden;
+    margin-bottom: 1rem;
+}
+
+.data-table {
+    width: 100%;
+    border-collapse: collapse;
+}
+
+.data-table th,
+.data-table td {
+    padding: 0.75rem;
+    text-align: left;
+    border-bottom: 1px solid var(--border-color);
+}
+
+.data-table th {
+    background-color: var(--light-color);
+    font-weight: 600;
+    color: var(--dark-color);
+    position: sticky;
+    top: 0;
+}
+
+.data-table tr:hover {
+    background-color: rgba(0, 123, 255, 0.05);
+}
+
+.data-table .loading {
+    text-align: center;
+    color: var(--text-muted);
+    font-style: italic;
+    padding: 2rem;
+}
+
+/* Status badges */
+.badge {
+    padding: 0.25rem 0.5rem;
+    border-radius: 0.25rem;
+    font-size: 0.75rem;
+    font-weight: 600;
+    text-transform: uppercase;
+}
+
+.badge-success {
+    background-color: #d4edda;
+    color: #155724;
+}
+
+.badge-warning {
+    background-color: #fff3cd;
+    color: #856404;
+}
+
+.badge-danger {
+    background-color: #f8d7da;
+    color: #721c24;
+}
+
+.badge-secondary {
+    background-color: #e9ecef;
+    color: #495057;
+}
+
+/* Pagination */
+.pagination {
+    display: flex;
+    justify-content: center;
+    align-items: center;
+    gap: 0.5rem;
+    margin-top: 1rem;
+}
+
+.pagination .btn {
+    padding: 0.5rem 0.75rem;
+}
+
+.pagination .page-info {
+    margin: 0 1rem;
+    color: var(--text-muted);
+}
+
+/* Forms */
+.form-group {
+    margin-bottom: 1rem;
+}
+
+.form-group label {
+    display: block;
+    margin-bottom: 0.25rem;
+    font-weight: 500;
+}
+
+.form-group input,
+.form-group select,
+.form-group textarea {
+    width: 100%;
+    padding: 0.5rem;
+    border: 1px solid var(--border-color);
+    border-radius: 0.25rem;
+    font-size: 0.9rem;
+}
+
+.form-group input:focus,
+.form-group select:focus,
+.form-group textarea:focus {
+    outline: none;
+    border-color: var(--primary-color);
+    box-shadow: 0 0 0 0.2rem rgba(0, 123, 255, 0.25);
+}
+
+.form-group small {
+    display: block;
+    margin-top: 0.25rem;
+    color: var(--text-muted);
+    font-size: 0.8rem;
+}
+
+.form-actions {
+    display: flex;
+    gap: 0.5rem;
+    justify-content: flex-end;
+    margin-top: 1.5rem;
+}
+
+/* Modal */
+.modal {
+    display: none;
+    position: fixed;
+    z-index: 1000;
+    left: 0;
+    top: 0;
+    width: 100%;
+    height: 100%;
+    background-color: rgba(0, 0, 0, 0.5);
+}
+
+.modal.active {
+    display: flex;
+    align-items: center;
+    justify-content: center;
+}
+
+.modal-content {
+    background: white;
+    border-radius: 0.5rem;
+    max-width: 500px;
+    width: 90%;
+    max-height: 90vh;
+    overflow-y: auto;
+    box-shadow: var(--shadow-lg);
+}
+
+.modal-header {
+    padding: 1rem 1.5rem;
+    border-bottom: 1px solid var(--border-color);
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+}
+
+.modal-header h3 {
+    margin: 0;
+}
+
+.modal-close {
+    background: none;
+    border: none;
+    font-size: 1.5rem;
+    cursor: pointer;
+    color: var(--text-muted);
+}
+
+.modal-close:hover {
+    color: var(--dark-color);
+}
+
+.modal-body {
+    padding: 1.5rem;
+}
+
+/* Higher z-index for edit modals that appear on top of other modals */
+#edit-modal, #smart-fix-modal {
+    z-index: 1100 !important;
+}
+
+/* Reports */
+.report-summary {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
+    gap: 1rem;
+    margin-bottom: 2rem;
+}
+
+.summary-card {
+    background: white;
+    padding: 1.5rem;
+    border-radius: 0.5rem;
+    box-shadow: var(--shadow);
+    text-align: center;
+}
+
+.summary-card h3 {
+    margin-bottom: 1rem;
+    color: var(--dark-color);
+}
+
+.summary-card p {
+    margin-bottom: 0.5rem;
+    font-size: 1.1rem;
+}
+
+.summary-card span {
+    font-weight: 700;
+    color: var(--primary-color);
+}
+
+.report-section {
+    margin-bottom: 2rem;
+}
+
+.report-section h3 {
+    margin-bottom: 1rem;
+    color: var(--dark-color);
+}
+
+/* Tools */
+.tools-grid {
+    display: grid;
+    grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
+    gap: 1.5rem;
+}
+
+.tool-card {
+    background: white;
+    border-radius: 0.5rem;
+    padding: 1.5rem;
+    box-shadow: var(--shadow);
+}
+
+.tool-card h3 {
+    margin-bottom: 0.5rem;
+    color: var(--dark-color);
+}
+
+.tool-card p {
+    margin-bottom: 1.5rem;
+    color: var(--text-muted);
+}
+
+.stats-display {
+    background: var(--light-color);
+    padding: 1rem;
+    border-radius: 0.25rem;
+    margin-bottom: 1rem;
+    min-height: 100px;
+}
+
+/* Toast notifications */
+.toast-container {
+    position: fixed;
+    top: 20px;
+    right: 20px;
+    z-index: 1050;
+}
+
+.toast {
+    background: white;
+    border-radius: 0.25rem;
+    box-shadow: var(--shadow-lg);
+    margin-bottom: 0.5rem;
+    padding: 0.75rem 1rem;
+    min-width: 300px;
+    border-left: 4px solid var(--primary-color);
+    animation: slideIn 0.3s ease;
+}
+
+.toast.success {
+    border-left-color: var(--success-color);
+}
+
+.toast.warning {
+    border-left-color: var(--warning-color);
+}
+
+.toast.error {
+    border-left-color: var(--danger-color);
+}
+
+@keyframes slideIn {
+    from {
+        transform: translateX(100%);
+        opacity: 0;
+    }
+    to {
+        transform: translateX(0);
+        opacity: 1;
+    }
+}
+
+/* Smart Fix Modal */
+.smart-fix-options {
+    display: flex;
+    flex-direction: column;
+    gap: 1rem;
+    margin-bottom: 1rem;
+}
+
+.option-card {
+    border: 2px solid var(--border-color);
+    border-radius: 0.5rem;
+    transition: all 0.2s ease;
+}
+
+.option-card:hover {
+    border-color: var(--primary-color);
+    box-shadow: var(--shadow);
+}
+
+.option-label {
+    display: block;
+    padding: 1rem;
+    cursor: pointer;
+    margin: 0;
+}
+
+.option-label input[type="radio"] {
+    margin-right: 0.75rem;
+    margin-top: 0.1rem;
+    width: auto;
+}
+
+.option-content h4 {
+    margin: 0 0 0.5rem 0;
+    color: var(--dark-color);
+    font-size: 1rem;
+}
+
+.option-content p {
+    margin: 0 0 0.5rem 0;
+    color: var(--text-muted);
+    font-size: 0.9rem;
+}
+
+.option-content small {
+    color: var(--text-muted);
+    font-size: 0.8rem;
+}
+
+.manual-date-input {
+    width: 100% !important;
+    margin-top: 0.5rem !important;
+}
+
+.option-card input[type="radio"]:checked + .option-content {
+    color: var(--primary-color);
+}
+
+.option-card:has(input[type="radio"]:checked) {
+    border-color: var(--primary-color);
+    background-color: rgba(0, 123, 255, 0.05);
+}
+
+/* Additional badge styles */
+.badge-info {
+    background-color: #d1ecf1;
+    color: #0c5460;
+}
+
+/* Enhanced Edit Modal Date Options */
+.date-options {
+    display: flex;
+    flex-direction: column;
+    gap: 0.75rem;
+    margin-bottom: 1rem;
+}
+
+.date-option-card {
+    border: 1px solid var(--border-color);
+    border-radius: 0.375rem;
+    transition: all 0.2s ease;
+}
+
+.date-option-card:hover {
+    border-color: var(--primary-color);
+    box-shadow: 0 0 0 2px rgba(0, 123, 255, 0.1);
+}
+
+.date-option-label {
+    display: block;
+    padding: 0.75rem;
+    cursor: pointer;
+    margin: 0;
+}
+
+.date-option-label input[type="radio"] {
+    margin-right: 0.5rem;
+    margin-top: 0.1rem;
+    width: auto;
+}
+
+.date-option-content h4 {
+    margin: 0 0 0.25rem 0;
+    color: var(--dark-color);
+    font-size: 0.9rem;
+    font-weight: 600;
+}
+
+.date-option-content p {
+    margin: 0 0 0.25rem 0;
+    color: var(--text-muted);
+    font-size: 0.8rem;
+}
+
+.date-option-content small {
+    color: var(--primary-color);
+    font-size: 0.75rem;
+    font-weight: 500;
+}
+
+.date-option-card input[type="radio"]:checked + .date-option-content h4 {
+    color: var(--primary-color);
+}
+
+.date-option-card:has(input[type="radio"]:checked) {
+    border-color: var(--primary-color);
+    background-color: rgba(0, 123, 255, 0.03);
+}
+
+/* Responsive */
+@media (max-width: 768px) {
+    .content-header {
+        flex-direction: column;
+        align-items: stretch;
+    }
+    
+    .content-controls {
+        justify-content: center;
+    }
+    
+    .search-box input {
+        width: 200px;
+    }
+    
+    .nav-tabs {
+        flex-direction: column;
+        gap: 0.25rem;
+    }
+    
+    .data-table {
+        font-size: 0.8rem;
+    }
+    
+    .data-table th,
+    .data-table td {
+        padding: 0.5rem 0.25rem;
+    }
+    
+    .dashboard-grid {
+        grid-template-columns: 1fr;
+    }
+    
+    .tools-grid {
+        grid-template-columns: 1fr;
+    }
+}
+
+/* Utility classes */
+.text-center { text-align: center; }
+.text-muted { color: var(--text-muted); }
+.mb-0 { margin-bottom: 0; }
+.mb-1 { margin-bottom: 0.5rem; }
+.mb-2 { margin-bottom: 1rem; }
+.mt-1 { margin-top: 0.5rem; }
+.mt-2 { margin-top: 1rem; }
+.d-none { display: none; }
+.d-block { display: block; }
+.d-flex { display: flex; }
+.justify-content-between { justify-content: space-between; }
+.align-items-center { align-items: center; }

static/index.html

@@ -0,0 +1,401 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>NFOGuard - Database Management</title>
+    <link rel="stylesheet" href="/static/css/styles.css">
+    <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css" rel="stylesheet">
+</head>
+<body>
+    <div class="app-container">
+        <!-- Header -->
+        <header class="app-header">
+            <div class="header-content">
+                <h1><i class="fas fa-shield-alt"></i> NFOGuard</h1>
+                <p>Database Management & Reporting</p>
+            </div>
+            <nav class="nav-tabs">
+                <button class="nav-tab active" data-tab="dashboard">
+                    <i class="fas fa-tachometer-alt"></i> Dashboard
+                </button>
+                <button class="nav-tab" data-tab="movies">
+                    <i class="fas fa-film"></i> Movies
+                </button>
+                <button class="nav-tab" data-tab="tv">
+                    <i class="fas fa-tv"></i> TV Series
+                </button>
+                <button class="nav-tab" data-tab="reports">
+                    <i class="fas fa-chart-bar"></i> Reports
+                </button>
+                <button class="nav-tab" data-tab="tools">
+                    <i class="fas fa-tools"></i> Tools
+                </button>
+            </nav>
+        </header>
+
+        <!-- Main Content -->
+        <main class="main-content">
+            <!-- Dashboard Tab -->
+            <div class="tab-content active" id="dashboard">
+                <div class="dashboard-grid">
+                    <div class="stat-card">
+                        <div class="stat-icon movies">
+                            <i class="fas fa-film"></i>
+                        </div>
+                        <div class="stat-info">
+                            <h3 id="movies-total">-</h3>
+                            <p>Total Movies</p>
+                            <small id="movies-with-dates">- with dates</small>
+                        </div>
+                    </div>
+                    
+                    <div class="stat-card">
+                        <div class="stat-icon tv">
+                            <i class="fas fa-tv"></i>
+                        </div>
+                        <div class="stat-info">
+                            <h3 id="series-total">-</h3>
+                            <p>TV Series</p>
+                            <small id="episodes-total">- episodes</small>
+                        </div>
+                    </div>
+                    
+                    <div class="stat-card">
+                        <div class="stat-icon missing">
+                            <i class="fas fa-exclamation-triangle"></i>
+                        </div>
+                        <div class="stat-info">
+                            <h3 id="missing-dates-total">-</h3>
+                            <p>Missing Dates</p>
+                            <small id="no-valid-source-total">- no valid source</small>
+                        </div>
+                    </div>
+                    
+                    <div class="stat-card">
+                        <div class="stat-icon activity">
+                            <i class="fas fa-history"></i>
+                        </div>
+                        <div class="stat-info">
+                            <h3 id="recent-activity">-</h3>
+                            <p>Recent Activity</p>
+                            <small>Last 7 days</small>
+                        </div>
+                    </div>
+                </div>
+
+                <div class="dashboard-charts">
+                    <div class="chart-card">
+                        <h3><i class="fas fa-chart-pie"></i> Movie Sources</h3>
+                        <div id="movie-sources-chart" class="chart-container"></div>
+                    </div>
+                    <div class="chart-card">
+                        <h3><i class="fas fa-chart-pie"></i> Episode Sources</h3>
+                        <div id="episode-sources-chart" class="chart-container"></div>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Movies Tab -->
+            <div class="tab-content" id="movies">
+                <div class="content-header">
+                    <h2><i class="fas fa-film"></i> Movies Database</h2>
+                    <div class="content-controls">
+                        <div class="search-controls">
+                            <div class="search-box">
+                                <i class="fas fa-search"></i>
+                                <input type="text" id="movies-search" placeholder="Search title/path...">
+                            </div>
+                            <div class="search-box">
+                                <i class="fas fa-hashtag"></i>
+                                <input type="text" id="movies-imdb-search" placeholder="Search IMDb ID...">
+                            </div>
+                        </div>
+                        <div class="filter-controls">
+                            <select id="movies-filter-date">
+                                <option value="">All Movies</option>
+                                <option value="true">With Dates</option>
+                                <option value="false">Missing Dates</option>
+                            </select>
+                            <select id="movies-filter-source">
+                                <option value="">All Sources</option>
+                            </select>
+                            <button class="btn btn-primary" onclick="refreshMovies()">
+                                <i class="fas fa-sync"></i> Refresh
+                            </button>
+                        </div>
+                    </div>
+                </div>
+                
+                <div class="table-container">
+                    <table class="data-table" id="movies-table">
+                        <thead>
+                            <tr>
+                                <th>Title</th>
+                                <th>IMDb ID</th>
+                                <th>Movie Released</th>
+                                <th>Date Added to Library</th>
+                                <th>Source</th>
+                                <th>Date Type</th>
+                                <th>Video File</th>
+                                <th>Actions</th>
+                            </tr>
+                        </thead>
+                        <tbody id="movies-tbody">
+                            <tr>
+                                <td colspan="8" class="loading">Loading movies...</td>
+                            </tr>
+                        </tbody>
+                    </table>
+                </div>
+                
+                <div class="pagination" id="movies-pagination"></div>
+            </div>
+
+            <!-- TV Series Tab -->
+            <div class="tab-content" id="tv">
+                <div class="content-header">
+                    <h2><i class="fas fa-tv"></i> TV Series Database</h2>
+                    <div class="content-controls">
+                        <div class="search-controls">
+                            <div class="search-box">
+                                <i class="fas fa-search"></i>
+                                <input type="text" id="series-search" placeholder="Search title/path...">
+                            </div>
+                            <div class="search-box">
+                                <i class="fas fa-hashtag"></i>
+                                <input type="text" id="series-imdb-search" placeholder="Search IMDb ID...">
+                            </div>
+                        </div>
+                        <div class="filter-controls">
+                            <select id="series-filter-date">
+                                <option value="">All Series</option>
+                                <option value="complete">Fully Dated</option>
+                                <option value="incomplete">Missing Dates</option>
+                                <option value="none">No Dates</option>
+                            </select>
+                            <select id="series-filter-source">
+                                <option value="">All Sources</option>
+                            </select>
+                            <button class="btn btn-primary" onclick="refreshSeries()">
+                                <i class="fas fa-sync"></i> Refresh
+                            </button>
+                        </div>
+                    </div>
+                </div>
+                
+                <div class="table-container">
+                    <table class="data-table" id="series-table">
+                        <thead>
+                            <tr>
+                                <th>Series Title</th>
+                                <th>IMDb ID</th>
+                                <th>Episodes</th>
+                                <th>With Dates</th>
+                                <th>With Video</th>
+                                <th>Actions</th>
+                            </tr>
+                        </thead>
+                        <tbody id="series-tbody">
+                            <tr>
+                                <td colspan="6" class="loading">Loading series...</td>
+                            </tr>
+                        </tbody>
+                    </table>
+                </div>
+                
+                <div class="pagination" id="series-pagination"></div>
+            </div>
+
+            <!-- Reports Tab -->
+            <div class="tab-content" id="reports">
+                <div class="content-header">
+                    <h2><i class="fas fa-chart-bar"></i> Missing Dates Report</h2>
+                    <div class="content-controls">
+                        <button class="btn btn-primary" onclick="refreshReport()">
+                            <i class="fas fa-sync"></i> Refresh Report
+                        </button>
+                    </div>
+                </div>
+
+                <div class="report-summary" id="report-summary">
+                    <div class="summary-card">
+                        <h3>Movies</h3>
+                        <p><span id="report-movies-with">-</span> with dates</p>
+                        <p><span id="report-movies-missing">-</span> missing dates</p>
+                    </div>
+                    <div class="summary-card">
+                        <h3>Episodes</h3>
+                        <p><span id="report-episodes-with">-</span> with dates</p>
+                        <p><span id="report-episodes-missing">-</span> missing dates</p>
+                    </div>
+                </div>
+
+                <div class="report-content">
+                    <div class="report-section">
+                        <h3><i class="fas fa-film"></i> Movies Missing Dates</h3>
+                        <div class="table-container">
+                            <table class="data-table">
+                                <thead>
+                                    <tr>
+                                        <th>Title</th>
+                                        <th>IMDb ID</th>
+                                        <th>Released</th>
+                                        <th>Source</th>
+                                        <th>Smart Fix</th>
+                                    </tr>
+                                </thead>
+                                <tbody id="report-movies-tbody">
+                                    <tr>
+                                        <td colspan="5" class="loading">Loading report...</td>
+                                    </tr>
+                                </tbody>
+                            </table>
+                        </div>
+                    </div>
+
+                    <div class="report-section">
+                        <h3><i class="fas fa-tv"></i> Episodes Missing Dates</h3>
+                        <div class="table-container">
+                            <table class="data-table">
+                                <thead>
+                                    <tr>
+                                        <th>Series</th>
+                                        <th>Episode</th>
+                                        <th>IMDb ID</th>
+                                        <th>Aired</th>
+                                        <th>Source</th>
+                                        <th>Smart Fix</th>
+                                    </tr>
+                                </thead>
+                                <tbody id="report-episodes-tbody">
+                                    <tr>
+                                        <td colspan="6" class="loading">Loading report...</td>
+                                    </tr>
+                                </tbody>
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Tools Tab -->
+            <div class="tab-content" id="tools">
+                <div class="content-header">
+                    <h2><i class="fas fa-tools"></i> Database Tools</h2>
+                </div>
+
+                <div class="tools-grid">
+                    <div class="tool-card">
+                        <h3><i class="fas fa-exchange-alt"></i> Bulk Source Update</h3>
+                        <p>Change source for multiple items at once</p>
+                        <form id="bulk-update-form">
+                            <div class="form-group">
+                                <label>Media Type:</label>
+                                <select id="bulk-media-type" required>
+                                    <option value="">Select type...</option>
+                                    <option value="movies">Movies</option>
+                                    <option value="episodes">Episodes</option>
+                                </select>
+                            </div>
+                            <div class="form-group">
+                                <label>From Source:</label>
+                                <input type="text" id="bulk-old-source" placeholder="e.g., no_valid_date_source" required>
+                            </div>
+                            <div class="form-group">
+                                <label>To Source:</label>
+                                <select id="bulk-new-source" required>
+                                    <option value="">Select new source...</option>
+                                    <option value="airdate">Air Date</option>
+                                    <option value="digital_release">Digital Release</option>
+                                    <option value="manual">Manual</option>
+                                    <option value="radarr:db.history.import">Radarr Import</option>
+                                    <option value="sonarr:history.import">Sonarr Import</option>
+                                </select>
+                            </div>
+                            <button type="submit" class="btn btn-warning">
+                                <i class="fas fa-exchange-alt"></i> Update Sources
+                            </button>
+                        </form>
+                    </div>
+
+                    <div class="tool-card">
+                        <h3><i class="fas fa-database"></i> Database Statistics</h3>
+                        <p>View detailed database information</p>
+                        <div class="stats-display" id="detailed-stats">
+                            <p>Click refresh to load detailed statistics</p>
+                        </div>
+                        <button class="btn btn-secondary" onclick="loadDetailedStats()">
+                            <i class="fas fa-sync"></i> Refresh Stats
+                        </button>
+                    </div>
+                </div>
+            </div>
+        </main>
+    </div>
+
+    <!-- Smart Fix Modal -->
+    <div class="modal" id="smart-fix-modal">
+        <div class="modal-content">
+            <div class="modal-header">
+                <h3 id="smart-fix-title">Choose Date Source</h3>
+                <button class="modal-close" onclick="closeSmartFixModal()">&times;</button>
+            </div>
+            <div class="modal-body">
+                <div id="smart-fix-content">
+                    <p>Loading available options...</p>
+                </div>
+                <div class="form-actions">
+                    <button type="button" class="btn btn-secondary" onclick="closeSmartFixModal()">Cancel</button>
+                </div>
+            </div>
+        </div>
+    </div>
+
+    <!-- Edit Modal -->
+    <div class="modal" id="edit-modal">
+        <div class="modal-content">
+            <div class="modal-header">
+                <h3 id="modal-title">Edit Entry</h3>
+                <button class="modal-close" onclick="closeModal()">&times;</button>
+            </div>
+            <div class="modal-body">
+                <form id="edit-form">
+                    <input type="hidden" id="edit-imdb-id">
+                    <input type="hidden" id="edit-season">
+                    <input type="hidden" id="edit-episode">
+                    <input type="hidden" id="edit-media-type">
+                    
+                    <div class="form-group">
+                        <label for="edit-dateadded">Date Added:</label>
+                        <input type="datetime-local" id="edit-dateadded">
+                        <small>Leave empty to clear date</small>
+                    </div>
+                    
+                    <div class="form-group">
+                        <label for="edit-source">Source:</label>
+                        <select id="edit-source" required>
+                            <option value="manual">Manual</option>
+                            <option value="airdate">Air Date</option>
+                            <option value="digital_release">Digital Release</option>
+                            <option value="radarr:db.history.import">Radarr Import</option>
+                            <option value="sonarr:history.import">Sonarr Import</option>
+                            <option value="no_valid_date_source">No Valid Source</option>
+                        </select>
+                    </div>
+                    
+                    <div class="form-actions">
+                        <button type="button" class="btn btn-secondary" onclick="closeModal()">Cancel</button>
+                        <button type="submit" class="btn btn-primary">Save Changes</button>
+                    </div>
+                </form>
+            </div>
+        </div>
+    </div>
+
+    <!-- Toast Notifications -->
+    <div class="toast-container" id="toast-container"></div>
+
+    <script src="/static/js/app.js"></script>
+</body>
+</html>

utils/file_utils.py

@@ -62,7 +62,8 @@ def find_media_path_by_imdb_and_title(
         
         # Search by IMDb ID first (more reliable)
         if imdb_id:
-            pattern = str(media_path / f"*[imdb-{imdb_id}]*")
+            # Use proper glob pattern - escape brackets to match literal [imdb-ID]
+            pattern = str(media_path / f"*\\[imdb-{imdb_id}\\]*")
             matches = glob.glob(pattern)
             if matches:
                 return Path(matches[0])

webhooks/webhook_batcher.py

@@ -14,7 +14,7 @@ from utils.logging import _log
 class WebhookBatcher:
     """Batches webhook events to avoid processing storms"""
     
-    def __init__(self):
+    def __init__(self, nfo_manager=None):
         self.pending: Dict[str, Dict] = {}
         self.timers: Dict[str, threading.Timer] = {}
         self.processing: Set[str] = set()
@@ -23,6 +23,8 @@ class WebhookBatcher:
         # Will be set by the application when processors are available
         self.tv_processor = None
         self.movie_processor = None
+        # NFO manager for comprehensive IMDb detection
+        self.nfo_manager = nfo_manager
     
     def set_processors(self, tv_processor, movie_processor):
         """Set the processor instances"""
@@ -81,14 +83,57 @@ class WebhookBatcher:
             # CRITICAL: Validate that the path contains the expected IMDb ID for movies
             if media_type == 'movie':
                 expected_imdb = key.replace('movie:', '') if key.startswith('movie:') else key
-                if expected_imdb not in path_str.lower():
-                    _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found in path {path_str}")
-                    _log("ERROR", f"This prevents processing wrong movies due to batch corruption")
-                    return
-                _log("DEBUG", f"Batch validation passed: IMDb {expected_imdb} found in path")
                 
-            # Process based on media type
+                # Use comprehensive IMDb detection (directory, filenames, NFO files)
+                if self.nfo_manager:
+                    detected_imdb = self.nfo_manager.find_movie_imdb_id(path_obj)
+                    imdb_match = False
+                    if detected_imdb:
+                        # Compare with and without 'tt' prefix for flexibility
+                        if detected_imdb == expected_imdb or detected_imdb.replace('tt', '') == expected_imdb.replace('tt', ''):
+                            imdb_match = True
+                    
+                    if not imdb_match:
+                        _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found via comprehensive detection in path {path_str}")
+                        _log("ERROR", f"Detected IMDb: {detected_imdb}, Expected: {expected_imdb}")
+                        _log("ERROR", f"This prevents processing wrong movies due to batch corruption")
+                        return
+                    _log("DEBUG", f"Batch validation passed: IMDb {expected_imdb} matches detected {detected_imdb}")
+                else:
+                    # Fallback to simple string search if nfo_manager not available
+                    if expected_imdb not in path_str.lower():
+                        _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found in path {path_str} (fallback mode)")
+                        _log("ERROR", f"This prevents processing wrong movies due to batch corruption")
+                        return
+                    _log("DEBUG", f"Batch validation passed: IMDb {expected_imdb} found in path (fallback mode)")
+            
+            # CRITICAL: Validate that the path contains the expected IMDb ID for TV shows
             if media_type == 'tv':
+                expected_imdb = key.replace('tv:', '') if key.startswith('tv:') else key
+                
+                # Use comprehensive IMDb detection (directory, filenames, tvshow.nfo files)
+                if self.nfo_manager:
+                    detected_imdb = self.nfo_manager.find_series_imdb_id(path_obj)
+                    imdb_match = False
+                    if detected_imdb:
+                        # Compare with and without 'tt' prefix for flexibility
+                        if detected_imdb == expected_imdb or detected_imdb.replace('tt', '') == expected_imdb.replace('tt', ''):
+                            imdb_match = True
+                    
+                    if not imdb_match:
+                        _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found via comprehensive detection in TV path {path_str}")
+                        _log("ERROR", f"Detected TV IMDb: {detected_imdb}, Expected: {expected_imdb}")
+                        _log("ERROR", f"This prevents processing wrong TV series due to batch corruption")
+                        return
+                    _log("DEBUG", f"TV batch validation passed: IMDb {expected_imdb} matches detected {detected_imdb}")
+                else:
+                    # Fallback to simple string search if nfo_manager not available
+                    if expected_imdb not in path_str.lower():
+                        _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found in TV path {path_str} (fallback mode)")
+                        _log("ERROR", f"This prevents processing wrong TV series due to batch corruption")
+                        return
+                    _log("DEBUG", f"TV batch validation passed: IMDb {expected_imdb} found in path (fallback mode)")
+                
                 if not self.tv_processor:
                     _log("ERROR", "TV processor not available")
                     return
@@ -128,3 +173,32 @@ class WebhookBatcher:
                 "pending_count": len(self.pending),
                 "processing_count": len(self.processing)
             }
+    
+    def shutdown(self):
+        """Shutdown the webhook batcher gracefully"""
+        _log("INFO", "Shutting down webhook batcher...")
+        
+        with self.lock:
+            # Cancel all pending timers
+            for timer in self.timers.values():
+                try:
+                    timer.cancel()
+                except Exception as e:
+                    _log("WARNING", f"Error canceling timer: {e}")
+            
+            self.timers.clear()
+            
+            # Log any remaining items
+            if self.pending:
+                _log("WARNING", f"Shutting down with {len(self.pending)} pending items")
+            if self.processing:
+                _log("INFO", f"Waiting for {len(self.processing)} items to finish processing...")
+        
+        # Shutdown the thread pool executor
+        try:
+            self.executor.shutdown(wait=True, timeout=10)  # Wait up to 10 seconds
+            _log("INFO", "Thread pool executor shut down successfully")
+        except Exception as e:
+            _log("WARNING", f"Error shutting down thread pool: {e}")
+        
+        _log("INFO", "Webhook batcher shutdown complete")