Merge pull request 'fix: fixing webhook processing' (#22) from webhook-fix into dev

Reviewed-on: #22

README.md

@@ -26,14 +26,37 @@ NFOGuard automatically updates movie and TV show NFO files with proper release d
 
 ## ✨ Features
 
-- 🎬 **Movie & TV Support** - Works with both Radarr and Sonarr
+### **🎬 Core Media Management**
-- 📅 **Smart Date Handling** - Prioritizes digital, physical, and theatrical release dates
+- **Movie & TV Support** - Works with both Radarr and Sonarr
-- 🔄 **Webhook Integration** - Triggers automatically on import, upgrade, and rename
+- **Smart Date Handling** - Prioritizes digital, physical, and theatrical release dates
-- 🗄️ **Database Integration** - Direct PostgreSQL access for better performance
+- **Webhook Integration** - Triggers automatically on import, upgrade, and rename
-- 📝 **NFO Preservation** - Maintains existing metadata, adds fields cleanly at bottom
+- **NFO Preservation** - Maintains existing metadata, adds fields cleanly at bottom
-- 🔒 **Metadata Locking** - Prevents overwrites with lockdata tags
+- **Metadata Locking** - Prevents overwrites with lockdata tags
-- ⚡ **Batch Processing** - Efficient handling of multiple files
+
-- 🐳 **Docker Ready** - Easy deployment with Docker Compose
+### **⚡ Performance & Scalability**
+- **Async I/O Operations** - High-performance concurrent file processing
+- **Batch Processing** - Efficient handling of multiple files simultaneously
+- **Database Integration** - Direct PostgreSQL/SQLite access for better performance
+- **Smart Concurrency** - Controlled parallel processing with rate limiting
+
+### **🔧 Configuration & Validation**
+- **Comprehensive Config Validation** - Validates all settings before startup
+- **Runtime Health Checks** - Monitors system health and dependencies
+- **Path Validation** - Ensures media directories are accessible
+- **Database Connectivity Tests** - Validates database connections
+
+### **📊 Monitoring & Observability**
+- **Health Check APIs** - `/api/v1/health`, `/api/v1/health/ready`, `/api/v1/health/live`
+- **Prometheus Metrics** - `/api/v1/metrics` endpoint for monitoring integration
+- **Performance Monitoring** - Operation timing, error rates, and bottleneck detection
+- **Structured Logging** - JSON logs with correlation IDs for request tracing
+- **System Status APIs** - Real-time performance and error metrics
+
+### **🐳 Production Ready**
+- **Docker & Kubernetes** - Health checks for orchestration platforms
+- **Grafana Compatible** - Metrics endpoints for dashboard integration
+- **Configuration CLI** - Validation tools for troubleshooting
+- **Modular Architecture** - Clean separation of concerns for maintainability
 
 ## 🚀 Quick Start
 
@@ -125,9 +148,11 @@ image: sbcrumb/nfoguard:dev
 
 ### Specific Version  
 ```yaml
-image: sbcrumb/nfoguard:v1.5.5
+image: sbcrumb/nfoguard:v2.0.0  # Latest with monitoring & validation
 ```
 
+> **🚀 Version 2.0.0** includes major architecture improvements, async I/O performance enhancements, comprehensive monitoring, and configuration validation.
+
 ## 🔗 Webhook Setup
 
 Configure these webhook URLs in your applications:
@@ -162,13 +187,38 @@ curl -X POST "http://localhost:8080/bulk/update"
 
 ### API Endpoints
 
+#### **Core Operations**
 | Endpoint | Method | Purpose |
 |----------|--------|---------|
-| `/health` | GET | Health check |
 | `/webhook/radarr` | POST | Radarr webhook handler |
 | `/webhook/sonarr` | POST | Sonarr webhook handler |
 | `/manual/scan` | POST | Manual media scanning |
-| `/bulk/update` | POST | Bulk movie updates from Radarr DB |
+| `/bulk/update` | POST | Bulk update from Radarr database |
+
+#### **Health & Monitoring** 
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/health` | GET | Comprehensive health status |
+| `/api/v1/health/ready` | GET | Kubernetes readiness probe |
+| `/api/v1/health/live` | GET | Kubernetes liveness probe |
+| `/api/v1/status` | GET | Complete system status |
+| `/api/v1/status/brief` | GET | Quick system overview |
+
+#### **Metrics & Performance**
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/metrics` | GET | Prometheus metrics (text format) |
+| `/api/v1/metrics/json` | GET | Metrics in JSON format |
+| `/api/v1/metrics/processing` | GET | Processing-specific metrics |
+| `/api/v1/metrics/errors` | GET | Error metrics and recent failures |
+| `/api/v1/metrics/system` | GET | System resource metrics |
+
+#### **Configuration & Debugging**
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/debug/movie/{imdb_id}/priority` | GET | Debug movie priority logic |
+| `/debug/tmdb/{imdb_id}` | GET | Debug TMDB lookup |
+| `/ping` | GET | Simple connectivity test |
 
 ### Manual Scan Parameters
 
@@ -226,6 +276,79 @@ PATH_DEBUG=true
 curl http://localhost:8080/health
 ```
 
+## 📊 Monitoring & Observability
+
+NFOGuard provides comprehensive monitoring capabilities for production deployments:
+
+### **Health Checks**
+```bash
+# Basic health status
+curl http://localhost:8080/api/v1/health
+
+# Kubernetes readiness probe  
+curl http://localhost:8080/api/v1/health/ready
+
+# Kubernetes liveness probe
+curl http://localhost:8080/api/v1/health/live
+
+# Quick system overview
+curl http://localhost:8080/api/v1/status/brief
+```
+
+### **Metrics for Grafana/Prometheus**
+```bash
+# Prometheus metrics format
+curl http://localhost:8080/api/v1/metrics
+
+# JSON metrics for dashboards
+curl http://localhost:8080/api/v1/metrics/json
+
+# Processing performance metrics
+curl http://localhost:8080/api/v1/metrics/processing
+
+# Error rates and recent failures
+curl http://localhost:8080/api/v1/metrics/errors
+
+# System resource usage
+curl http://localhost:8080/api/v1/metrics/system
+```
+
+### **Configuration Validation**
+```bash
+# Validate configuration before deployment
+python config/validation_cli.py
+
+# Include runtime tests (database, APIs, file access)
+python config/validation_cli.py --runtime
+
+# JSON output for automation
+python config/validation_cli.py --runtime --json
+```
+
+### **Docker Health Checks**
+Add to your `docker-compose.yml`:
+```yaml
+healthcheck:
+  test: ["CMD", "curl", "-f", "http://localhost:8080/api/v1/health/live"]
+  interval: 30s
+  timeout: 10s
+  retries: 3
+  start_period: 40s
+```
+
+### **Structured Logging**
+NFOGuard uses structured JSON logging with correlation IDs for request tracing:
+```bash
+# Enable structured logging
+STRUCTURED_LOGGING=true
+
+# Log level configuration  
+LOG_LEVEL=INFO
+
+# Example log output with correlation ID
+{"timestamp": "2024-01-01T12:00:00Z", "level": "INFO", "correlation_id": "req_123", "message": "Webhook received", "context": {"webhook_type": "radarr", "media_type": "movie"}}
+```
+
 ### Common Issues
 
 1. **Permission Errors**: Ensure NFOGuard can write to mounted directories

VERSION

@@ -1 +1 @@
-2.0.0
+2.0.1

main.py

@@ -94,8 +94,8 @@ def initialize_components():
     tv_processor = TVProcessor(db, nfo_manager, path_mapper)
     movie_processor = MovieProcessor(db, nfo_manager, path_mapper)
     
-    # Initialize webhook batcher
+    # Initialize webhook batcher with nfo_manager for comprehensive IMDb detection
-    batcher = WebhookBatcher()
+    batcher = WebhookBatcher(nfo_manager)
     batcher.set_processors(tv_processor, movie_processor)
     
     return {

nfoguard.py

@@ -1614,7 +1614,7 @@ nfo_manager = NFOManager(config.manager_brand, config.debug)
 path_mapper = PathMapper(config)  # FIXED: Pass config to PathMapper
 tv_processor = TVProcessor(db, nfo_manager, path_mapper)
 movie_processor = MovieProcessor(db, nfo_manager, path_mapper)
-batcher = WebhookBatcher()
+batcher = WebhookBatcher(nfo_manager)
 
 # ---------------------------
 # Webhook Handlers

webhooks/webhook_batcher.py

@@ -14,7 +14,7 @@ from utils.logging import _log
 class WebhookBatcher:
     """Batches webhook events to avoid processing storms"""
     
-    def __init__(self):
+    def __init__(self, nfo_manager=None):
         self.pending: Dict[str, Dict] = {}
         self.timers: Dict[str, threading.Timer] = {}
         self.processing: Set[str] = set()
@@ -23,6 +23,8 @@ class WebhookBatcher:
         # Will be set by the application when processors are available
         self.tv_processor = None
         self.movie_processor = None
+        # NFO manager for comprehensive IMDb detection
+        self.nfo_manager = nfo_manager
     
     def set_processors(self, tv_processor, movie_processor):
         """Set the processor instances"""
@@ -81,11 +83,29 @@ class WebhookBatcher:
             # CRITICAL: Validate that the path contains the expected IMDb ID for movies
             if media_type == 'movie':
                 expected_imdb = key.replace('movie:', '') if key.startswith('movie:') else key
-                if expected_imdb not in path_str.lower():
+                
-                    _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found in path {path_str}")
+                # Use comprehensive IMDb detection (directory, filenames, NFO files)
+                if self.nfo_manager:
+                    detected_imdb = self.nfo_manager.find_movie_imdb_id(path_obj)
+                    imdb_match = False
+                    if detected_imdb:
+                        # Compare with and without 'tt' prefix for flexibility
+                        if detected_imdb == expected_imdb or detected_imdb.replace('tt', '') == expected_imdb.replace('tt', ''):
+                            imdb_match = True
+                    
+                    if not imdb_match:
+                        _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found via comprehensive detection in path {path_str}")
+                        _log("ERROR", f"Detected IMDb: {detected_imdb}, Expected: {expected_imdb}")
                         _log("ERROR", f"This prevents processing wrong movies due to batch corruption")
                         return
-                _log("DEBUG", f"Batch validation passed: IMDb {expected_imdb} found in path")
+                    _log("DEBUG", f"Batch validation passed: IMDb {expected_imdb} matches detected {detected_imdb}")
+                else:
+                    # Fallback to simple string search if nfo_manager not available
+                    if expected_imdb not in path_str.lower():
+                        _log("ERROR", f"BATCH VALIDATION FAILED: Expected IMDb {expected_imdb} not found in path {path_str} (fallback mode)")
+                        _log("ERROR", f"This prevents processing wrong movies due to batch corruption")
+                        return
+                    _log("DEBUG", f"Batch validation passed: IMDb {expected_imdb} found in path (fallback mode)")
             
             # Process based on media type
             if media_type == 'tv':