Fix Neon Vortex button input and add Jellyfin WAL mode

- Comment out CORS headers in nginx.conf to fix button input blocking - Add WAL enable job for Jellyfin to resolve database locking issues - Include comprehensive troubleshooting documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-22 17:51:34 -05:00
parent 5b52eb9999
commit ce8d9029c1
4 changed files with 737 additions and 2 deletions
--- a/JELLYFIN_ERROR_ANALYSIS.md
+++ b/JELLYFIN_ERROR_ANALYSIS.md
@@ -0,0 +1,413 @@
+# Jellyfin Error Analysis and Recommendations
+
+## Overview
+Analysis of Jellyfin errors found in the media namespace logs, with recommendations for resolution.
+
+---
+
+## Issues Identified
+
+### 1. **SQLite Database Locking Errors** (CRITICAL)
+
+**Error Messages:**
+```
+[04:28:37] [ERR] Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
+[09:00:00] [ERR] SQLite Error 5: 'database is locked For more information on this error code see https://www.sqlite.org/rescode.html'.
+```
+
+**Impact:**
+- Prevents metadata updates during library scans
+- Causes API request failures (GET /Items)
+- Occurs when database optimization runs concurrently with library operations
+
+**Root Cause:**
+SQLite's default locking mechanism allows only one writer at a time. The errors occur when:
+1. Database optimization task runs (VACUUM operation)
+2. Library scan attempts to update metadata simultaneously
+3. Both operations try to write to the database
+
+**Frequency:**
+- Database optimization runs approximately every 6 hours
+- Library scans run every 12 hours
+- Conflicts occur when these overlap
+
+---
+
+### 2. **Audio Normalization Failures** (MODERATE)
+
+**Error Message:**
+```
+[04:24:54] [ERR] Emby.Server.Implementations.ScheduledTasks.Tasks.AudioNormalizationTask: Failed to find LUFS value in output
+```
+
+**Impact:**
+- Audio normalization task cannot measure loudness
+- Prevents automatic volume leveling across music library
+- Task completes without performing any normalization
+
+**Root Cause:**
+The LUFS (Loudness Units relative to Full Scale) detection failure suggests:
+1. FFmpeg may be missing required audio analysis filters
+2. Audio files may have incompatible formats
+3. FFmpeg output parsing may be failing
+
+**Frequency:**
+- Runs daily at 04:24 (during scheduled tasks)
+- Consistently fails to find LUFS values
+
+---
+
+### 3. **Library Folder Warnings** (LOW PRIORITY)
+
+**Warning Messages:**
+```
+[04:25:47] [WRN] Library folder /ebooks is inaccessible or empty, skipping
+[04:25:47] [WRN] Library folder /config/data/playlists is inaccessible or empty, skipping
+```
+
+**Impact:**
+- Minimal - warnings only
+- No functional issues if these folders are intentionally empty
+- Causes log noise
+
+**Root Cause:**
+- Library folders are configured but:
+  - `/ebooks` folder doesn't exist or is empty
+  - `/config/data/playlists` folder is empty
+- Jellyfin skips these during library scans
+
+---
+
+## Recommendations
+
+### Priority 1: Fix Database Locking Issues
+
+#### Option A: Enable WAL Mode (RECOMMENDED)
+
+WAL (Write-Ahead Logging) mode allows concurrent readers during writes, reducing lock contention.
+
+**Implementation:**
+
+1. **Create a job to enable WAL mode:**
+
+Create file: `/storage/emulated/0/Download/nv/jellyfin-wal-enable.yaml`
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: jellyfin-enable-wal
+  namespace: media
+spec:
+  ttlSecondsAfterFinished: 600
+  template:
+    spec:
+      restartPolicy: Never
+      containers:
+      - name: enable-wal
+        image: alpine:latest
+        command:
+        - sh
+        - -c
+        - |
+          apk add --no-cache sqlite
+          echo "Enabling WAL mode on jellyfin.db..."
+          sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode=WAL;"
+          echo "Verifying WAL mode..."
+          sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode;"
+          echo "Done!"
+        volumeMounts:
+        - name: jellyfin-data
+          mountPath: /config
+      volumes:
+      - name: jellyfin-data
+        persistentVolumeClaim:
+          claimName: jellyfin-config
+```
+
+**Benefits:**
+- Multiple readers can access database during writes
+- Reduces lock contention significantly
+- Better performance under concurrent load
+- Industry standard for high-concurrency SQLite
+
+**Caveats:**
+- Creates additional files (`jellyfin.db-wal`, `jellyfin.db-shm`)
+- Requires slightly more disk space
+- NFS must support file locking (appears to work based on current setup)
+
+---
+
+#### Option B: Adjust Task Scheduling
+
+Prevent database optimization and library scans from running simultaneously.
+
+**Current Schedule:**
+- Library scan: Every 12 hours (04:26, 16:26)
+- Database optimization: Every 6 hours (04:27, 10:28, 16:29)
+
+**Recommended Change:**
+Modify Jellyfin's scheduled tasks via the web UI:
+1. Go to Dashboard → Scheduled Tasks
+2. Change "Optimize database" to run at different times:
+   - Change from every 6 hours to daily at 03:00 (before library scan)
+3. This ensures optimization completes before library scan starts
+
+**Benefits:**
+- Simple configuration change
+- No code modifications needed
+- Reduces overlap probability
+
+**Caveats:**
+- Doesn't eliminate all race conditions
+- API requests during optimization can still fail
+- Less robust than WAL mode
+
+---
+
+#### Option C: Increase SQLite Timeout
+
+Configure Jellyfin to wait longer when database is locked.
+
+**Implementation:**
+Requires adding environment variable to Jellyfin deployment:
+
+```yaml
+env:
+- name: SQLITE_BUSY_TIMEOUT
+  value: "30000"  # 30 seconds
+```
+
+**Note:** This may not be supported by Jellyfin directly. WAL mode is preferred.
+
+---
+
+### Priority 2: Fix Audio Normalization
+
+#### Diagnosis Steps:
+
+1. **Check FFmpeg capabilities:**
+
+```bash
+kubectl exec -n media deployment/jellyfin -- ffmpeg -filters 2>&1 | grep -E "(loudnorm|ebur128)"
+```
+
+2. **Test LUFS detection manually:**
+
+```bash
+kubectl exec -n media deployment/jellyfin -- ffmpeg -i /music/sample.mp3 -af loudnorm=print_format=json -f null - 2>&1
+```
+
+Expected output should include:
+```json
+{
+  "input_i": "-23.0",
+  "input_tp": "-5.0",
+  "input_lra": "7.0",
+  "input_thresh": "-33.5",
+  "target_offset": "0.0"
+}
+```
+
+#### Solution Options:
+
+**If filters are missing:**
+- Jellyfin's FFmpeg build may lack audio normalization filters
+- Solution: Disable audio normalization task in Jellyfin (Dashboard → Scheduled Tasks)
+
+**If filters exist but fail:**
+- Some audio files may have incompatible formats
+- Solution: Check task logs for specific files causing issues
+
+**To disable audio normalization:**
+1. Go to Dashboard → Scheduled Tasks
+2. Find "Audio Normalization"
+3. Click on it and disable the task
+4. This is acceptable - normalization is optional for most users
+
+---
+
+### Priority 3: Clean Up Library Warnings
+
+#### Option A: Remove Empty Library Folders
+
+If you don't use ebooks or playlists:
+
+1. Go to Dashboard → Libraries
+2. Remove the "ebooks" library if it exists
+3. The playlists folder is auto-created and can be ignored
+
+#### Option B: Create the Folders
+
+If you plan to use these features:
+
+```bash
+kubectl exec -n media deployment/jellyfin -- mkdir -p /ebooks
+kubectl exec -n media deployment/jellyfin -- mkdir -p /config/data/playlists
+```
+
+---
+
+## Implementation Plan
+
+### Step 1: Enable WAL Mode (RECOMMENDED)
+
+This will resolve the critical database locking issues.
+
+```bash
+# Apply the WAL enable job
+kubectl apply -f jellyfin-wal-enable.yaml
+
+# Wait for completion
+kubectl wait --for=condition=complete --timeout=300s job/jellyfin-enable-wal -n media
+
+# Check logs
+kubectl logs -n media job/jellyfin-enable-wal
+
+# Restart Jellyfin to ensure WAL mode is active
+kubectl rollout restart deployment/jellyfin -n media
+```
+
+### Step 2: Disable Audio Normalization (OPTIONAL)
+
+Via Jellyfin Web UI:
+1. Navigate to https://jellyfin.caffeinetux.com/web/index.html#!/scheduledtasks.html
+2. Find "Audio Normalization"
+3. Click "Disable"
+
+### Step 3: Monitor Logs
+
+After implementing WAL mode:
+
+```bash
+# Watch for database lock errors (should disappear)
+kubectl logs -n media deployment/jellyfin -f | grep "database is locked"
+
+# Check if WAL mode is active
+kubectl exec -n media deployment/jellyfin -- sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode;"
+# Should output: wal
+```
+
+---
+
+## Performance Impact Analysis
+
+### Current Issues:
+- Database locks during library scans (2+ minutes)
+- Failed API requests during optimization
+- Metadata updates delayed or skipped
+
+### After WAL Mode:
+- ✅ Concurrent reads during writes
+- ✅ API requests succeed during optimization
+- ✅ Faster metadata updates
+- ✅ Better overall responsiveness
+
+### Trade-offs:
+- Slightly more disk space (typically <10% of DB size)
+- Additional files in config directory
+- Minimal performance overhead (usually net positive)
+
+---
+
+## Additional Recommendations
+
+### 1. Database Optimization Frequency
+
+Current: Every 6 hours may be excessive
+
+**Recommendation:** Change to once daily at 03:00
+
+Benefits:
+- Reduces I/O load on NFS
+- Less frequent database locks
+- Sufficient for most libraries
+
+### 2. Library Scan Optimization
+
+**Current behavior:**
+- Full library scan every 12 hours
+- Stops directory watching during scan
+
+**Recommendation:**
+- Keep 12-hour schedule for now
+- Monitor scan duration (currently 2-5 minutes)
+- If scans take too long, consider:
+  - Increasing interval to 24 hours
+  - Using real-time monitoring only
+
+### 3. NFS Considerations
+
+Your Jellyfin database is on NFS. WAL mode works with NFS if:
+- ✅ File locking is supported (appears to be working)
+- ✅ NFS client supports mmap operations
+- ⚠️ Network latency is low (check if you experience slowness)
+
+**If WAL mode causes issues on NFS:**
+- Switch back to DELETE mode: `PRAGMA journal_mode=DELETE;`
+- Consider moving database to local storage (emptyDir volume)
+- Keep media on NFS, database local
+
+---
+
+## Monitoring and Validation
+
+### Success Criteria:
+
+After implementing WAL mode, verify:
+
+1. **No more database lock errors:**
+```bash
+kubectl logs -n media deployment/jellyfin --since=24h | grep "database is locked"
+# Should return no results
+```
+
+2. **WAL mode is active:**
+```bash
+kubectl exec -n media deployment/jellyfin -- sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode;"
+# Should output: wal
+```
+
+3. **Library scans complete successfully:**
+```bash
+kubectl logs -n media deployment/jellyfin | grep "Scan Media Library Completed"
+# Should show successful completions
+```
+
+4. **Database optimization runs without conflicts:**
+```bash
+kubectl logs -n media deployment/jellyfin | grep "jellyfin.db optimized successfully"
+# Should show successful optimizations
+```
+
+---
+
+## Summary
+
+### Critical Issues (Fix Now):
+1. **SQLite Database Locking** → Enable WAL mode
+
+### Optional Improvements:
+2. **Audio Normalization** → Disable if not needed
+3. **Library Warnings** → Remove unused libraries or create folders
+4. **Task Scheduling** → Adjust optimization frequency to daily
+
+### Expected Outcome:
+- Elimination of database lock errors
+- Smoother library scans
+- Better API responsiveness
+- Cleaner logs
+
+---
+
+## Resources
+
+- [SQLite WAL Mode Documentation](https://www.sqlite.org/wal.html)
+- [Jellyfin Database Documentation](https://jellyfin.org/docs/general/administration/configuration.html)
+- [SQLite Error Codes](https://www.sqlite.org/rescode.html)
+
+---
+
+**Analysis Date:** 2025-11-22
+**Jellyfin Namespace:** media
+**Database Location:** /config/data/jellyfin.db (NFS-backed PVC)