neon-vortex/JELLYFIN_ERROR_ANALYSIS.md

# Jellyfin Error Analysis and Recommendations

## Overview
Analysis of Jellyfin errors found in the media namespace logs, with recommendations for resolution.

---

## Issues Identified

### 1. **SQLite Database Locking Errors** (CRITICAL)

**Error Messages:**
```
[04:28:37] [ERR] Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
[09:00:00] [ERR] SQLite Error 5: 'database is locked For more information on this error code see https://www.sqlite.org/rescode.html'.
```

**Impact:**
- Prevents metadata updates during library scans
- Causes API request failures (GET /Items)
- Occurs when database optimization runs concurrently with library operations

**Root Cause:**
SQLite's default locking mechanism allows only one writer at a time. The errors occur when:
1. Database optimization task runs (VACUUM operation)
2. Library scan attempts to update metadata simultaneously
3. Both operations try to write to the database

**Frequency:**
- Database optimization runs approximately every 6 hours
- Library scans run every 12 hours
- Conflicts occur when these overlap

---

### 2. **Audio Normalization Failures** (MODERATE)

**Error Message:**
```
[04:24:54] [ERR] Emby.Server.Implementations.ScheduledTasks.Tasks.AudioNormalizationTask: Failed to find LUFS value in output
```

**Impact:**
- Audio normalization task cannot measure loudness
- Prevents automatic volume leveling across music library
- Task completes without performing any normalization

**Root Cause:**
The LUFS (Loudness Units relative to Full Scale) detection failure suggests:
1. FFmpeg may be missing required audio analysis filters
2. Audio files may have incompatible formats
3. FFmpeg output parsing may be failing

**Frequency:**
- Runs daily at 04:24 (during scheduled tasks)
- Consistently fails to find LUFS values

---

### 3. **Library Folder Warnings** (LOW PRIORITY)

**Warning Messages:**
```
[04:25:47] [WRN] Library folder /ebooks is inaccessible or empty, skipping
[04:25:47] [WRN] Library folder /config/data/playlists is inaccessible or empty, skipping
```

**Impact:**
- Minimal - warnings only
- No functional issues if these folders are intentionally empty
- Causes log noise

**Root Cause:**
- Library folders are configured but:
  - `/ebooks` folder doesn't exist or is empty
  - `/config/data/playlists` folder is empty
- Jellyfin skips these during library scans

---

## Recommendations

### Priority 1: Fix Database Locking Issues

#### Option A: Enable WAL Mode (RECOMMENDED)

WAL (Write-Ahead Logging) mode allows concurrent readers during writes, reducing lock contention.

**Implementation:**

1. **Create a job to enable WAL mode:**

Create file: `/storage/emulated/0/Download/nv/jellyfin-wal-enable.yaml`

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: jellyfin-enable-wal
  namespace: media
spec:
  ttlSecondsAfterFinished: 600
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: enable-wal
        image: alpine:latest
        command:
        - sh
        - -c
        - |
          apk add --no-cache sqlite
          echo "Enabling WAL mode on jellyfin.db..."
          sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode=WAL;"
          echo "Verifying WAL mode..."
          sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode;"
          echo "Done!"
        volumeMounts:
        - name: jellyfin-data
          mountPath: /config
      volumes:
      - name: jellyfin-data
        persistentVolumeClaim:
          claimName: jellyfin-config
```

**Benefits:**
- Multiple readers can access database during writes
- Reduces lock contention significantly
- Better performance under concurrent load
- Industry standard for high-concurrency SQLite

**Caveats:**
- Creates additional files (`jellyfin.db-wal`, `jellyfin.db-shm`)
- Requires slightly more disk space
- NFS must support file locking (appears to work based on current setup)

---

#### Option B: Adjust Task Scheduling

Prevent database optimization and library scans from running simultaneously.

**Current Schedule:**
- Library scan: Every 12 hours (04:26, 16:26)
- Database optimization: Every 6 hours (04:27, 10:28, 16:29)

**Recommended Change:**
Modify Jellyfin's scheduled tasks via the web UI:
1. Go to Dashboard → Scheduled Tasks
2. Change "Optimize database" to run at different times:
   - Change from every 6 hours to daily at 03:00 (before library scan)
3. This ensures optimization completes before library scan starts

**Benefits:**
- Simple configuration change
- No code modifications needed
- Reduces overlap probability

**Caveats:**
- Doesn't eliminate all race conditions
- API requests during optimization can still fail
- Less robust than WAL mode

---

#### Option C: Increase SQLite Timeout

Configure Jellyfin to wait longer when database is locked.

**Implementation:**
Requires adding environment variable to Jellyfin deployment:

```yaml
env:
- name: SQLITE_BUSY_TIMEOUT
  value: "30000"  # 30 seconds
```

**Note:** This may not be supported by Jellyfin directly. WAL mode is preferred.

---

### Priority 2: Fix Audio Normalization

#### Diagnosis Steps:

1. **Check FFmpeg capabilities:**

```bash
kubectl exec -n media deployment/jellyfin -- ffmpeg -filters 2>&1 | grep -E "(loudnorm|ebur128)"
```

2. **Test LUFS detection manually:**

```bash
kubectl exec -n media deployment/jellyfin -- ffmpeg -i /music/sample.mp3 -af loudnorm=print_format=json -f null - 2>&1
```

Expected output should include:
```json
{
  "input_i": "-23.0",
  "input_tp": "-5.0",
  "input_lra": "7.0",
  "input_thresh": "-33.5",
  "target_offset": "0.0"
}
```

#### Solution Options:

**If filters are missing:**
- Jellyfin's FFmpeg build may lack audio normalization filters
- Solution: Disable audio normalization task in Jellyfin (Dashboard → Scheduled Tasks)

**If filters exist but fail:**
- Some audio files may have incompatible formats
- Solution: Check task logs for specific files causing issues

**To disable audio normalization:**
1. Go to Dashboard → Scheduled Tasks
2. Find "Audio Normalization"
3. Click on it and disable the task
4. This is acceptable - normalization is optional for most users

---

### Priority 3: Clean Up Library Warnings

#### Option A: Remove Empty Library Folders

If you don't use ebooks or playlists:

1. Go to Dashboard → Libraries
2. Remove the "ebooks" library if it exists
3. The playlists folder is auto-created and can be ignored

#### Option B: Create the Folders

If you plan to use these features:

```bash
kubectl exec -n media deployment/jellyfin -- mkdir -p /ebooks
kubectl exec -n media deployment/jellyfin -- mkdir -p /config/data/playlists
```

---

## Implementation Plan

### Step 1: Enable WAL Mode (RECOMMENDED)

This will resolve the critical database locking issues.

```bash
# Apply the WAL enable job
kubectl apply -f jellyfin-wal-enable.yaml

# Wait for completion
kubectl wait --for=condition=complete --timeout=300s job/jellyfin-enable-wal -n media

# Check logs
kubectl logs -n media job/jellyfin-enable-wal

# Restart Jellyfin to ensure WAL mode is active
kubectl rollout restart deployment/jellyfin -n media
```

### Step 2: Disable Audio Normalization (OPTIONAL)

Via Jellyfin Web UI:
1. Navigate to https://jellyfin.caffeinetux.com/web/index.html#!/scheduledtasks.html
2. Find "Audio Normalization"
3. Click "Disable"

### Step 3: Monitor Logs

After implementing WAL mode:

```bash
# Watch for database lock errors (should disappear)
kubectl logs -n media deployment/jellyfin -f | grep "database is locked"

# Check if WAL mode is active
kubectl exec -n media deployment/jellyfin -- sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode;"
# Should output: wal
```

---

## Performance Impact Analysis

### Current Issues:
- Database locks during library scans (2+ minutes)
- Failed API requests during optimization
- Metadata updates delayed or skipped

### After WAL Mode:
- ✅ Concurrent reads during writes
- ✅ API requests succeed during optimization
- ✅ Faster metadata updates
- ✅ Better overall responsiveness

### Trade-offs:
- Slightly more disk space (typically <10% of DB size)
- Additional files in config directory
- Minimal performance overhead (usually net positive)

---

## Additional Recommendations

### 1. Database Optimization Frequency

Current: Every 6 hours may be excessive

**Recommendation:** Change to once daily at 03:00

Benefits:
- Reduces I/O load on NFS
- Less frequent database locks
- Sufficient for most libraries

### 2. Library Scan Optimization

**Current behavior:**
- Full library scan every 12 hours
- Stops directory watching during scan

**Recommendation:**
- Keep 12-hour schedule for now
- Monitor scan duration (currently 2-5 minutes)
- If scans take too long, consider:
  - Increasing interval to 24 hours
  - Using real-time monitoring only

### 3. NFS Considerations

Your Jellyfin database is on NFS. WAL mode works with NFS if:
- ✅ File locking is supported (appears to be working)
- ✅ NFS client supports mmap operations
- ⚠️ Network latency is low (check if you experience slowness)

**If WAL mode causes issues on NFS:**
- Switch back to DELETE mode: `PRAGMA journal_mode=DELETE;`
- Consider moving database to local storage (emptyDir volume)
- Keep media on NFS, database local

---

## Monitoring and Validation

### Success Criteria:

After implementing WAL mode, verify:

1. **No more database lock errors:**
```bash
kubectl logs -n media deployment/jellyfin --since=24h | grep "database is locked"
# Should return no results
```

2. **WAL mode is active:**
```bash
kubectl exec -n media deployment/jellyfin -- sqlite3 /config/data/jellyfin.db "PRAGMA journal_mode;"
# Should output: wal
```

3. **Library scans complete successfully:**
```bash
kubectl logs -n media deployment/jellyfin | grep "Scan Media Library Completed"
# Should show successful completions
```

4. **Database optimization runs without conflicts:**
```bash
kubectl logs -n media deployment/jellyfin | grep "jellyfin.db optimized successfully"
# Should show successful optimizations
```

---

## Summary

### Critical Issues (Fix Now):
1. **SQLite Database Locking** → Enable WAL mode

### Optional Improvements:
2. **Audio Normalization** → Disable if not needed
3. **Library Warnings** → Remove unused libraries or create folders
4. **Task Scheduling** → Adjust optimization frequency to daily

### Expected Outcome:
- Elimination of database lock errors
- Smoother library scans
- Better API responsiveness
- Cleaner logs

---

## Resources

- [SQLite WAL Mode Documentation](https://www.sqlite.org/wal.html)
- [Jellyfin Database Documentation](https://jellyfin.org/docs/general/administration/configuration.html)
- [SQLite Error Codes](https://www.sqlite.org/rescode.html)

---

**Analysis Date:** 2025-11-22
**Jellyfin Namespace:** media
**Database Location:** /config/data/jellyfin.db (NFS-backed PVC)