neon-vortex/NAMESPACE_MIGRATION_GUIDE.md

# Namespace Migration Complete!

The Neon Vortex application has been successfully reorganized into dedicated namespaces with automatic cleanup.

## New Namespace Structure

### `neonvortex` Namespace
**Purpose**: Application deployment and webhook listener

**Resources**:
- `HelmRelease/neon-vortex` - Main application deployment
- `Deployment/neon-vortex` - Application pods (2 replicas)
- `Service/neon-vortex` - ClusterIP service
- `Deployment/webhook-build-listener` - Webhook receiver for builds
- `Service/webhook-build-listener` - NodePort 30091
- `CronJob/neon-vortex-build-trigger` - Backup build trigger (every 5 min)
- `ServiceAccount/build-trigger-sa` - For creating build jobs
- `PVC/build-trigger-data` - Stores last commit hash

### `flux-builds` Namespace
**Purpose**: Build job isolation and management

**Resources**:
- Build Jobs (e.g., `build-<commit>-<timestamp>`)
- `CronJob/cleanup-old-build-jobs` - Automatic cleanup (every 30 min)
- `ServiceAccount/job-cleanup-sa` - For cleanup operations

### `flux-system` Namespace
**Purpose**: Flux CD control plane (unchanged)

**Resources**:
- `GitRepository/neon-vortex` - Git source
- `Receiver/neon-vortex-receiver` - Webhook endpoint
- `Provider/neon-vortex-notify` - Gotify notifications
- `Alert/neon-vortex-*` - Event notifications

## Automatic Cleanup

### Build Job TTL
All build jobs now have `ttlSecondsAfterFinished: 1800` (30 minutes)
- Jobs are automatically deleted 30 minutes after completion
- Reduces cluster clutter

### Cleanup CronJob
Runs every 30 minutes to clean up:

1. **Old Successful Jobs**: Keeps only last 10 successful builds
2. **Failed Jobs**: Deletes failures older than 1 hour
3. **Ancient Jobs**: Removes anything older than 24 hours

## Monitoring Commands

### Check Application Status
```bash
# Overall status
kubectl get all -n neonvortex

# HelmRelease status
flux get helmrelease -n neonvortex

# Check pods
kubectl get pods -n neonvortex

# View application logs
kubectl logs -n neonvortex deployment/neon-vortex -f
```

### Monitor Build Jobs
```bash
# List all builds
kubectl get jobs -n flux-builds

# Watch for new builds
kubectl get jobs -n flux-builds -w

# View specific build logs
kubectl logs -n flux-builds job/<job-name> -c kaniko -f

# Check cleanup CronJob
kubectl get cronjob -n flux-builds
```

### Check Webhook Listener
```bash
# View webhook logs
kubectl logs -n neonvortex deployment/webhook-build-listener -f

# Test webhook
curl -X POST http://192.168.1.50:30091/webhook
```

### Verify Cleanup
```bash
# Watch cleanup job run
kubectl get jobs -n flux-builds -l job-name=cleanup-old-build-jobs

# View cleanup logs
kubectl logs -n flux-builds -l job-name=cleanup-old-build-jobs --tail=50
```

## Namespace Resource Quotas

Current setup has NO quotas. To add protection:

```bash
# Example: Limit flux-builds namespace
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: build-quota
  namespace: flux-builds
spec:
  hard:
    pods: "20"
    jobs.batch: "15"
EOF
```

## Migration Summary

### What Changed
✅ Application moved: `default` → `neonvortex`
✅ Build jobs moved: `default` → `flux-builds`
✅ Default namespace cleaned up
✅ Added automatic job cleanup (30 min TTL + CronJob)
✅ Notifications still work (Gotify)
✅ Webhooks still work (NodePort 30091)

### What Stayed the Same
- Flux components remain in `flux-system`
- Harbor registry at `images.caffeinetux.com`
- Image tags: `latest` and `<commit-sha>`
- Gitea repository unchanged
- Webhook URLs unchanged (same NodePorts)

## Updated Webhook URLs

No changes needed! The webhook endpoints remain the same:

1. **Flux Receiver**: `http://192.168.1.50:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a`
2. **Build Trigger**: `http://192.168.1.50:30091/webhook`

## Troubleshooting

### Jobs Not Being Cleaned Up

Check cleanup CronJob:
```bash
kubectl describe cronjob cleanup-old-build-jobs -n flux-builds
kubectl get jobs -n flux-builds -l cronjob=cleanup-old-build-jobs
```

Manually trigger cleanup:
```bash
kubectl create job --from=cronjob/cleanup-old-build-jobs manual-cleanup -n flux-builds
```

### Application Not Deploying

Check HelmRelease:
```bash
flux get helmrelease neon-vortex -n neonvortex
kubectl describe helmrelease neon-vortex -n neonvortex
```

Force reconciliation:
```bash
flux reconcile helmrelease neon-vortex -n neonvortex
```

### Builds Not Triggering

Check webhook listener:
```bash
kubectl get deployment webhook-build-listener -n neonvortex
kubectl logs -n neonvortex deployment/webhook-build-listener
```

Check RBAC permissions:
```bash
kubectl get role build-trigger-role -n flux-builds
kubectl get rolebinding build-trigger-binding -n flux-builds
```

### Harbor Secret Missing

Copy secret if needed:
```bash
# To neonvortex
kubectl get secret harbor-registry -n default -o yaml | \
  sed 's/namespace: default/namespace: neonvortex/' | \
  kubectl apply -f -

# To flux-builds
kubectl get secret harbor-registry -n default -o yaml | \
  sed 's/namespace: default/namespace: flux-builds/' | \
  kubectl apply -f -
```

## Cleanup Policies

### Current Configuration

| Resource Type | Cleanup Method | Retention |
|--------------|----------------|-----------|
| Build Jobs | TTL | 30 minutes after completion |
| Successful Builds | CronJob | Keep last 10 |
| Failed Builds | CronJob | 1 hour |
| Any Build | CronJob | 24 hours max |
| Cleanup Jobs | successfulJobsHistoryLimit | 1 |
| CronJob Trigger | successfulJobsHistoryLimit | 1 |

### Adjusting Cleanup

Edit cleanup CronJob:
```bash
kubectl edit cronjob cleanup-old-build-jobs -n flux-builds
```

Change:
- **Schedule**: Modify `schedule: "*/30 * * * *"` (every 30 min)
- **Keep N builds**: Change `head -n -10` to keep more/fewer
- **Failed job age**: Change `1 hour ago` to different duration
- **Max age**: Change `24 hours ago` to different duration

### Disable Cleanup Temporarily

```bash
# Suspend cleanup CronJob
kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":true}}'

# Resume
kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":false}}'
```

## Performance Impact

### Before Migration
- All resources in `default` namespace
- Jobs accumulating indefinitely
- Manual cleanup required

### After Migration
- Clean namespace separation
- Automatic cleanup every 30 minutes
- Build history maintained (last 10 successful)
- Reduced cluster resource usage
- Improved visibility and monitoring

## Quick Reference

```bash
# One-liner status check
kubectl get helmrelease,deployment,pods,jobs,cronjob --all-namespaces | grep neon

# Check all namespaces
kubectl get ns | grep -E "neonvortex|flux-builds|flux-system"

# Total resource usage
kubectl top pods -n neonvortex
kubectl top pods -n flux-builds

# Events
kubectl get events -n neonvortex --sort-by='.lastTimestamp' | tail -20
kubectl get events -n flux-builds --sort-by='.lastTimestamp' | tail -20
```

## Next Steps

Consider these enhancements:

1. **Network Policies**: Isolate namespace traffic
2. **Resource Quotas**: Limit namespace resource usage
3. **Pod Security**: Add PodSecurityAdmissions
4. **Monitoring**: Add Prometheus metrics for build success rate
5. **Alerting**: Integrate with AlertManager for failures
6. **Multi-tenancy**: If deploying more apps, follow this pattern

## Files Updated

New files created:
- `namespaces.yaml` - Namespace definitions
- `flux-helmrelease-neonvortex.yaml` - Updated HelmRelease
- `webhook-build-trigger-neonvortex.yaml` - Webhook in new namespace
- `build-trigger-cronjob-neonvortex.yaml` - Backup CronJob
- `build-job-cleanup.yaml` - Automated cleanup
- `flux-alerts-neonvortex.yaml` - Updated alerts

Old files (can be removed):
- `flux-helmrelease.yaml` (replaced)
- `webhook-build-trigger.yaml` (replaced)
- `build-trigger-cronjob.yaml` (replaced)
- `flux-alerts.yaml` (replaced)