diff --git a/NAMESPACE_MIGRATION_GUIDE.md b/NAMESPACE_MIGRATION_GUIDE.md new file mode 100644 index 0000000..636254a --- /dev/null +++ b/NAMESPACE_MIGRATION_GUIDE.md @@ -0,0 +1,294 @@ +# Namespace Migration Complete! + +The Neon Vortex application has been successfully reorganized into dedicated namespaces with automatic cleanup. + +## New Namespace Structure + +### `neonvortex` Namespace +**Purpose**: Application deployment and webhook listener + +**Resources**: +- `HelmRelease/neon-vortex` - Main application deployment +- `Deployment/neon-vortex` - Application pods (2 replicas) +- `Service/neon-vortex` - ClusterIP service +- `Deployment/webhook-build-listener` - Webhook receiver for builds +- `Service/webhook-build-listener` - NodePort 30091 +- `CronJob/neon-vortex-build-trigger` - Backup build trigger (every 5 min) +- `ServiceAccount/build-trigger-sa` - For creating build jobs +- `PVC/build-trigger-data` - Stores last commit hash + +### `flux-builds` Namespace +**Purpose**: Build job isolation and management + +**Resources**: +- Build Jobs (e.g., `build--`) +- `CronJob/cleanup-old-build-jobs` - Automatic cleanup (every 30 min) +- `ServiceAccount/job-cleanup-sa` - For cleanup operations + +### `flux-system` Namespace +**Purpose**: Flux CD control plane (unchanged) + +**Resources**: +- `GitRepository/neon-vortex` - Git source +- `Receiver/neon-vortex-receiver` - Webhook endpoint +- `Provider/neon-vortex-notify` - Gotify notifications +- `Alert/neon-vortex-*` - Event notifications + +## Automatic Cleanup + +### Build Job TTL +All build jobs now have `ttlSecondsAfterFinished: 1800` (30 minutes) +- Jobs are automatically deleted 30 minutes after completion +- Reduces cluster clutter + +### Cleanup CronJob +Runs every 30 minutes to clean up: + +1. **Old Successful Jobs**: Keeps only last 10 successful builds +2. **Failed Jobs**: Deletes failures older than 1 hour +3. **Ancient Jobs**: Removes anything older than 24 hours + +## Monitoring Commands + +### Check Application Status +```bash +# Overall status +kubectl get all -n neonvortex + +# HelmRelease status +flux get helmrelease -n neonvortex + +# Check pods +kubectl get pods -n neonvortex + +# View application logs +kubectl logs -n neonvortex deployment/neon-vortex -f +``` + +### Monitor Build Jobs +```bash +# List all builds +kubectl get jobs -n flux-builds + +# Watch for new builds +kubectl get jobs -n flux-builds -w + +# View specific build logs +kubectl logs -n flux-builds job/ -c kaniko -f + +# Check cleanup CronJob +kubectl get cronjob -n flux-builds +``` + +### Check Webhook Listener +```bash +# View webhook logs +kubectl logs -n neonvortex deployment/webhook-build-listener -f + +# Test webhook +curl -X POST http://192.168.1.50:30091/webhook +``` + +### Verify Cleanup +```bash +# Watch cleanup job run +kubectl get jobs -n flux-builds -l job-name=cleanup-old-build-jobs + +# View cleanup logs +kubectl logs -n flux-builds -l job-name=cleanup-old-build-jobs --tail=50 +``` + +## Namespace Resource Quotas + +Current setup has NO quotas. To add protection: + +```bash +# Example: Limit flux-builds namespace +kubectl apply -f - <` +- Gitea repository unchanged +- Webhook URLs unchanged (same NodePorts) + +## Updated Webhook URLs + +No changes needed! The webhook endpoints remain the same: + +1. **Flux Receiver**: `http://192.168.1.50:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a` +2. **Build Trigger**: `http://192.168.1.50:30091/webhook` + +## Troubleshooting + +### Jobs Not Being Cleaned Up + +Check cleanup CronJob: +```bash +kubectl describe cronjob cleanup-old-build-jobs -n flux-builds +kubectl get jobs -n flux-builds -l cronjob=cleanup-old-build-jobs +``` + +Manually trigger cleanup: +```bash +kubectl create job --from=cronjob/cleanup-old-build-jobs manual-cleanup -n flux-builds +``` + +### Application Not Deploying + +Check HelmRelease: +```bash +flux get helmrelease neon-vortex -n neonvortex +kubectl describe helmrelease neon-vortex -n neonvortex +``` + +Force reconciliation: +```bash +flux reconcile helmrelease neon-vortex -n neonvortex +``` + +### Builds Not Triggering + +Check webhook listener: +```bash +kubectl get deployment webhook-build-listener -n neonvortex +kubectl logs -n neonvortex deployment/webhook-build-listener +``` + +Check RBAC permissions: +```bash +kubectl get role build-trigger-role -n flux-builds +kubectl get rolebinding build-trigger-binding -n flux-builds +``` + +### Harbor Secret Missing + +Copy secret if needed: +```bash +# To neonvortex +kubectl get secret harbor-registry -n default -o yaml | \ + sed 's/namespace: default/namespace: neonvortex/' | \ + kubectl apply -f - + +# To flux-builds +kubectl get secret harbor-registry -n default -o yaml | \ + sed 's/namespace: default/namespace: flux-builds/' | \ + kubectl apply -f - +``` + +## Cleanup Policies + +### Current Configuration + +| Resource Type | Cleanup Method | Retention | +|--------------|----------------|-----------| +| Build Jobs | TTL | 30 minutes after completion | +| Successful Builds | CronJob | Keep last 10 | +| Failed Builds | CronJob | 1 hour | +| Any Build | CronJob | 24 hours max | +| Cleanup Jobs | successfulJobsHistoryLimit | 1 | +| CronJob Trigger | successfulJobsHistoryLimit | 1 | + +### Adjusting Cleanup + +Edit cleanup CronJob: +```bash +kubectl edit cronjob cleanup-old-build-jobs -n flux-builds +``` + +Change: +- **Schedule**: Modify `schedule: "*/30 * * * *"` (every 30 min) +- **Keep N builds**: Change `head -n -10` to keep more/fewer +- **Failed job age**: Change `1 hour ago` to different duration +- **Max age**: Change `24 hours ago` to different duration + +### Disable Cleanup Temporarily + +```bash +# Suspend cleanup CronJob +kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":true}}' + +# Resume +kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":false}}' +``` + +## Performance Impact + +### Before Migration +- All resources in `default` namespace +- Jobs accumulating indefinitely +- Manual cleanup required + +### After Migration +- Clean namespace separation +- Automatic cleanup every 30 minutes +- Build history maintained (last 10 successful) +- Reduced cluster resource usage +- Improved visibility and monitoring + +## Quick Reference + +```bash +# One-liner status check +kubectl get helmrelease,deployment,pods,jobs,cronjob --all-namespaces | grep neon + +# Check all namespaces +kubectl get ns | grep -E "neonvortex|flux-builds|flux-system" + +# Total resource usage +kubectl top pods -n neonvortex +kubectl top pods -n flux-builds + +# Events +kubectl get events -n neonvortex --sort-by='.lastTimestamp' | tail -20 +kubectl get events -n flux-builds --sort-by='.lastTimestamp' | tail -20 +``` + +## Next Steps + +Consider these enhancements: + +1. **Network Policies**: Isolate namespace traffic +2. **Resource Quotas**: Limit namespace resource usage +3. **Pod Security**: Add PodSecurityAdmissions +4. **Monitoring**: Add Prometheus metrics for build success rate +5. **Alerting**: Integrate with AlertManager for failures +6. **Multi-tenancy**: If deploying more apps, follow this pattern + +## Files Updated + +New files created: +- `namespaces.yaml` - Namespace definitions +- `flux-helmrelease-neonvortex.yaml` - Updated HelmRelease +- `webhook-build-trigger-neonvortex.yaml` - Webhook in new namespace +- `build-trigger-cronjob-neonvortex.yaml` - Backup CronJob +- `build-job-cleanup.yaml` - Automated cleanup +- `flux-alerts-neonvortex.yaml` - Updated alerts + +Old files (can be removed): +- `flux-helmrelease.yaml` (replaced) +- `webhook-build-trigger.yaml` (replaced) +- `build-trigger-cronjob.yaml` (replaced) +- `flux-alerts.yaml` (replaced) diff --git a/neon-vortex-ingress-neonvortex.yaml b/neon-vortex-ingress-neonvortex.yaml new file mode 100644 index 0000000..90e0409 --- /dev/null +++ b/neon-vortex-ingress-neonvortex.yaml @@ -0,0 +1,25 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: neon-vortex + namespace: neonvortex + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + nginx.ingress.kubernetes.io/ssl-redirect: "true" +spec: + ingressClassName: nginx + rules: + - host: nv.caffeinetux.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: neon-vortex + port: + number: 80 + tls: + - hosts: + - nv.caffeinetux.com + secretName: neon-vortex-tls