neon-vortex/NAMESPACE_MIGRATION_GUIDE.md

Namespace Migration Complete!

The Neon Vortex application has been successfully reorganized into dedicated namespaces with automatic cleanup.

New Namespace Structure

neonvortex Namespace

Purpose: Application deployment and webhook listener

Resources:

• HelmRelease/neon-vortex - Main application deployment
• Deployment/neon-vortex - Application pods (2 replicas)
• Service/neon-vortex - ClusterIP service
• Deployment/webhook-build-listener - Webhook receiver for builds
• Service/webhook-build-listener - NodePort 30091
• CronJob/neon-vortex-build-trigger - Backup build trigger (every 5 min)
• ServiceAccount/build-trigger-sa - For creating build jobs
• PVC/build-trigger-data - Stores last commit hash

flux-builds Namespace

Purpose: Build job isolation and management

Resources:

• Build Jobs (e.g., build-<commit>-<timestamp>)
• CronJob/cleanup-old-build-jobs - Automatic cleanup (every 30 min)
• ServiceAccount/job-cleanup-sa - For cleanup operations

flux-system Namespace

Purpose: Flux CD control plane (unchanged)

Resources:

• GitRepository/neon-vortex - Git source
• Receiver/neon-vortex-receiver - Webhook endpoint
• Provider/neon-vortex-notify - Gotify notifications
• Alert/neon-vortex-* - Event notifications

Automatic Cleanup

Build Job TTL

All build jobs now have ttlSecondsAfterFinished: 1800 (30 minutes)

• Jobs are automatically deleted 30 minutes after completion
• Reduces cluster clutter

Cleanup CronJob

Runs every 30 minutes to clean up:

1. Old Successful Jobs: Keeps only last 10 successful builds
2. Failed Jobs: Deletes failures older than 1 hour
3. Ancient Jobs: Removes anything older than 24 hours

Monitoring Commands

Check Application Status

# Overall status
kubectl get all -n neonvortex

# HelmRelease status
flux get helmrelease -n neonvortex

# Check pods
kubectl get pods -n neonvortex

# View application logs
kubectl logs -n neonvortex deployment/neon-vortex -f

Monitor Build Jobs

# List all builds
kubectl get jobs -n flux-builds

# Watch for new builds
kubectl get jobs -n flux-builds -w

# View specific build logs
kubectl logs -n flux-builds job/<job-name> -c kaniko -f

# Check cleanup CronJob
kubectl get cronjob -n flux-builds

Check Webhook Listener

# View webhook logs
kubectl logs -n neonvortex deployment/webhook-build-listener -f

# Test webhook
curl -X POST http://192.168.1.50:30091/webhook

Verify Cleanup

# Watch cleanup job run
kubectl get jobs -n flux-builds -l job-name=cleanup-old-build-jobs

# View cleanup logs
kubectl logs -n flux-builds -l job-name=cleanup-old-build-jobs --tail=50

Namespace Resource Quotas

Current setup has NO quotas. To add protection:

# Example: Limit flux-builds namespace
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: build-quota
  namespace: flux-builds
spec:
  hard:
    pods: "20"
    jobs.batch: "15"
EOF

Migration Summary

What Changed

✅ Application moved: default → neonvortex ✅ Build jobs moved: default → flux-builds ✅ Default namespace cleaned up ✅ Added automatic job cleanup (30 min TTL + CronJob) ✅ Notifications still work (Gotify) ✅ Webhooks still work (NodePort 30091)

What Stayed the Same

• Flux components remain in flux-system
• Harbor registry at images.caffeinetux.com
• Image tags: latest and <commit-sha>
• Gitea repository unchanged
• Webhook URLs unchanged (same NodePorts)

Updated Webhook URLs

No changes needed! The webhook endpoints remain the same:

1. Flux Receiver: http://192.168.1.50:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
2. Build Trigger: http://192.168.1.50:30091/webhook

Troubleshooting

Jobs Not Being Cleaned Up

Check cleanup CronJob:

kubectl describe cronjob cleanup-old-build-jobs -n flux-builds
kubectl get jobs -n flux-builds -l cronjob=cleanup-old-build-jobs

Manually trigger cleanup:

kubectl create job --from=cronjob/cleanup-old-build-jobs manual-cleanup -n flux-builds

Application Not Deploying

Check HelmRelease:

flux get helmrelease neon-vortex -n neonvortex
kubectl describe helmrelease neon-vortex -n neonvortex

Force reconciliation:

flux reconcile helmrelease neon-vortex -n neonvortex

Builds Not Triggering

Check webhook listener:

kubectl get deployment webhook-build-listener -n neonvortex
kubectl logs -n neonvortex deployment/webhook-build-listener

Check RBAC permissions:

kubectl get role build-trigger-role -n flux-builds
kubectl get rolebinding build-trigger-binding -n flux-builds

Harbor Secret Missing

Copy secret if needed:

# To neonvortex
kubectl get secret harbor-registry -n default -o yaml | \
  sed 's/namespace: default/namespace: neonvortex/' | \
  kubectl apply -f -

# To flux-builds
kubectl get secret harbor-registry -n default -o yaml | \
  sed 's/namespace: default/namespace: flux-builds/' | \
  kubectl apply -f -

Cleanup Policies

Current Configuration

Resource Type	Cleanup Method	Retention
Build Jobs	TTL	30 minutes after completion
Successful Builds	CronJob	Keep last 10
Failed Builds	CronJob	1 hour
Any Build	CronJob	24 hours max
Cleanup Jobs	successfulJobsHistoryLimit	1
CronJob Trigger	successfulJobsHistoryLimit	1

Adjusting Cleanup

Edit cleanup CronJob:

kubectl edit cronjob cleanup-old-build-jobs -n flux-builds

Change:

• Schedule: Modify schedule: "*/30 * * * *" (every 30 min)
• Keep N builds: Change head -n -10 to keep more/fewer
• Failed job age: Change 1 hour ago to different duration
• Max age: Change 24 hours ago to different duration

Disable Cleanup Temporarily

# Suspend cleanup CronJob
kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":true}}'

# Resume
kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":false}}'

Performance Impact

Before Migration

• All resources in default namespace
• Jobs accumulating indefinitely
• Manual cleanup required

After Migration

• Clean namespace separation
• Automatic cleanup every 30 minutes
• Build history maintained (last 10 successful)
• Reduced cluster resource usage
• Improved visibility and monitoring

Quick Reference

# One-liner status check
kubectl get helmrelease,deployment,pods,jobs,cronjob --all-namespaces | grep neon

# Check all namespaces
kubectl get ns | grep -E "neonvortex|flux-builds|flux-system"

# Total resource usage
kubectl top pods -n neonvortex
kubectl top pods -n flux-builds

# Events
kubectl get events -n neonvortex --sort-by='.lastTimestamp' | tail -20
kubectl get events -n flux-builds --sort-by='.lastTimestamp' | tail -20

Next Steps

Consider these enhancements:

1. Network Policies: Isolate namespace traffic
2. Resource Quotas: Limit namespace resource usage
3. Pod Security: Add PodSecurityAdmissions
4. Monitoring: Add Prometheus metrics for build success rate
5. Alerting: Integrate with AlertManager for failures
6. Multi-tenancy: If deploying more apps, follow this pattern

Files Updated

New files created:

• namespaces.yaml - Namespace definitions
• flux-helmrelease-neonvortex.yaml - Updated HelmRelease
• webhook-build-trigger-neonvortex.yaml - Webhook in new namespace
• build-trigger-cronjob-neonvortex.yaml - Backup CronJob
• build-job-cleanup.yaml - Automated cleanup
• flux-alerts-neonvortex.yaml - Updated alerts

Old files (can be removed):

• flux-helmrelease.yaml (replaced)
• webhook-build-trigger.yaml (replaced)
• build-trigger-cronjob.yaml (replaced)
• flux-alerts.yaml (replaced)