Fix 503 error by moving ingress to neonvortex namespace
Some checks failed
Build and Push to Harbor / build-and-push (push) Has been cancelled
Some checks failed
Build and Push to Harbor / build-and-push (push) Has been cancelled
The ingress was still in the default namespace trying to route to a service that no longer exists there. Moved ingress to the neonvortex namespace where the service actually resides. Fixes: 503 Service Unavailable error Tested: https://nv.caffeinetux.com now returns HTTP 200 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Neon Vortex
294
NAMESPACE_MIGRATION_GUIDE.md
Normal file
294
NAMESPACE_MIGRATION_GUIDE.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# Namespace Migration Complete!
|
||||
|
||||
The Neon Vortex application has been successfully reorganized into dedicated namespaces with automatic cleanup.
|
||||
|
||||
## New Namespace Structure
|
||||
|
||||
### `neonvortex` Namespace
|
||||
**Purpose**: Application deployment and webhook listener
|
||||
|
||||
**Resources**:
|
||||
- `HelmRelease/neon-vortex` - Main application deployment
|
||||
- `Deployment/neon-vortex` - Application pods (2 replicas)
|
||||
- `Service/neon-vortex` - ClusterIP service
|
||||
- `Deployment/webhook-build-listener` - Webhook receiver for builds
|
||||
- `Service/webhook-build-listener` - NodePort 30091
|
||||
- `CronJob/neon-vortex-build-trigger` - Backup build trigger (every 5 min)
|
||||
- `ServiceAccount/build-trigger-sa` - For creating build jobs
|
||||
- `PVC/build-trigger-data` - Stores last commit hash
|
||||
|
||||
### `flux-builds` Namespace
|
||||
**Purpose**: Build job isolation and management
|
||||
|
||||
**Resources**:
|
||||
- Build Jobs (e.g., `build-<commit>-<timestamp>`)
|
||||
- `CronJob/cleanup-old-build-jobs` - Automatic cleanup (every 30 min)
|
||||
- `ServiceAccount/job-cleanup-sa` - For cleanup operations
|
||||
|
||||
### `flux-system` Namespace
|
||||
**Purpose**: Flux CD control plane (unchanged)
|
||||
|
||||
**Resources**:
|
||||
- `GitRepository/neon-vortex` - Git source
|
||||
- `Receiver/neon-vortex-receiver` - Webhook endpoint
|
||||
- `Provider/neon-vortex-notify` - Gotify notifications
|
||||
- `Alert/neon-vortex-*` - Event notifications
|
||||
|
||||
## Automatic Cleanup
|
||||
|
||||
### Build Job TTL
|
||||
All build jobs now have `ttlSecondsAfterFinished: 1800` (30 minutes)
|
||||
- Jobs are automatically deleted 30 minutes after completion
|
||||
- Reduces cluster clutter
|
||||
|
||||
### Cleanup CronJob
|
||||
Runs every 30 minutes to clean up:
|
||||
|
||||
1. **Old Successful Jobs**: Keeps only last 10 successful builds
|
||||
2. **Failed Jobs**: Deletes failures older than 1 hour
|
||||
3. **Ancient Jobs**: Removes anything older than 24 hours
|
||||
|
||||
## Monitoring Commands
|
||||
|
||||
### Check Application Status
|
||||
```bash
|
||||
# Overall status
|
||||
kubectl get all -n neonvortex
|
||||
|
||||
# HelmRelease status
|
||||
flux get helmrelease -n neonvortex
|
||||
|
||||
# Check pods
|
||||
kubectl get pods -n neonvortex
|
||||
|
||||
# View application logs
|
||||
kubectl logs -n neonvortex deployment/neon-vortex -f
|
||||
```
|
||||
|
||||
### Monitor Build Jobs
|
||||
```bash
|
||||
# List all builds
|
||||
kubectl get jobs -n flux-builds
|
||||
|
||||
# Watch for new builds
|
||||
kubectl get jobs -n flux-builds -w
|
||||
|
||||
# View specific build logs
|
||||
kubectl logs -n flux-builds job/<job-name> -c kaniko -f
|
||||
|
||||
# Check cleanup CronJob
|
||||
kubectl get cronjob -n flux-builds
|
||||
```
|
||||
|
||||
### Check Webhook Listener
|
||||
```bash
|
||||
# View webhook logs
|
||||
kubectl logs -n neonvortex deployment/webhook-build-listener -f
|
||||
|
||||
# Test webhook
|
||||
curl -X POST http://192.168.1.50:30091/webhook
|
||||
```
|
||||
|
||||
### Verify Cleanup
|
||||
```bash
|
||||
# Watch cleanup job run
|
||||
kubectl get jobs -n flux-builds -l job-name=cleanup-old-build-jobs
|
||||
|
||||
# View cleanup logs
|
||||
kubectl logs -n flux-builds -l job-name=cleanup-old-build-jobs --tail=50
|
||||
```
|
||||
|
||||
## Namespace Resource Quotas
|
||||
|
||||
Current setup has NO quotas. To add protection:
|
||||
|
||||
```bash
|
||||
# Example: Limit flux-builds namespace
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: ResourceQuota
|
||||
metadata:
|
||||
name: build-quota
|
||||
namespace: flux-builds
|
||||
spec:
|
||||
hard:
|
||||
pods: "20"
|
||||
jobs.batch: "15"
|
||||
EOF
|
||||
```
|
||||
|
||||
## Migration Summary
|
||||
|
||||
### What Changed
|
||||
✅ Application moved: `default` → `neonvortex`
|
||||
✅ Build jobs moved: `default` → `flux-builds`
|
||||
✅ Default namespace cleaned up
|
||||
✅ Added automatic job cleanup (30 min TTL + CronJob)
|
||||
✅ Notifications still work (Gotify)
|
||||
✅ Webhooks still work (NodePort 30091)
|
||||
|
||||
### What Stayed the Same
|
||||
- Flux components remain in `flux-system`
|
||||
- Harbor registry at `images.caffeinetux.com`
|
||||
- Image tags: `latest` and `<commit-sha>`
|
||||
- Gitea repository unchanged
|
||||
- Webhook URLs unchanged (same NodePorts)
|
||||
|
||||
## Updated Webhook URLs
|
||||
|
||||
No changes needed! The webhook endpoints remain the same:
|
||||
|
||||
1. **Flux Receiver**: `http://192.168.1.50:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a`
|
||||
2. **Build Trigger**: `http://192.168.1.50:30091/webhook`
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Jobs Not Being Cleaned Up
|
||||
|
||||
Check cleanup CronJob:
|
||||
```bash
|
||||
kubectl describe cronjob cleanup-old-build-jobs -n flux-builds
|
||||
kubectl get jobs -n flux-builds -l cronjob=cleanup-old-build-jobs
|
||||
```
|
||||
|
||||
Manually trigger cleanup:
|
||||
```bash
|
||||
kubectl create job --from=cronjob/cleanup-old-build-jobs manual-cleanup -n flux-builds
|
||||
```
|
||||
|
||||
### Application Not Deploying
|
||||
|
||||
Check HelmRelease:
|
||||
```bash
|
||||
flux get helmrelease neon-vortex -n neonvortex
|
||||
kubectl describe helmrelease neon-vortex -n neonvortex
|
||||
```
|
||||
|
||||
Force reconciliation:
|
||||
```bash
|
||||
flux reconcile helmrelease neon-vortex -n neonvortex
|
||||
```
|
||||
|
||||
### Builds Not Triggering
|
||||
|
||||
Check webhook listener:
|
||||
```bash
|
||||
kubectl get deployment webhook-build-listener -n neonvortex
|
||||
kubectl logs -n neonvortex deployment/webhook-build-listener
|
||||
```
|
||||
|
||||
Check RBAC permissions:
|
||||
```bash
|
||||
kubectl get role build-trigger-role -n flux-builds
|
||||
kubectl get rolebinding build-trigger-binding -n flux-builds
|
||||
```
|
||||
|
||||
### Harbor Secret Missing
|
||||
|
||||
Copy secret if needed:
|
||||
```bash
|
||||
# To neonvortex
|
||||
kubectl get secret harbor-registry -n default -o yaml | \
|
||||
sed 's/namespace: default/namespace: neonvortex/' | \
|
||||
kubectl apply -f -
|
||||
|
||||
# To flux-builds
|
||||
kubectl get secret harbor-registry -n default -o yaml | \
|
||||
sed 's/namespace: default/namespace: flux-builds/' | \
|
||||
kubectl apply -f -
|
||||
```
|
||||
|
||||
## Cleanup Policies
|
||||
|
||||
### Current Configuration
|
||||
|
||||
| Resource Type | Cleanup Method | Retention |
|
||||
|--------------|----------------|-----------|
|
||||
| Build Jobs | TTL | 30 minutes after completion |
|
||||
| Successful Builds | CronJob | Keep last 10 |
|
||||
| Failed Builds | CronJob | 1 hour |
|
||||
| Any Build | CronJob | 24 hours max |
|
||||
| Cleanup Jobs | successfulJobsHistoryLimit | 1 |
|
||||
| CronJob Trigger | successfulJobsHistoryLimit | 1 |
|
||||
|
||||
### Adjusting Cleanup
|
||||
|
||||
Edit cleanup CronJob:
|
||||
```bash
|
||||
kubectl edit cronjob cleanup-old-build-jobs -n flux-builds
|
||||
```
|
||||
|
||||
Change:
|
||||
- **Schedule**: Modify `schedule: "*/30 * * * *"` (every 30 min)
|
||||
- **Keep N builds**: Change `head -n -10` to keep more/fewer
|
||||
- **Failed job age**: Change `1 hour ago` to different duration
|
||||
- **Max age**: Change `24 hours ago` to different duration
|
||||
|
||||
### Disable Cleanup Temporarily
|
||||
|
||||
```bash
|
||||
# Suspend cleanup CronJob
|
||||
kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":true}}'
|
||||
|
||||
# Resume
|
||||
kubectl patch cronjob cleanup-old-build-jobs -n flux-builds -p '{"spec":{"suspend":false}}'
|
||||
```
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### Before Migration
|
||||
- All resources in `default` namespace
|
||||
- Jobs accumulating indefinitely
|
||||
- Manual cleanup required
|
||||
|
||||
### After Migration
|
||||
- Clean namespace separation
|
||||
- Automatic cleanup every 30 minutes
|
||||
- Build history maintained (last 10 successful)
|
||||
- Reduced cluster resource usage
|
||||
- Improved visibility and monitoring
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# One-liner status check
|
||||
kubectl get helmrelease,deployment,pods,jobs,cronjob --all-namespaces | grep neon
|
||||
|
||||
# Check all namespaces
|
||||
kubectl get ns | grep -E "neonvortex|flux-builds|flux-system"
|
||||
|
||||
# Total resource usage
|
||||
kubectl top pods -n neonvortex
|
||||
kubectl top pods -n flux-builds
|
||||
|
||||
# Events
|
||||
kubectl get events -n neonvortex --sort-by='.lastTimestamp' | tail -20
|
||||
kubectl get events -n flux-builds --sort-by='.lastTimestamp' | tail -20
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
Consider these enhancements:
|
||||
|
||||
1. **Network Policies**: Isolate namespace traffic
|
||||
2. **Resource Quotas**: Limit namespace resource usage
|
||||
3. **Pod Security**: Add PodSecurityAdmissions
|
||||
4. **Monitoring**: Add Prometheus metrics for build success rate
|
||||
5. **Alerting**: Integrate with AlertManager for failures
|
||||
6. **Multi-tenancy**: If deploying more apps, follow this pattern
|
||||
|
||||
## Files Updated
|
||||
|
||||
New files created:
|
||||
- `namespaces.yaml` - Namespace definitions
|
||||
- `flux-helmrelease-neonvortex.yaml` - Updated HelmRelease
|
||||
- `webhook-build-trigger-neonvortex.yaml` - Webhook in new namespace
|
||||
- `build-trigger-cronjob-neonvortex.yaml` - Backup CronJob
|
||||
- `build-job-cleanup.yaml` - Automated cleanup
|
||||
- `flux-alerts-neonvortex.yaml` - Updated alerts
|
||||
|
||||
Old files (can be removed):
|
||||
- `flux-helmrelease.yaml` (replaced)
|
||||
- `webhook-build-trigger.yaml` (replaced)
|
||||
- `build-trigger-cronjob.yaml` (replaced)
|
||||
- `flux-alerts.yaml` (replaced)
|
||||
25
neon-vortex-ingress-neonvortex.yaml
Normal file
25
neon-vortex-ingress-neonvortex.yaml
Normal file
@@ -0,0 +1,25 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: neon-vortex
|
||||
namespace: neonvortex
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
nginx.ingress.kubernetes.io/ssl-redirect: "true"
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
rules:
|
||||
- host: nv.caffeinetux.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: neon-vortex
|
||||
port:
|
||||
number: 80
|
||||
tls:
|
||||
- hosts:
|
||||
- nv.caffeinetux.com
|
||||
secretName: neon-vortex-tls
|
||||
Reference in New Issue
Block a user