Add complete webhook-based CI/CD with automatic builds and notifications

This commit implements a full webhook-triggered CI/CD pipeline: **Flux Components:** - Flux Receiver for Gitea webhooks (generic type, NodePort 30090) - Notification Provider for notify.caffeinetux.com - Alerts for git updates, builds, and deployments **Build Automation:** - Webhook listener deployment that triggers on git push - Automatic Kaniko build jobs with git metadata - Images tagged with both 'latest' and commit SHA - Build notifications sent at start and completion **Workflow:** 1. Push to Gitea → Webhooks trigger Flux receiver & build listener 2. Build listener creates Kaniko job with commit info 3. Kaniko builds and pushes to Harbor (latest + SHA tags) 4. Flux auto-deploys latest image to cluster 5. Notifications sent to notify.caffeinetux.com at each stage **Configuration:** - Token: APMvTuncQJmm6vd - Webhook path: /hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a - Build trigger: http://<node-ip>:30091/webhook - Comprehensive setup documentation in WEBHOOK_SETUP_GUIDE.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-22 22:50:07 -05:00
parent e61dc3bd58
commit 2c61efff72
11 changed files with 954 additions and 7 deletions
--- a/WEBHOOK_SETUP_GUIDE.md
+++ b/WEBHOOK_SETUP_GUIDE.md
@@ -0,0 +1,364 @@
+# Neon Vortex Webhook CI/CD Setup Guide
+
+This guide explains how to configure Gitea webhooks to automatically build and deploy Neon Vortex on every push, with notifications sent to notify.caffeinetux.com.
+
+## Overview
+
+The pipeline now works as follows:
+
+1. **Push to Gitea** → Triggers webhook
+2. **Flux Receiver** → Detects push and reconciles GitRepository
+3. **Webhook Listener** → Triggers Kaniko build job
+4. **Kaniko** → Builds and pushes image to Harbor
+5. **Flux HelmRelease** → Auto-deploys latest image
+6. **Notifications** → Sent to notify.caffeinetux.com at each stage
+
+## Components Deployed
+
+### Flux Components
+- **Receiver**: `neon-vortex-receiver` (flux-system namespace)
+  - Webhook path: `/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a`
+  - NodePort: 30090
+- **Provider**: `neon-vortex-notify` (notifications endpoint)
+- **Alerts**: Git updates, build status, deployment status
+
+### Build Trigger
+- **Deployment**: `webhook-build-listener` (default namespace)
+  - NodePort: 30091
+  - Automatically creates build jobs on webhook trigger
+
+## Gitea Webhook Configuration
+
+You need to configure TWO webhooks in Gitea:
+
+### Webhook 1: Flux Receiver (for Git sync)
+
+1. Go to: `http://192.168.1.49:13001/admin/neon-vortex/settings/hooks`
+2. Click **Add Webhook** → **Gitea**
+3. Configure:
+   - **Target URL**: `http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a`
+   - **HTTP Method**: `POST`
+   - **POST Content Type**: `application/json`
+   - **Secret**: `neon-vortex-webhook-secret-2024`
+   - **Trigger On**: ✅ Push events
+   - **Branch filter**: `main`
+   - **Active**: ✅ Enabled
+4. Click **Add Webhook**
+
+### Webhook 2: Build Trigger (for automatic builds)
+
+1. Go to: `http://192.168.1.49:13001/admin/neon-vortex/settings/hooks`
+2. Click **Add Webhook** → **Gitea**
+3. Configure:
+   - **Target URL**: `http://<NODE_IP>:30091/webhook`
+   - **HTTP Method**: `POST`
+   - **POST Content Type**: `application/json`
+   - **Secret**: (leave empty)
+   - **Trigger On**: ✅ Push events
+   - **Branch filter**: `main`
+   - **Active**: ✅ Enabled
+4. Click **Add Webhook**
+
+## Finding Your Node IP
+
+```bash
+# Get node IP
+kubectl get nodes -o wide
+
+# Test Flux webhook endpoint
+curl -X POST http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
+
+# Test build trigger endpoint
+curl -X POST http://<NODE_IP>:30091/webhook
+```
+
+## Notification Configuration
+
+Notifications are sent to `https://notify.caffeinetux.com` using token: `APMvTuncQJmm6vd`
+
+You'll receive notifications for:
+- ✅ **Build Started**: When a new build job is created
+- ✅ **Build Complete**: When the image is successfully built
+- ✅ **Deployment Events**: When Flux updates the application
+- ❌ **Failures**: If any step fails
+
+## How It Works
+
+### On Git Push:
+
+1. **Gitea** sends webhook to both endpoints
+2. **Flux Receiver** reconciles the GitRepository immediately
+3. **Build Listener** receives webhook and triggers build script
+4. **Build Script**:
+   - Clones repository
+   - Extracts commit info (SHA, message, author)
+   - Sends "Build Started" notification
+   - Creates Kaniko build job with unique name
+5. **Kaniko Job**:
+   - Clones and checks out specific commit
+   - Builds Docker image
+   - Pushes to Harbor with tags:
+     - `latest` (used for deployment)
+     - `<commit-short-sha>` (for versioning)
+   - Adds git metadata as image labels
+6. **Notification Container**:
+   - Sends "Build Complete" notification
+7. **Flux HelmRelease**:
+   - Detects new `latest` image
+   - Performs rolling update
+   - Sends deployment notification
+
+## Monitoring
+
+### Check Webhook Status
+
+```bash
+# View Flux receiver status
+kubectl get receiver neon-vortex-receiver -n flux-system
+
+# Check build listener
+kubectl get deployment webhook-build-listener -n default
+kubectl logs -n default deployment/webhook-build-listener
+```
+
+### Monitor Build Jobs
+
+```bash
+# List recent build jobs
+kubectl get jobs -n default -l build-trigger=webhook
+
+# Watch for new builds
+kubectl get jobs -n default -w
+
+# View specific build logs
+kubectl logs -n default job/<job-name> -c kaniko -f
+```
+
+### Check Notifications
+
+```bash
+# View notification provider status
+kubectl get provider neon-vortex-notify -n flux-system
+
+# Check alert status
+kubectl get alerts -n flux-system
+```
+
+### View Deployment Status
+
+```bash
+# Check HelmRelease
+flux get helmrelease neon-vortex -n default
+
+# Check pods
+kubectl get pods -n default -l app.kubernetes.io/name=neon-vortex
+
+# Check current image
+kubectl get deployment neon-vortex -n default -o jsonpath='{.spec.template.spec.containers[0].image}'
+```
+
+## Testing the Pipeline
+
+### Test Flux Webhook Receiver
+
+```bash
+# Trigger Flux to reconcile
+curl -X POST http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
+
+# Check if GitRepository reconciled
+flux get sources git -n flux-system
+```
+
+### Test Build Trigger
+
+```bash
+# Manually trigger a build
+curl -X POST http://<NODE_IP>:30091/webhook
+
+# Check for new build job
+kubectl get jobs -n default -l build-trigger=webhook
+```
+
+### Full End-to-End Test
+
+1. Make a small change to your code
+2. Commit and push:
+   ```bash
+   echo "# Test webhook" >> README.md
+   git add README.md
+   git commit -m "Test webhook pipeline"
+   git push origin main
+   ```
+3. Watch the pipeline:
+   ```bash
+   # Watch jobs being created
+   watch kubectl get jobs -n default
+
+   # Check notifications (should receive on your device)
+   # Monitor deployment
+   watch kubectl get helmrelease,deployment,pods -n default
+   ```
+
+## Troubleshooting
+
+### Webhooks Not Triggering
+
+1. **Check webhook delivery in Gitea**:
+   - Go to repository → Settings → Webhooks
+   - Click on the webhook
+   - Scroll down to "Recent Deliveries"
+   - Check response codes and errors
+
+2. **Verify services are accessible**:
+   ```bash
+   # Test from within cluster
+   kubectl run -it --rm debug --image=alpine --restart=Never -- sh
+   apk add curl
+   curl -v http://flux-receiver.flux-system.svc.cluster.local/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
+   curl -v http://webhook-build-listener.default.svc.cluster.local:8080/webhook
+   ```
+
+3. **Check logs**:
+   ```bash
+   # Flux notification controller
+   kubectl logs -n flux-system deployment/notification-controller
+
+   # Build listener
+   kubectl logs -n default deployment/webhook-build-listener
+   ```
+
+### Builds Not Starting
+
+1. **Check build listener is running**:
+   ```bash
+   kubectl get pods -n default -l app=webhook-build-listener
+   kubectl logs -n default -l app=webhook-build-listener
+   ```
+
+2. **Verify RBAC permissions**:
+   ```bash
+   kubectl get role build-trigger-role -n default
+   kubectl get rolebinding build-trigger-binding -n default
+   ```
+
+3. **Check Harbor credentials**:
+   ```bash
+   kubectl get secret harbor-registry -n default
+   ```
+
+### Notifications Not Received
+
+1. **Test notification endpoint**:
+   ```bash
+   curl -X POST "https://notify.caffeinetux.com?token=APMvTuncQJmm6vd" \
+     -H "Content-Type: application/json" \
+     -d '{"title":"Test","message":"Testing notifications","priority":3}'
+   ```
+
+2. **Check provider status**:
+   ```bash
+   kubectl get provider neon-vortex-notify -n flux-system -o yaml
+   kubectl describe alert -n flux-system
+   ```
+
+3. **View notification controller logs**:
+   ```bash
+   kubectl logs -n flux-system deployment/notification-controller | grep notify
+   ```
+
+### Build Failures
+
+1. **Check job status**:
+   ```bash
+   kubectl get job <job-name> -n default
+   kubectl describe job <job-name> -n default
+   ```
+
+2. **View build logs**:
+   ```bash
+   # Git clone logs
+   kubectl logs job/<job-name> -n default -c git-clone
+
+   # Kaniko build logs
+   kubectl logs job/<job-name> -n default -c kaniko
+
+   # Notification logs
+   kubectl logs job/<job-name> -n default -c notify-completion
+   ```
+
+## Cleanup Old Jobs
+
+Build jobs are auto-deleted after 1 hour (ttlSecondsAfterFinished: 3600).
+
+Manual cleanup:
+```bash
+# Delete completed jobs
+kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.successful=1
+
+# Delete failed jobs
+kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.failed=1
+```
+
+## Disabling Auto-Build
+
+To temporarily disable automatic builds:
+
+```bash
+# Scale down build listener
+kubectl scale deployment webhook-build-listener -n default --replicas=0
+
+# Or suspend the Flux receiver
+flux suspend receiver neon-vortex-receiver -n flux-system
+```
+
+To re-enable:
+```bash
+# Scale up build listener
+kubectl scale deployment webhook-build-listener -n default --replicas=1
+
+# Or resume Flux receiver
+flux resume receiver neon-vortex-receiver -n flux-system
+```
+
+## Security Notes
+
+1. **Webhook Secret**: The Flux receiver uses token `neon-vortex-webhook-secret-2024`
+2. **Notification Token**: Stored in secret `notify-token` in flux-system namespace
+3. **Harbor Credentials**: Stored in secret `harbor-registry` in default namespace
+4. **RBAC**: Build trigger has minimal permissions (only manage jobs)
+
+## Quick Reference
+
+```bash
+# Status check one-liner
+kubectl get receiver,provider,alert -n flux-system && kubectl get deployment webhook-build-listener -n default && kubectl get jobs -n default -l build-trigger=webhook
+
+# Force reconciliation
+flux reconcile source git neon-vortex -n flux-system
+
+# Manual build trigger
+curl -X POST http://<NODE_IP>:30091/webhook
+
+# Watch everything
+watch 'kubectl get jobs,pods -n default | grep -E "(build-|neon-vortex)"'
+```
+
+## Image Tags
+
+Each build creates tags:
+- `latest` - Always points to most recent build
+- `<short-sha>` - Specific commit (e.g., `e61dc3b`)
+
+Images also include labels:
+- `git.commit` - Full commit SHA
+- `git.short` - Short commit SHA
+- `git.message` - Commit message
+- `git.author` - Commit author
+
+## Next Steps
+
+- Set up staging environment with different image tags
+- Add automated testing before build
+- Implement approval workflow for production
+- Add Slack integration alongside notifications
+- Set up Prometheus metrics for build success rate