From 2c61efff726a922c8978ed92f7a4e8d60766eefb Mon Sep 17 00:00:00 2001 From: Neon Vortex Date: Sat, 22 Nov 2025 22:50:07 -0500 Subject: [PATCH] Add complete webhook-based CI/CD with automatic builds and notifications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit implements a full webhook-triggered CI/CD pipeline: **Flux Components:** - Flux Receiver for Gitea webhooks (generic type, NodePort 30090) - Notification Provider for notify.caffeinetux.com - Alerts for git updates, builds, and deployments **Build Automation:** - Webhook listener deployment that triggers on git push - Automatic Kaniko build jobs with git metadata - Images tagged with both 'latest' and commit SHA - Build notifications sent at start and completion **Workflow:** 1. Push to Gitea → Webhooks trigger Flux receiver & build listener 2. Build listener creates Kaniko job with commit info 3. Kaniko builds and pushes to Harbor (latest + SHA tags) 4. Flux auto-deploys latest image to cluster 5. Notifications sent to notify.caffeinetux.com at each stage **Configuration:** - Token: APMvTuncQJmm6vd - Webhook path: /hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a - Build trigger: http://:30091/webhook - Comprehensive setup documentation in WEBHOOK_SETUP_GUIDE.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- WEBHOOK_SETUP_GUIDE.md | 364 ++++++++++++++++++++++++++++++++ flux-alerts.yaml | 52 +++++ flux-build-kustomization.yaml | 55 +++++ flux-build-on-push.yaml | 197 +++++++++++++++++ flux-notification-provider.yaml | 20 ++ flux-webhook-receiver.yaml | 40 ++++ htlm/Neon Vortex.html | 5 +- htlm/Neon Vortex.js | 5 +- htlm/Neon Vortex.pck | Bin 256776 -> 256728 bytes htlm/Neon Vortex.wasm | Bin 1660816 -> 36160334 bytes webhook-build-trigger.yaml | 223 +++++++++++++++++++ 11 files changed, 954 insertions(+), 7 deletions(-) create mode 100644 WEBHOOK_SETUP_GUIDE.md create mode 100644 flux-alerts.yaml create mode 100644 flux-build-kustomization.yaml create mode 100644 flux-build-on-push.yaml create mode 100644 flux-notification-provider.yaml create mode 100644 flux-webhook-receiver.yaml create mode 100644 webhook-build-trigger.yaml diff --git a/WEBHOOK_SETUP_GUIDE.md b/WEBHOOK_SETUP_GUIDE.md new file mode 100644 index 0000000..1f566fb --- /dev/null +++ b/WEBHOOK_SETUP_GUIDE.md @@ -0,0 +1,364 @@ +# Neon Vortex Webhook CI/CD Setup Guide + +This guide explains how to configure Gitea webhooks to automatically build and deploy Neon Vortex on every push, with notifications sent to notify.caffeinetux.com. + +## Overview + +The pipeline now works as follows: + +1. **Push to Gitea** → Triggers webhook +2. **Flux Receiver** → Detects push and reconciles GitRepository +3. **Webhook Listener** → Triggers Kaniko build job +4. **Kaniko** → Builds and pushes image to Harbor +5. **Flux HelmRelease** → Auto-deploys latest image +6. **Notifications** → Sent to notify.caffeinetux.com at each stage + +## Components Deployed + +### Flux Components +- **Receiver**: `neon-vortex-receiver` (flux-system namespace) + - Webhook path: `/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a` + - NodePort: 30090 +- **Provider**: `neon-vortex-notify` (notifications endpoint) +- **Alerts**: Git updates, build status, deployment status + +### Build Trigger +- **Deployment**: `webhook-build-listener` (default namespace) + - NodePort: 30091 + - Automatically creates build jobs on webhook trigger + +## Gitea Webhook Configuration + +You need to configure TWO webhooks in Gitea: + +### Webhook 1: Flux Receiver (for Git sync) + +1. Go to: `http://192.168.1.49:13001/admin/neon-vortex/settings/hooks` +2. Click **Add Webhook** → **Gitea** +3. Configure: + - **Target URL**: `http://:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a` + - **HTTP Method**: `POST` + - **POST Content Type**: `application/json` + - **Secret**: `neon-vortex-webhook-secret-2024` + - **Trigger On**: ✅ Push events + - **Branch filter**: `main` + - **Active**: ✅ Enabled +4. Click **Add Webhook** + +### Webhook 2: Build Trigger (for automatic builds) + +1. Go to: `http://192.168.1.49:13001/admin/neon-vortex/settings/hooks` +2. Click **Add Webhook** → **Gitea** +3. Configure: + - **Target URL**: `http://:30091/webhook` + - **HTTP Method**: `POST` + - **POST Content Type**: `application/json` + - **Secret**: (leave empty) + - **Trigger On**: ✅ Push events + - **Branch filter**: `main` + - **Active**: ✅ Enabled +4. Click **Add Webhook** + +## Finding Your Node IP + +```bash +# Get node IP +kubectl get nodes -o wide + +# Test Flux webhook endpoint +curl -X POST http://:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a + +# Test build trigger endpoint +curl -X POST http://:30091/webhook +``` + +## Notification Configuration + +Notifications are sent to `https://notify.caffeinetux.com` using token: `APMvTuncQJmm6vd` + +You'll receive notifications for: +- ✅ **Build Started**: When a new build job is created +- ✅ **Build Complete**: When the image is successfully built +- ✅ **Deployment Events**: When Flux updates the application +- ❌ **Failures**: If any step fails + +## How It Works + +### On Git Push: + +1. **Gitea** sends webhook to both endpoints +2. **Flux Receiver** reconciles the GitRepository immediately +3. **Build Listener** receives webhook and triggers build script +4. **Build Script**: + - Clones repository + - Extracts commit info (SHA, message, author) + - Sends "Build Started" notification + - Creates Kaniko build job with unique name +5. **Kaniko Job**: + - Clones and checks out specific commit + - Builds Docker image + - Pushes to Harbor with tags: + - `latest` (used for deployment) + - `` (for versioning) + - Adds git metadata as image labels +6. **Notification Container**: + - Sends "Build Complete" notification +7. **Flux HelmRelease**: + - Detects new `latest` image + - Performs rolling update + - Sends deployment notification + +## Monitoring + +### Check Webhook Status + +```bash +# View Flux receiver status +kubectl get receiver neon-vortex-receiver -n flux-system + +# Check build listener +kubectl get deployment webhook-build-listener -n default +kubectl logs -n default deployment/webhook-build-listener +``` + +### Monitor Build Jobs + +```bash +# List recent build jobs +kubectl get jobs -n default -l build-trigger=webhook + +# Watch for new builds +kubectl get jobs -n default -w + +# View specific build logs +kubectl logs -n default job/ -c kaniko -f +``` + +### Check Notifications + +```bash +# View notification provider status +kubectl get provider neon-vortex-notify -n flux-system + +# Check alert status +kubectl get alerts -n flux-system +``` + +### View Deployment Status + +```bash +# Check HelmRelease +flux get helmrelease neon-vortex -n default + +# Check pods +kubectl get pods -n default -l app.kubernetes.io/name=neon-vortex + +# Check current image +kubectl get deployment neon-vortex -n default -o jsonpath='{.spec.template.spec.containers[0].image}' +``` + +## Testing the Pipeline + +### Test Flux Webhook Receiver + +```bash +# Trigger Flux to reconcile +curl -X POST http://:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a + +# Check if GitRepository reconciled +flux get sources git -n flux-system +``` + +### Test Build Trigger + +```bash +# Manually trigger a build +curl -X POST http://:30091/webhook + +# Check for new build job +kubectl get jobs -n default -l build-trigger=webhook +``` + +### Full End-to-End Test + +1. Make a small change to your code +2. Commit and push: + ```bash + echo "# Test webhook" >> README.md + git add README.md + git commit -m "Test webhook pipeline" + git push origin main + ``` +3. Watch the pipeline: + ```bash + # Watch jobs being created + watch kubectl get jobs -n default + + # Check notifications (should receive on your device) + # Monitor deployment + watch kubectl get helmrelease,deployment,pods -n default + ``` + +## Troubleshooting + +### Webhooks Not Triggering + +1. **Check webhook delivery in Gitea**: + - Go to repository → Settings → Webhooks + - Click on the webhook + - Scroll down to "Recent Deliveries" + - Check response codes and errors + +2. **Verify services are accessible**: + ```bash + # Test from within cluster + kubectl run -it --rm debug --image=alpine --restart=Never -- sh + apk add curl + curl -v http://flux-receiver.flux-system.svc.cluster.local/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a + curl -v http://webhook-build-listener.default.svc.cluster.local:8080/webhook + ``` + +3. **Check logs**: + ```bash + # Flux notification controller + kubectl logs -n flux-system deployment/notification-controller + + # Build listener + kubectl logs -n default deployment/webhook-build-listener + ``` + +### Builds Not Starting + +1. **Check build listener is running**: + ```bash + kubectl get pods -n default -l app=webhook-build-listener + kubectl logs -n default -l app=webhook-build-listener + ``` + +2. **Verify RBAC permissions**: + ```bash + kubectl get role build-trigger-role -n default + kubectl get rolebinding build-trigger-binding -n default + ``` + +3. **Check Harbor credentials**: + ```bash + kubectl get secret harbor-registry -n default + ``` + +### Notifications Not Received + +1. **Test notification endpoint**: + ```bash + curl -X POST "https://notify.caffeinetux.com?token=APMvTuncQJmm6vd" \ + -H "Content-Type: application/json" \ + -d '{"title":"Test","message":"Testing notifications","priority":3}' + ``` + +2. **Check provider status**: + ```bash + kubectl get provider neon-vortex-notify -n flux-system -o yaml + kubectl describe alert -n flux-system + ``` + +3. **View notification controller logs**: + ```bash + kubectl logs -n flux-system deployment/notification-controller | grep notify + ``` + +### Build Failures + +1. **Check job status**: + ```bash + kubectl get job -n default + kubectl describe job -n default + ``` + +2. **View build logs**: + ```bash + # Git clone logs + kubectl logs job/ -n default -c git-clone + + # Kaniko build logs + kubectl logs job/ -n default -c kaniko + + # Notification logs + kubectl logs job/ -n default -c notify-completion + ``` + +## Cleanup Old Jobs + +Build jobs are auto-deleted after 1 hour (ttlSecondsAfterFinished: 3600). + +Manual cleanup: +```bash +# Delete completed jobs +kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.successful=1 + +# Delete failed jobs +kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.failed=1 +``` + +## Disabling Auto-Build + +To temporarily disable automatic builds: + +```bash +# Scale down build listener +kubectl scale deployment webhook-build-listener -n default --replicas=0 + +# Or suspend the Flux receiver +flux suspend receiver neon-vortex-receiver -n flux-system +``` + +To re-enable: +```bash +# Scale up build listener +kubectl scale deployment webhook-build-listener -n default --replicas=1 + +# Or resume Flux receiver +flux resume receiver neon-vortex-receiver -n flux-system +``` + +## Security Notes + +1. **Webhook Secret**: The Flux receiver uses token `neon-vortex-webhook-secret-2024` +2. **Notification Token**: Stored in secret `notify-token` in flux-system namespace +3. **Harbor Credentials**: Stored in secret `harbor-registry` in default namespace +4. **RBAC**: Build trigger has minimal permissions (only manage jobs) + +## Quick Reference + +```bash +# Status check one-liner +kubectl get receiver,provider,alert -n flux-system && kubectl get deployment webhook-build-listener -n default && kubectl get jobs -n default -l build-trigger=webhook + +# Force reconciliation +flux reconcile source git neon-vortex -n flux-system + +# Manual build trigger +curl -X POST http://:30091/webhook + +# Watch everything +watch 'kubectl get jobs,pods -n default | grep -E "(build-|neon-vortex)"' +``` + +## Image Tags + +Each build creates tags: +- `latest` - Always points to most recent build +- `` - Specific commit (e.g., `e61dc3b`) + +Images also include labels: +- `git.commit` - Full commit SHA +- `git.short` - Short commit SHA +- `git.message` - Commit message +- `git.author` - Commit author + +## Next Steps + +- Set up staging environment with different image tags +- Add automated testing before build +- Implement approval workflow for production +- Add Slack integration alongside notifications +- Set up Prometheus metrics for build success rate diff --git a/flux-alerts.yaml b/flux-alerts.yaml new file mode 100644 index 0000000..74d1e57 --- /dev/null +++ b/flux-alerts.yaml @@ -0,0 +1,52 @@ +--- +apiVersion: notification.toolkit.fluxcd.io/v1beta3 +kind: Alert +metadata: + name: neon-vortex-git + namespace: flux-system +spec: + summary: "Neon Vortex Git Updates" + providerRef: + name: neon-vortex-notify + eventSeverity: info + eventSources: + - kind: GitRepository + name: neon-vortex + namespace: flux-system + exclusionList: + - ".*health check.*" +--- +apiVersion: notification.toolkit.fluxcd.io/v1beta3 +kind: Alert +metadata: + name: neon-vortex-build + namespace: flux-system +spec: + summary: "Neon Vortex Build Status" + providerRef: + name: neon-vortex-notify + eventSeverity: info + eventSources: + - kind: Kustomization + name: neon-vortex-build + namespace: flux-system + exclusionList: + - ".*health check.*" +--- +apiVersion: notification.toolkit.fluxcd.io/v1beta3 +kind: Alert +metadata: + name: neon-vortex-deploy + namespace: flux-system +spec: + summary: "Neon Vortex Deployment Status" + providerRef: + name: neon-vortex-notify + eventSeverity: info + eventSources: + - kind: HelmRelease + name: neon-vortex + namespace: default + exclusionList: + - ".*health check.*" + - ".*reconciliation in progress.*" diff --git a/flux-build-kustomization.yaml b/flux-build-kustomization.yaml new file mode 100644 index 0000000..d651ec0 --- /dev/null +++ b/flux-build-kustomization.yaml @@ -0,0 +1,55 @@ +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: neon-vortex-build + namespace: flux-system +spec: + interval: 1m + path: ./ + prune: false + sourceRef: + kind: GitRepository + name: neon-vortex + namespace: flux-system + targetNamespace: default + patches: + - patch: | + apiVersion: batch/v1 + kind: Job + metadata: + name: neon-vortex-build + namespace: default + spec: + template: + metadata: + annotations: + git-commit: "${GIT_COMMIT:-unknown}" + spec: + initContainers: + - name: git-clone + env: + - name: GIT_COMMIT + value: "${GIT_COMMIT:-main}" + containers: + - name: kaniko + args: + - "--dockerfile=/workspace/htlm/Dockerfile" + - "--context=/workspace/htlm" + - "--destination=images.caffeinetux.com/apps/neon-vortex:latest" + - "--destination=images.caffeinetux.com/apps/neon-vortex:${GIT_COMMIT:-latest}" + - "--cache=true" + - "--cache-repo=images.caffeinetux.com/apps/neon-vortex/cache" + target: + kind: Job + name: neon-vortex-build + postBuild: + substituteFrom: + - kind: ConfigMap + name: git-commit-info + optional: true + healthChecks: + - apiVersion: batch/v1 + kind: Job + name: neon-vortex-build + namespace: default diff --git a/flux-build-on-push.yaml b/flux-build-on-push.yaml new file mode 100644 index 0000000..de95eb7 --- /dev/null +++ b/flux-build-on-push.yaml @@ -0,0 +1,197 @@ +--- +# ConfigMap with build script that extracts git commit info +apiVersion: v1 +kind: ConfigMap +metadata: + name: build-on-push-script + namespace: default +data: + build.sh: | + #!/bin/bash + set -e + + echo "===== Neon Vortex Build Triggered by Git Push =====" + + # Get latest commit info from git + cd /workspace + GIT_COMMIT=$(git rev-parse HEAD) + GIT_SHORT_COMMIT=$(git rev-parse --short HEAD) + GIT_MESSAGE=$(git log -1 --pretty=%B) + + echo "Commit: $GIT_COMMIT" + echo "Short: $GIT_SHORT_COMMIT" + echo "Message: $GIT_MESSAGE" + + # Create unique job name + TIMESTAMP=$(date +%s) + JOB_NAME="neon-vortex-build-${GIT_SHORT_COMMIT}-${TIMESTAMP}" + + echo "Creating build job: $JOB_NAME" + + # Create the build job + cat < - @@ -112,8 +111,8 @@ body {