neon-vortex/WEBHOOK_SETUP_GUIDE.md

Neon Vortex Webhook CI/CD Setup Guide

This guide explains how to configure Gitea webhooks to automatically build and deploy Neon Vortex on every push, with notifications sent to notify.caffeinetux.com.

Overview

The pipeline now works as follows:

1. Push to Gitea → Triggers webhook
2. Flux Receiver → Detects push and reconciles GitRepository
3. Webhook Listener → Triggers Kaniko build job
4. Kaniko → Builds and pushes image to Harbor
5. Flux HelmRelease → Auto-deploys latest image
6. Notifications → Sent to notify.caffeinetux.com at each stage

Components Deployed

Flux Components

• Receiver: neon-vortex-receiver (flux-system namespace)
  • Webhook path: /hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
  • NodePort: 30090
• Provider: neon-vortex-notify (notifications endpoint)
• Alerts: Git updates, build status, deployment status

Build Trigger

• Deployment: webhook-build-listener (default namespace)
  • NodePort: 30091
  • Automatically creates build jobs on webhook trigger

Gitea Webhook Configuration

You need to configure TWO webhooks in Gitea:

Webhook 1: Flux Receiver (for Git sync)

1. Go to: http://192.168.1.49:13001/admin/neon-vortex/settings/hooks
2. Click Add Webhook → Gitea
3. Configure:
   • Target URL: http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
   • HTTP Method: POST
   • POST Content Type: application/json
   • Secret: neon-vortex-webhook-secret-2024
   • Trigger On: ✅ Push events
   • Branch filter: main
   • Active: ✅ Enabled
4. Click Add Webhook

Webhook 2: Build Trigger (for automatic builds)

1. Go to: http://192.168.1.49:13001/admin/neon-vortex/settings/hooks
2. Click Add Webhook → Gitea
3. Configure:
   • Target URL: http://<NODE_IP>:30091/webhook
   • HTTP Method: POST
   • POST Content Type: application/json
   • Secret: (leave empty)
   • Trigger On: ✅ Push events
   • Branch filter: main
   • Active: ✅ Enabled
4. Click Add Webhook

Finding Your Node IP

# Get node IP
kubectl get nodes -o wide

# Test Flux webhook endpoint
curl -X POST http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a

# Test build trigger endpoint
curl -X POST http://<NODE_IP>:30091/webhook

Notification Configuration

Notifications are sent to https://notify.caffeinetux.com using token: APMvTuncQJmm6vd

You'll receive notifications for:

• ✅ Build Started: When a new build job is created
• ✅ Build Complete: When the image is successfully built
• ✅ Deployment Events: When Flux updates the application
• ❌ Failures: If any step fails

How It Works

On Git Push:

1. Gitea sends webhook to both endpoints
2. Flux Receiver reconciles the GitRepository immediately
3. Build Listener receives webhook and triggers build script
4. Build Script:
   • Clones repository
   • Extracts commit info (SHA, message, author)
   • Sends "Build Started" notification
   • Creates Kaniko build job with unique name
5. Kaniko Job:
   • Clones and checks out specific commit
   • Builds Docker image
   • Pushes to Harbor with tags:
     • latest (used for deployment)
     • <commit-short-sha> (for versioning)
   • Adds git metadata as image labels
6. Notification Container:
   • Sends "Build Complete" notification
7. Flux HelmRelease:
   • Detects new latest image
   • Performs rolling update
   • Sends deployment notification

Monitoring

Check Webhook Status

# View Flux receiver status
kubectl get receiver neon-vortex-receiver -n flux-system

# Check build listener
kubectl get deployment webhook-build-listener -n default
kubectl logs -n default deployment/webhook-build-listener

Monitor Build Jobs

# List recent build jobs
kubectl get jobs -n default -l build-trigger=webhook

# Watch for new builds
kubectl get jobs -n default -w

# View specific build logs
kubectl logs -n default job/<job-name> -c kaniko -f

Check Notifications

# View notification provider status
kubectl get provider neon-vortex-notify -n flux-system

# Check alert status
kubectl get alerts -n flux-system

View Deployment Status

# Check HelmRelease
flux get helmrelease neon-vortex -n default

# Check pods
kubectl get pods -n default -l app.kubernetes.io/name=neon-vortex

# Check current image
kubectl get deployment neon-vortex -n default -o jsonpath='{.spec.template.spec.containers[0].image}'

Testing the Pipeline

Test Flux Webhook Receiver

# Trigger Flux to reconcile
curl -X POST http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a

# Check if GitRepository reconciled
flux get sources git -n flux-system

Test Build Trigger

# Manually trigger a build
curl -X POST http://<NODE_IP>:30091/webhook

# Check for new build job
kubectl get jobs -n default -l build-trigger=webhook

Full End-to-End Test

1. Make a small change to your code
2. Commit and push:

   echo "# Test webhook" >> README.md
   git add README.md
   git commit -m "Test webhook pipeline"
   git push origin main
   

3. Watch the pipeline:

   # Watch jobs being created
   watch kubectl get jobs -n default
   
   # Check notifications (should receive on your device)
   # Monitor deployment
   watch kubectl get helmrelease,deployment,pods -n default
   

Troubleshooting

Webhooks Not Triggering

1. Check webhook delivery in Gitea:

   • Go to repository → Settings → Webhooks
   • Click on the webhook
   • Scroll down to "Recent Deliveries"
   • Check response codes and errors

2. Verify services are accessible:

   # Test from within cluster
   kubectl run -it --rm debug --image=alpine --restart=Never -- sh
   apk add curl
   curl -v http://flux-receiver.flux-system.svc.cluster.local/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
   curl -v http://webhook-build-listener.default.svc.cluster.local:8080/webhook
   

3. Check logs:

   # Flux notification controller
   kubectl logs -n flux-system deployment/notification-controller
   
   # Build listener
   kubectl logs -n default deployment/webhook-build-listener
   

Builds Not Starting

1. Check build listener is running:

   kubectl get pods -n default -l app=webhook-build-listener
   kubectl logs -n default -l app=webhook-build-listener
   

2. Verify RBAC permissions:

   kubectl get role build-trigger-role -n default
   kubectl get rolebinding build-trigger-binding -n default
   

3. Check Harbor credentials:

   kubectl get secret harbor-registry -n default
   

Notifications Not Received

1. Test notification endpoint:

   curl -X POST "https://notify.caffeinetux.com?token=APMvTuncQJmm6vd" \
     -H "Content-Type: application/json" \
     -d '{"title":"Test","message":"Testing notifications","priority":3}'
   

2. Check provider status:

   kubectl get provider neon-vortex-notify -n flux-system -o yaml
   kubectl describe alert -n flux-system
   

3. View notification controller logs:

   kubectl logs -n flux-system deployment/notification-controller | grep notify
   

Build Failures

1. Check job status:

   kubectl get job <job-name> -n default
   kubectl describe job <job-name> -n default
   

2. View build logs:

   # Git clone logs
   kubectl logs job/<job-name> -n default -c git-clone
   
   # Kaniko build logs
   kubectl logs job/<job-name> -n default -c kaniko
   
   # Notification logs
   kubectl logs job/<job-name> -n default -c notify-completion
   

Cleanup Old Jobs

Build jobs are auto-deleted after 1 hour (ttlSecondsAfterFinished: 3600).

Manual cleanup:

# Delete completed jobs
kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.successful=1

# Delete failed jobs
kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.failed=1

Disabling Auto-Build

To temporarily disable automatic builds:

# Scale down build listener
kubectl scale deployment webhook-build-listener -n default --replicas=0

# Or suspend the Flux receiver
flux suspend receiver neon-vortex-receiver -n flux-system

To re-enable:

# Scale up build listener
kubectl scale deployment webhook-build-listener -n default --replicas=1

# Or resume Flux receiver
flux resume receiver neon-vortex-receiver -n flux-system

Security Notes

1. Webhook Secret: The Flux receiver uses token neon-vortex-webhook-secret-2024
2. Notification Token: Stored in secret notify-token in flux-system namespace
3. Harbor Credentials: Stored in secret harbor-registry in default namespace
4. RBAC: Build trigger has minimal permissions (only manage jobs)

Quick Reference

# Status check one-liner
kubectl get receiver,provider,alert -n flux-system && kubectl get deployment webhook-build-listener -n default && kubectl get jobs -n default -l build-trigger=webhook

# Force reconciliation
flux reconcile source git neon-vortex -n flux-system

# Manual build trigger
curl -X POST http://<NODE_IP>:30091/webhook

# Watch everything
watch 'kubectl get jobs,pods -n default | grep -E "(build-|neon-vortex)"'

Image Tags

Each build creates tags:

• latest - Always points to most recent build
• <short-sha> - Specific commit (e.g., e61dc3b)

Images also include labels:

• git.commit - Full commit SHA
• git.short - Short commit SHA
• git.message - Commit message
• git.author - Commit author

Next Steps

• Set up staging environment with different image tags
• Add automated testing before build
• Implement approval workflow for production
• Add Slack integration alongside notifications
• Set up Prometheus metrics for build success rate