neon-vortex/WEBHOOK_SETUP_GUIDE.md

# Neon Vortex Webhook CI/CD Setup Guide

This guide explains how to configure Gitea webhooks to automatically build and deploy Neon Vortex on every push, with notifications sent to notify.caffeinetux.com.

## Overview

The pipeline now works as follows:

1. **Push to Gitea** → Triggers webhook
2. **Flux Receiver** → Detects push and reconciles GitRepository
3. **Webhook Listener** → Triggers Kaniko build job
4. **Kaniko** → Builds and pushes image to Harbor
5. **Flux HelmRelease** → Auto-deploys latest image
6. **Notifications** → Sent to notify.caffeinetux.com at each stage

## Components Deployed

### Flux Components
- **Receiver**: `neon-vortex-receiver` (flux-system namespace)
  - Webhook path: `/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a`
  - NodePort: 30090
- **Provider**: `neon-vortex-notify` (notifications endpoint)
- **Alerts**: Git updates, build status, deployment status

### Build Trigger
- **Deployment**: `webhook-build-listener` (default namespace)
  - NodePort: 30091
  - Automatically creates build jobs on webhook trigger

## Gitea Webhook Configuration

You need to configure TWO webhooks in Gitea:

### Webhook 1: Flux Receiver (for Git sync)

1. Go to: `http://192.168.1.49:13001/admin/neon-vortex/settings/hooks`
2. Click **Add Webhook** → **Gitea**
3. Configure:
   - **Target URL**: `http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a`
   - **HTTP Method**: `POST`
   - **POST Content Type**: `application/json`
   - **Secret**: `neon-vortex-webhook-secret-2024`
   - **Trigger On**: ✅ Push events
   - **Branch filter**: `main`
   - **Active**: ✅ Enabled
4. Click **Add Webhook**

### Webhook 2: Build Trigger (for automatic builds)

1. Go to: `http://192.168.1.49:13001/admin/neon-vortex/settings/hooks`
2. Click **Add Webhook** → **Gitea**
3. Configure:
   - **Target URL**: `http://<NODE_IP>:30091/webhook`
   - **HTTP Method**: `POST`
   - **POST Content Type**: `application/json`
   - **Secret**: (leave empty)
   - **Trigger On**: ✅ Push events
   - **Branch filter**: `main`
   - **Active**: ✅ Enabled
4. Click **Add Webhook**

## Finding Your Node IP

```bash
# Get node IP
kubectl get nodes -o wide

# Test Flux webhook endpoint
curl -X POST http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a

# Test build trigger endpoint
curl -X POST http://<NODE_IP>:30091/webhook
```

## Notification Configuration

Notifications are sent to `https://notify.caffeinetux.com` using token: `APMvTuncQJmm6vd`

You'll receive notifications for:
- ✅ **Build Started**: When a new build job is created
- ✅ **Build Complete**: When the image is successfully built
- ✅ **Deployment Events**: When Flux updates the application
- ❌ **Failures**: If any step fails

## How It Works

### On Git Push:

1. **Gitea** sends webhook to both endpoints
2. **Flux Receiver** reconciles the GitRepository immediately
3. **Build Listener** receives webhook and triggers build script
4. **Build Script**:
   - Clones repository
   - Extracts commit info (SHA, message, author)
   - Sends "Build Started" notification
   - Creates Kaniko build job with unique name
5. **Kaniko Job**:
   - Clones and checks out specific commit
   - Builds Docker image
   - Pushes to Harbor with tags:
     - `latest` (used for deployment)
     - `<commit-short-sha>` (for versioning)
   - Adds git metadata as image labels
6. **Notification Container**:
   - Sends "Build Complete" notification
7. **Flux HelmRelease**:
   - Detects new `latest` image
   - Performs rolling update
   - Sends deployment notification

## Monitoring

### Check Webhook Status

```bash
# View Flux receiver status
kubectl get receiver neon-vortex-receiver -n flux-system

# Check build listener
kubectl get deployment webhook-build-listener -n default
kubectl logs -n default deployment/webhook-build-listener
```

### Monitor Build Jobs

```bash
# List recent build jobs
kubectl get jobs -n default -l build-trigger=webhook

# Watch for new builds
kubectl get jobs -n default -w

# View specific build logs
kubectl logs -n default job/<job-name> -c kaniko -f
```

### Check Notifications

```bash
# View notification provider status
kubectl get provider neon-vortex-notify -n flux-system

# Check alert status
kubectl get alerts -n flux-system
```

### View Deployment Status

```bash
# Check HelmRelease
flux get helmrelease neon-vortex -n default

# Check pods
kubectl get pods -n default -l app.kubernetes.io/name=neon-vortex

# Check current image
kubectl get deployment neon-vortex -n default -o jsonpath='{.spec.template.spec.containers[0].image}'
```

## Testing the Pipeline

### Test Flux Webhook Receiver

```bash
# Trigger Flux to reconcile
curl -X POST http://<NODE_IP>:30090/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a

# Check if GitRepository reconciled
flux get sources git -n flux-system
```

### Test Build Trigger

```bash
# Manually trigger a build
curl -X POST http://<NODE_IP>:30091/webhook

# Check for new build job
kubectl get jobs -n default -l build-trigger=webhook
```

### Full End-to-End Test

1. Make a small change to your code
2. Commit and push:
   ```bash
   echo "# Test webhook" >> README.md
   git add README.md
   git commit -m "Test webhook pipeline"
   git push origin main
   ```
3. Watch the pipeline:
   ```bash
   # Watch jobs being created
   watch kubectl get jobs -n default

   # Check notifications (should receive on your device)
   # Monitor deployment
   watch kubectl get helmrelease,deployment,pods -n default
   ```

## Troubleshooting

### Webhooks Not Triggering

1. **Check webhook delivery in Gitea**:
   - Go to repository → Settings → Webhooks
   - Click on the webhook
   - Scroll down to "Recent Deliveries"
   - Check response codes and errors

2. **Verify services are accessible**:
   ```bash
   # Test from within cluster
   kubectl run -it --rm debug --image=alpine --restart=Never -- sh
   apk add curl
   curl -v http://flux-receiver.flux-system.svc.cluster.local/hook/548969c2b24c717fe9e5af8c78ddfeec40d3024c270c7e85ac8f986259aeec9a
   curl -v http://webhook-build-listener.default.svc.cluster.local:8080/webhook
   ```

3. **Check logs**:
   ```bash
   # Flux notification controller
   kubectl logs -n flux-system deployment/notification-controller

   # Build listener
   kubectl logs -n default deployment/webhook-build-listener
   ```

### Builds Not Starting

1. **Check build listener is running**:
   ```bash
   kubectl get pods -n default -l app=webhook-build-listener
   kubectl logs -n default -l app=webhook-build-listener
   ```

2. **Verify RBAC permissions**:
   ```bash
   kubectl get role build-trigger-role -n default
   kubectl get rolebinding build-trigger-binding -n default
   ```

3. **Check Harbor credentials**:
   ```bash
   kubectl get secret harbor-registry -n default
   ```

### Notifications Not Received

1. **Test notification endpoint**:
   ```bash
   curl -X POST "https://notify.caffeinetux.com?token=APMvTuncQJmm6vd" \
     -H "Content-Type: application/json" \
     -d '{"title":"Test","message":"Testing notifications","priority":3}'
   ```

2. **Check provider status**:
   ```bash
   kubectl get provider neon-vortex-notify -n flux-system -o yaml
   kubectl describe alert -n flux-system
   ```

3. **View notification controller logs**:
   ```bash
   kubectl logs -n flux-system deployment/notification-controller | grep notify
   ```

### Build Failures

1. **Check job status**:
   ```bash
   kubectl get job <job-name> -n default
   kubectl describe job <job-name> -n default
   ```

2. **View build logs**:
   ```bash
   # Git clone logs
   kubectl logs job/<job-name> -n default -c git-clone

   # Kaniko build logs
   kubectl logs job/<job-name> -n default -c kaniko

   # Notification logs
   kubectl logs job/<job-name> -n default -c notify-completion
   ```

## Cleanup Old Jobs

Build jobs are auto-deleted after 1 hour (ttlSecondsAfterFinished: 3600).

Manual cleanup:
```bash
# Delete completed jobs
kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.successful=1

# Delete failed jobs
kubectl delete jobs -n default -l build-trigger=webhook --field-selector status.failed=1
```

## Disabling Auto-Build

To temporarily disable automatic builds:

```bash
# Scale down build listener
kubectl scale deployment webhook-build-listener -n default --replicas=0

# Or suspend the Flux receiver
flux suspend receiver neon-vortex-receiver -n flux-system
```

To re-enable:
```bash
# Scale up build listener
kubectl scale deployment webhook-build-listener -n default --replicas=1

# Or resume Flux receiver
flux resume receiver neon-vortex-receiver -n flux-system
```

## Security Notes

1. **Webhook Secret**: The Flux receiver uses token `neon-vortex-webhook-secret-2024`
2. **Notification Token**: Stored in secret `notify-token` in flux-system namespace
3. **Harbor Credentials**: Stored in secret `harbor-registry` in default namespace
4. **RBAC**: Build trigger has minimal permissions (only manage jobs)

## Quick Reference

```bash
# Status check one-liner
kubectl get receiver,provider,alert -n flux-system && kubectl get deployment webhook-build-listener -n default && kubectl get jobs -n default -l build-trigger=webhook

# Force reconciliation
flux reconcile source git neon-vortex -n flux-system

# Manual build trigger
curl -X POST http://<NODE_IP>:30091/webhook

# Watch everything
watch 'kubectl get jobs,pods -n default | grep -E "(build-|neon-vortex)"'
```

## Image Tags

Each build creates tags:
- `latest` - Always points to most recent build
- `<short-sha>` - Specific commit (e.g., `e61dc3b`)

Images also include labels:
- `git.commit` - Full commit SHA
- `git.short` - Short commit SHA
- `git.message` - Commit message
- `git.author` - Commit author

## Next Steps

- Set up staging environment with different image tags
- Add automated testing before build
- Implement approval workflow for production
- Add Slack integration alongside notifications
- Set up Prometheus metrics for build success rate