neon-vortex/SIMPLE_CICD_SETUP.md

Simple CI/CD Pipeline for Neon Vortex

This pipeline automatically builds and deploys your application when you push changes to Gitea.

How It Works

The pipeline uses a simple, reliable approach with Kubernetes CronJob:

1. Git Push → You push code to your Gitea repository
2. CronJob Poll → Every 2 minutes, a CronJob checks for new commits
3. Kaniko Build → When changes are detected, Kaniko builds a new container image
4. Harbor Registry → The new image is pushed to Harbor with tags:
   • latest - Always points to the most recent build
   • v1.0.<commit-sha> - Specific version tag (e.g., v1.0.473de53)
5. Flux CD → Automatically deploys the latest tag to your cluster

What Was Installed

The following resources were created in your cluster:

# Check the CronJob (runs every 2 minutes)
kubectl get cronjob neon-vortex-build-trigger -n default

# View recent build trigger jobs
kubectl get jobs -n default -l app=neon-vortex

# Check build logs
kubectl logs -n default -l job-name=<job-name> -c kaniko

Current Status

# Verify the CronJob is active
kubectl get cronjob -n default

# Check when the next build check will run
kubectl describe cronjob neon-vortex-build-trigger -n default | grep "Last Schedule"

# See if any builds are currently running
kubectl get jobs -n default

How to Trigger a Build

Automatic (Recommended)

Just push your code to Gitea:

git add .
git commit -m "Your changes"
git push origin main

Within 2 minutes, the CronJob will detect the change and start a build automatically.

Manual Trigger

To manually trigger a build immediately:

# Create a job from the CronJob
kubectl create job --from=cronjob/neon-vortex-build-trigger manual-build-$(date +%s) -n default

Monitoring Builds

Watch for New Builds

# Watch jobs being created
kubectl get jobs -n default -w

# Follow build logs (replace JOB_NAME)
kubectl logs -f -n default job/JOB_NAME -c kaniko

Check Build History

# List all build jobs
kubectl get jobs -n default -l app=neon-vortex

# Get details of a specific job
kubectl describe job <job-name> -n default

Check Deployed Version

# See current image in use
kubectl get deployment neon-vortex -n default -o jsonpath='{.spec.template.spec.containers[0].image}'

# Check HelmRelease status
flux get helmrelease neon-vortex -n default

Image Tagging Strategy

Each build creates two tags:

• latest: Always updated, used by Flux for deployment
• v1.0.<commit-sha>: Permanent version tag for rollbacks

Example:

images.caffeinetux.com/apps/neon-vortex:latest
images.caffeinetux.com/apps/neon-vortex:v1.0.473de53

Rolling Back

To roll back to a previous version:

# List available versions in Harbor
# (You'll need to check your Harbor web UI or use Docker registry API)

# Update the HelmRelease to use a specific version
flux suspend helmrelease neon-vortex -n default

kubectl patch helmrelease neon-vortex -n default --type merge -p '
{
  "spec": {
    "values": {
      "image": {
        "tag": "v1.0.67a6ae1"
      }
    }
  }
}'

flux resume helmrelease neon-vortex -n default
flux reconcile helmrelease neon-vortex -n default

Adjusting Build Frequency

The CronJob checks for changes every 2 minutes by default. To change this:

# Edit the CronJob
kubectl edit cronjob neon-vortex-build-trigger -n default

# Change the schedule line:
# */2 * * * * = every 2 minutes
# */5 * * * * = every 5 minutes
# */10 * * * * = every 10 minutes

Troubleshooting

Builds Not Starting

1. Check if CronJob is running:

kubectl get cronjob neon-vortex-build-trigger -n default

2. Check recent CronJob executions:

kubectl get jobs -n default -l app=neon-vortex

3. View trigger logs:

# Find the most recent trigger job
TRIGGER_JOB=$(kubectl get jobs -n default -l cronjob=neon-vortex-build-trigger --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}')

# View its logs
kubectl logs job/$TRIGGER_JOB -n default

Build Failures

1. Check the build job status:

kubectl get jobs -n default
kubectl describe job <failed-job-name> -n default

2. View build logs:

kubectl logs -n default job/<job-name> -c git-clone
kubectl logs -n default job/<job-name> -c kaniko

3. Common issues:
   • Harbor authentication: Check harbor-registry secret exists
   • Git access: Ensure Gitea is accessible from cluster
   • Dockerfile errors: Check Dockerfile syntax in htlm/Dockerfile

Harbor Authentication Issues

Verify the Harbor registry secret:

kubectl get secret harbor-registry -n default
kubectl get secret harbor-registry -n default -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Recreate if needed:

kubectl delete secret harbor-registry -n default

kubectl create secret docker-registry harbor-registry \
  --docker-server=images.caffeinetux.com \
  --docker-username=admin \
  --docker-password=YOUR_PASSWORD \
  -n default

Deployment Not Updating

1. Check if new image exists in Harbor
2. Verify HelmRelease is using pullPolicy: Always:

kubectl get helmrelease neon-vortex -n default -o yaml | grep -A5 image

3. Force Flux to reconcile:

flux reconcile helmrelease neon-vortex -n default

4. Force pods to restart and pull new image:

kubectl rollout restart deployment neon-vortex -n default

Cleanup Old Build Jobs

Jobs are automatically deleted after 1 hour (ttlSecondsAfterFinished: 3600). To manually clean up:

# Delete all completed jobs older than 1 hour
kubectl delete jobs -n default -l app=neon-vortex --field-selector status.successful=1

Security Notes

1. Git Access: The CronJob pulls from Gitea without authentication (public repo)
   • If your repo is private, add credentials to the git clone command
2. Harbor Credentials: Stored securely in Kubernetes secret harbor-registry
3. RBAC: The build trigger has minimal permissions (only create/manage jobs)

Next Steps

Consider these enhancements:

1. Add Tests: Run tests before building the image
2. Notifications: Send Slack/email alerts on build success/failure
3. Multi-stage Builds: Optimize image size
4. Image Scanning: Add Trivy or Clair for vulnerability scanning
5. Blue/Green Deployment: Implement canary releases
6. Prometheus Metrics: Monitor build success rates

Quick Reference

# Check everything is running
kubectl get cronjob,jobs,pods -n default -l app=neon-vortex

# Trigger immediate build
kubectl create job --from=cronjob/neon-vortex-build-trigger manual-$(date +%s) -n default

# Watch deployment status
watch kubectl get helmrelease,deployment,pods -n default

# View recent changes
flux get helmrelease neon-vortex -n default
kubectl get events -n default --sort-by='.lastTimestamp' | grep neon-vortex