In the beginning, there was MicroK8s on a Mac Studio. It was fast with 3 controlplane and 3 worker nodes, it was ARM64, but it was lonely. Today, I stand before a high-availability monument built on Proxmox with Terraform, orchestrated with Ansible, and maintained with GitOps using FluxCD.

Not long ago, my entire Kubernetes universe lived inside a humble Mac Studio — a single microk8s cluster with 6 nodes running on ARM64. It was cute, quiet, and completely unfit for the kind of multi‑DC, production‑grade nonsense I wanted to learn.

So I burned it down. And built this new place of worship.

Today, I run a high‑availability kubeadm cluster across three bare‑metal Proxmox Datacenters, all managed with Terraform, Ansible, and FluxCD. No cloud vendor lock‑in. No magic. Just a rack full of metal, a bunch of cables, and a lot of terminal time.

This is the story of my shrine — and how you can build one too.

UGLY WIRING:

MAJOR REASON WHY I CALLED IT SHRINE 😂

Traffic Flow at a Glance

Before we dive into the layers, here's how the traffic moves from my "pulpit" (Mac Studio) to the "shrine" (the cluster):

No inbound holes – all management traffic originates from my Mac or the cluster itself (GitOps pulls). This is how real datacenters work.

┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🖥️ macOS COMMAND CENTER (The Pulpit)                         │
│                                                                                      │
│              kubectl  │  Terraform  │  Ansible  │  Flux CLI  │  Git                  │
│                                                                                      │
│                        (All management tools installed locally)                      │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                          SSH │ API (HTTPS) │ Git (SSH/HTTPS)
                                          │
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🛡️ OPNsense Firewall (10.0.1.x)                              │
│                                                                                      │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐                      │
│   │ DHCP Server     │  │ Static DHCP     │  │ WireGuard VPN   │                      │
│   │ 10.0.1.100-xxx  │  │ MAC → IP Pinning │  │ Remote Access   │                      │
│   └─────────────────┘  └─────────────────┘  └─────────────────┘                      │
│                                                                                      │
│   • Split-Horizon DNS: *.georgehomelab.com → 10.0.1.x                               │
│   • Gateway for all Proxmox + Kubernetes traffic                                     │
│   • Firewall rules: WAN → LAN passes for management                                  │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                                          │ LAN (10.0.1.0/16)
                                          │ 2.5GbE Links
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🔌 Zyxel XMG1915-10E Switch                                  │
│                                                                                      │
│                     Star topology │ 8× 2.5GbE + 2× SFP+                              │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
              ┌───────────────────────────┼───────────────────────────┐
              │                           │                           │
              ▼                           ▼                           ▼
┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│                         │ │                         │ │                         │
│  🏗️ Proxmox Node 1      │ │  🏗️ Proxmox Node 2      │ │  🏗️ Proxmox Node 3      │
│  (proxmox-dc-1)         │ │  (proxmox-dc-2)         │ │  (proxmox-dc-3)         │
│  10.0.1.1x              │ │  10.0.1.1x              │ │  10.0.1.1x              │
│                         │ │                         │ │                         │
│  • Local ZFS Storage    │ │  • Local ZFS Storage    │ │  • Local ZFS Storage    │
│  • vmbr0 Bridge         │ │  • vmbr0 Bridge         │ │  • vmbr0 Bridge         │
│  • NFS Client (Backups) │ │  • NFS Client (Backups) │ │  • NFS Client (Backups) │
│                         │ │                         │ │                         │
│  Terraform → VM Creation via Proxmox API (telmate/provider)                          │
│  Packer → Ubuntu Cloud-Init Templates                                                │
│                                                                                      │
└─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
              │                           │                           │
              │ Cloud-Init DHCP (Static Reservations → Predictable IPs)                │
              │                           │                           │
              └───────────────────────────┼───────────────────────────┘
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         ☸️ HA Kubernetes Cluster (kubeadm)                           │
│                                                                                      │
│   ┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│   │ Control Plane Node 1    │ │ Control Plane Node 2    │ │ Control Plane Node 3    │
│   │ k8s-cp-1                │ │ k8s-cp-2                │ │ k8s-cp-3                │
│   │ 10.0.1.1xx              │ │ 10.0.1.1xx              │ │ 10.0.1.1xx              │
│   │                         │ │                         │ │                         │
│   │ • etcd (stacked)        │ │ • etcd (stacked)        │ │ • etcd (stacked)        │
│   │ • kube-apiserver        │ │ • kube-apiserver        │ │ • kube-apiserver        │
│   │ • kube-vip (VIP)        │ │ • kube-vip (VIP)        │ │ • kube-vip (VIP)        │
│   └─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
│                                                                                      │
│   ┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│   │ Worker Node 1           │ │ Worker Node 2           │ │ Worker Node 3           │
│   │ k8s-worker-1            │ │ k8s-worker-2            │ │ k8s-worker-3            │
│   │ 10.0.1.1xx              │ │ 10.0.1.1xx              │ │ 10.0.1.1xx              │
│   │                         │ │                         │ │                         │
│   │ • Calico CNI (BGP)      │ │ • Calico CNI (BGP)      │ │ • Calico CNI (BGP)      │
│   │ • kube-proxy            │ │ • kube-proxy            │ │ • kube-proxy            │
│   │ • Workload Pods         │ │ • Workload Pods         │ │ • Workload Pods         │
│   └─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
│                                                                                      │
│   Pod CIDR: 10.244.0.0/16 │ Service CIDR: 10.245.0.0/16 │ MetalLB: 10.0.1.2xx-2xx  │
│                                                                                      │
│   🔧 Bootstrapped entirely by Ansible (kubeadm playbook)                            │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                                          │ GitOps Sync (Outbound Only)
                                          │ FluxCD pulls from GitHub (no inbound!)
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🔄 FluxCD System (Inside Cluster)                            │
│                                                                                      │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐                      │
│   │ source-         │  │ kustomize-      │  │ helm-           │                      │
│   │ controller      │  │ controller      │  │ controller      │                      │
│   └─────────────────┘  └─────────────────┘  └─────────────────┘                      │
│                                                                                      │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐                      │
│   │ notification-   │  │ image-reflector-│  │ image-          │                      │
│   │ controller      │  │ controller      │  │ automation-     │                      │
│   └─────────────────┘  └─────────────────┘  └─────────────────┘                      │
│                                                                                      │
│   • Deployed as part of Ansible playbook (not a separate step)                       │
│   • Continuously reconciles cluster state with Git                                   │
│   • Auto-heals configuration drift                                                   │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                                          │ HTTPS/SSH (Outbound Pull)
                                          │
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                              📦 GitHub Private Repository                            │
│                                                                                      │
│   ┌─────────────────────────────────────────────────────────────────────────────┐    │
│   │  clusters/prod/                                                             │    │
│   │  ├── flux-system/          # Flux bootstrapping config                      │    │
│   │  │   ├── gotk-components.yaml                                               │    │
│   │  │   └── gotk-sync.yaml                                                     │    │
│   │  ├── apps/                  # Application deployments                       │    │
│   │  │   ├── metallb/                                                          │    │
│   │  │   ├── istio-ingress/                                                    │    │
│   │  │   └── prometheus-stack/                                                 │    │
│   │  └── infrastructure/        # Cluster-wide config                          │    │
│   │      ├── namespaces.yaml                                                   │    │
│   │      └── storage-class.yaml                                                │    │
│   └─────────────────────────────────────────────────────────────────────────────┘    │
│                                                                                      │
│   🔑 Source of Truth: Every change starts as a PR, reviewed, merged, then applied   │
│                                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────┘

Level 1: The Physical Layer (The Foundations)

Every altar begins with something tangible.

• Hardware: A fleet of Minisforum MS-01 machines acting as compute Datacenter (96GB RAM and 8GB nvidia GPU on each machine) . That's Total of 288GB RAM and 24GB nvidia GPU in the 3 Minisforum MS-01

• Network Entry Point: My Mac Studio (the “pulpit”), connected via Wi-Fi

• Firewall: OPNsense bridging external (192.168.1.x) to internal lab network (10.0.1.x)

• Out-of-Band Access: TinyPilot Voyager 2a and TESmart 4‑port HDMI KVM — BIOS-level control even when the OS is down

• Switch: Zyxel XMG1915-10E (2.5GbE + SFP+) is the Central Nervous System. With High-Velocity East-West Traffic ( Low Latency / High Throughput for etcd and storage )

Why I worship here:

Physical simplicity enables logical complexity.

No mystery cables. Everything is deliberate. This playground makes me the opportunity to play with any cloud-native tool with ease.

Level 2: The Infrastructure Layer (Proxmox Datacenter)

Before automation, there must be a foundation.

• Proxmox VE installed manually on all three Minisforum MS-01 machine

• Clustered into a single datacenter abstraction

• Networking:

  • vmbr0 → Kubernetes network

  • Static host IPs:

    • 10.0.1.1x

    • 10.0.1.1x

    • 10.0.1.1x

  • Gateway: 10.0.1.x (OPNsense)

• Storage:

  • Local ZFS (NVMe)

  • NFS for shared ISO + backups

The ritual:

I installed the first Proxmox VE manually on each machine via TinyPilot’s virtual media from my MAC-Studio browser over wifi.

No HDMI cable ever touched my desk.

Level 3: The Node Layer (Terraform Automation)

I no longer click buttons to create infrastructure.

I declare it.

Using the Proxmox Terraform provider, I define:

• VM CPU, memory, disk

• Network interfaces

• Clone source (Ubuntu template from Packer)

resource "proxmox_vm_qemu" "k8s_node" {
  for_each = var.nodes

  name        = each.value.name
  target_node = each.value.proxmox_node
  clone       = "ubuntu-24-04-template"
  cores       = each.value.cores
  memory      = each.value.memory

  network {
    model     = "virtio"
    bridge    = "vmbr0"
    ipconfig0 = "ip=dhcp"
  }
}

The DHCP Decision (And Why It Matters)

This was one of the most important lessons in my journey.

In my old Mac Studio setup, I used pure DHCP for Kubernetes nodes.

It worked… until every restart broke my cluster access.

What went wrong?

• Control plane nodes changed IPs

• kubeconfig became invalid

• API server endpoints broke

• etcd stability was at risk

Even with 3 control planes, the cluster wasn’t truly stable.

Why Not Static IPs?

Because static IPs inside the OS mean:

• Manual netplan configuration

• Hardcoding network logic into templates

• Reduced rebuild flexibility

That’s not how cloud-native systems behave.

The Solution: DHCP + Reservations

I used DHCP everywhere — but configured static reservations in OPNsense.

✔ Nodes auto-configure
✔ IPs never change
✔ Rebuilds are seamless
✔ etcd remains stable

💡 The Real Insight

Kubernetes doesn’t care how IPs are assigned — only that they don’t change.

Level 4: The Cluster Layer (Ansible + Kubeadm)

Once the infrastructure exists, it must be transformed.

Using Ansible:

• OS hardening

• Swap disabled

• containerd installed

• kubeadm, kubelet, kubectl configured

HA Control Plane

• 3 control plane nodes

• Stacked etcd (homelab-friendly)

• kube-vip for API virtual IP

Level 5: The Application Layer (GitOps with FluxCD)

This is where everything changes.

Instead of imperative deployments or declarative deployment with kubectl, I use GitOps FluxCD.

GitOps From Day One

FluxCD is not an add-on.

It is deployed during cluster creation via Ansible.

That means:

• Cluster is GitOps-ready immediately

• No manual bootstrap later

• No drift from day one

The Pull Model

• Flux runs inside the cluster

• Watches Git repository

• Pulls changes automatically

No inbound access required.

Traffic Flow

Mac Studio (192.168.1.x)
        │
        ▼
OPNsense Firewall (10.0.1.x)
        │
        ▼
Proxmox Cluster (10.0.1.1x–1x)
        │
        ▼
Kubernetes Nodes (DHCP → Reserved IPs)
        │
        ▼
FluxCD Controllers (inside cluster)
        │
        ▼
GitHub (OUTBOUND pull model)

Key Insight:

• ❌ GitHub never connects to your cluster

• ❌ No firewall holes needed

• ✅ Flux initiates outbound sync

Current State of the Shrine

• 3 control plane nodes ✅

• 3 worker nodes ✅

• etcd cluster healthy ✅

• Flux controllers distributed across nodes ✅

• Calico networking active ✅

This is no longer a lab.

It is a self-healing platform.

Before vs After

What I Learned

1. DHCP + reservations is the sweet spot

2. etcd requires stable identity, not static config

3. GitOps removes human drift completely

4. Terraform + Ansible + FluxCD = powerful combination

5. Firewalls must allow internal routing for automation

6. Never use root API for automation — use scoped tokens

What’s Next on the Altar

• Ceph or Longhorn for HA storage

• Velero for cluster backups

• External Secrets + Vault

• Cluster autoscaler experiments

Final Words

This homelab is more than a project.

It is a practice ground for real-world platform engineering.

The move from a single ARM node to a distributed HA cluster wasn’t just an upgrade in hardware — it was an upgrade in mindset.

My Mac Studio is no longer the host.

It is the pulpit.

The Shrine runs independently.

If you’re thinking of building something like this — do it.
Start small. Break things. Rebuild them better.

Now go build your own altar. 🛐

🤝 Stay Connected

Found this guide helpful? Follow my journey into Homelabing on LinkedIn! Click the blue LinkedIn button to connect: George Ezejiofor . Let’s keep building scalable, secure cloud-native systems, one project at a time! 🌐🔧

Modern Kubernetes platforms are inherently complex.

A single production cluster can run hundreds of microservices, service mesh components, CI/CD controllers, and security systems — all evolving continuously across both application and infrastructure layers.

Over the past few years, observability tooling has matured significantly. Platforms like Prometheus, Grafana, and Jaeger provide deep visibility into system behaviour.

But during an incident, visibility alone is not enough.

SREs are still required to manually interpret and correlate signals across multiple systems:

• Metrics must be queried and interpreted

• Logs must be searched and correlated

• Traces must be followed across service boundaries

• Infrastructure changes must be identified and linked to symptoms

Despite having all the data, the investigation process remains fundamentally manual.

Observability tools provide data — but they don’t provide reasoning.

The Real Bottleneck: Correlation and Time

The challenge is no longer data collection.

The real bottleneck is how quickly that data can be turned into understanding during an incident.

In practice, incident response often involves:

• switching between multiple dashboards

• writing ad hoc queries

• forming and testing hypotheses

• mentally correlating signals across systems

This process is:

• time-consuming

• cognitively demanding

• highly dependent on individual expertise

As systems scale, this model does not scale with them.

Rethinking Observability

This project explores a different approach.

Instead of treating observability as a collection of tools, it treats it as a reasoning problem.

What if observability data could be investigated automatically by an AI-driven SRE platform — one that understands Kubernetes, infrastructure behaviour, and failure patterns, while operating within strict, read-only security boundaries?

From Signals to Intelligence

The goal is to move from:

• dashboards → decisions

• alerts → investigations

• data → actionable insight

By introducing an AI-driven investigation layer, the platform aims to reduce the time required to:

• detect

• analyze

• and understand

failures in modern Kubernetes environments.

Prerequisites

Private AKS Cluster behind Azure Firewall: Deployed via Terraform Cloud, Using Gitops (Argocd with SSO) for deployment

Core Observability Components

The platform integrates multiple tools, each responsible for a specific signal type.

Metrics — Prometheus

Prometheus is the central metrics engine of the platform.

It is responsible for:

• scraping metrics from Kubernetes components

• collecting node and pod-level telemetry

• storing time-series data

• enabling PromQL-based querying

From your cluster:

• prometheus-prometheus-prometheus-0

• prometheus-node-exporter-*

• kube-state-metrics

• blackbox-exporter

These components allow the system to answer questions like:

• CPU / memory spikes

• pod restarts

• service latency trends

  

Visualisation — Grafana

Grafana provides real-time dashboards and visualisations.

Role in the platform:

• visualising Prometheus metrics

• building SRE dashboards

• supporting manual and AI-assisted investigations

Example pod:

• prometheus-grafana-*

  

Logs — Elasticsearch + Kibana

Logging is handled using Elasticsearch and Kibana.

Responsibilities:

• centralized log aggregation

• indexing and searching logs

• enabling correlation with metrics and traces

From your cluster:

• elasticsearch-es-default-0

• kibana-kb-*

This enables:

• application log analysis

• error tracing

• debugging failed workloads

  

Distributed Tracing — Jaeger

Jaeger provides end-to-end request tracing across services.

Role:

• track request flow across microservices

• identify latency bottlenecks

• debug service-to-service communication

From your cluster:

• jaeger-*

This is critical in service mesh environments where requests traverse multiple services.

Service Mesh Observability — Kiali

Kiali is used to visualize the service mesh topology.

It provides:

• traffic flow visualization

• service dependencies

• health status of services

From your cluster:

• kiali-*

This is especially useful with Istio-based environments.

Telemetry Collection — OpenTelemetry

OpenTelemetry acts as the telemetry pipeline.

Responsibilities:

• collecting metrics, logs, and traces

• exporting data to observability backends

• enabling standardized instrumentation

From your cluster:

• opentelemetry-collector-*

• opentelemetry-operator-*

• opentelemetry-ebpf-instrumentation-*

The use of eBPF-based instrumentation is particularly powerful:

• no application code changes required

• deep kernel-level visibility

• automatic tracing and metrics collection

  

Kubernetes State Monitoring

Additional components provide insight into cluster state:

• kube-state-metrics → Kubernetes object state

• cadvisor → container resource usage

• node-exporter → node-level metrics

These ensure the platform can inspect:

• deployments

• pods

• nodes

• resource utilization

Alerting — Alertmanager

While Prometheus is responsible for collecting and evaluating metrics, Alertmanager handles what happens when something goes wrong.

In this platform, Alertmanager acts as the bridge between detection and response.

Role in the Platform

Alertmanager is responsible for:

• receiving alerts from Prometheus

• grouping and deduplicating alerts

• routing alerts to the appropriate channels

• managing alert silencing and escalation

From your cluster:

• alertmanager-prometheus-alertmanager-0

Why Alertmanager Matters

In traditional setups, Alertmanager typically sends notifications to:

• email

• Slack

• PagerDuty

However, this introduces a key limitation:

Alerts notify humans — but they do not investigate the problem.

This creates a workflow like:

Alert Triggered
     ↓
Human SRE Responds
     ↓
Manual Investigation Begins
     ↓
Root Cause Found (Eventually)

This process is:

• reactive

• slow

• dependent on human availability

Why This Stack Matters

Each tool provides a piece of the puzzle, but none provides full understanding alone.

The Core Limitation

Despite having a powerful observability stack:

These tools provide data, not reasoning.

An SRE still has to:

• jump between dashboards

• write queries manually

• correlate signals mentally

• identify root causes

This is:

• time-consuming

• error-prone

• not scalable

Transition to the Next Section

This limitation led to the key idea behind this project:

What if we could add an intelligent layer on top of this observability stack — one that can reason across metrics, logs, traces, and infrastructure changes automatically?

This is where the AI-driven SRE platform comes in.

Project Vision

The goal of this project is simple:

Build a secure AI-driven SRE platform that can investigate Kubernetes incidents by correlating signals across observability systems and the cloud control plane.

The platform focuses on investigation, not just visibility.

Core capabilities:

• Investigate incidents end-to-end

• Query observability systems programmatically

• Understand Kubernetes state in real time

• Correlate cloud-level events (Azure Activity Logs)

• Attribute changes to identities (Entra ID)

• Operate under strict read-only RBAC

The design principle is clear:

No write access. No assumptions. Only evidence-driven analysis.

High-Level Platform Architecture

The platform is structured into four layers:

AI SRE Reasoning Layer
        │
Agent Tooling Layer (MCP)
        │
Observability Platform
        │
Kubernetes + Azure Infrastructure

The Agentic SRE Model

The platform follows an agent-based investigation model.

George-GPT (Lead SRE Agent)

Acts as the coordinator.

Responsibilities:

• receive investigation requests

• decide which agents to use

• aggregate findings

• produce root cause analysis

Specialist Agents

Each agent is scoped to a single domain:

Delegation Model

• The lead agent does not query systems directly

• It delegates tasks to specialist agents

• Each agent returns structured results

• Final output is synthesized into a single conclusion

This keeps reasoning focused and composable.

MY CHAT WITH GEORGE-GPT

user

who are you and what can you do?

kagent/george-gpt

I am GeorgeGPT, Lead SRE Engineer and orchestrator for a production Kubernetes cluster, powered by Gemini 2.5 Flash. I am a specialized SRE agent, not a generic AI.

I work alongside several peer agents, all deployed in the kagent namespace and powered by Gemini Flash-Lite:

k8s-agent: Kubernetes resource management helm-agent: Helm release management istio-agent: Istio service mesh analysis promql-agent: Prometheus PromQL queries observability-agent: Grafana dashboards, alerts, and datasources k8sgpt: Automated Kubernetes diagnostics I can also talk to 11 specialized MCP (Model Context Protocol) servers, which expose a total of 74 tools for various SRE tasks:

kagent-tool-server (namespace: kagent) — 16 tools for Kubernetes resources, Helm releases, Istio proxy status, and Grafana operations (k8s_get_resources, helm_list_releases, istio_proxy_status, grafana_get_dashboard, etc.) mcp-elasticsearch (namespace: mcp-servers) — 5 tools for Elasticsearch log search, ESQL queries, and cluster health (search, esql, list_indices, etc.) mcp-jaeger (namespace: mcp-servers) — 8 tools for Jaeger distributed tracing (list_jaeger_services, search_jaeger_traces, get_error_jaeger_traces, etc.) mcp-kiali (namespace: mcp-servers) — 9 tools for Kiali Istio service mesh observability (diagnose_503, get_traffic_graph, get_service_health, etc.) mcp-auth (namespace: mcp-servers) — 3 tools for JWT/OIDC token analysis (diagnose_auth_failure, validate_jwt, decode_jwt) mcp-keycloak (namespace: mcp-servers) — 5 tools for Keycloak identity management (diagnose_kubeflow_auth, get_client_config, get_realm_settings, etc.) mcp-oauth2-proxy (namespace: mcp-servers) — 7 tools for OAuth2 proxy diagnostics (diagnose_kubeflow_503, get_oauth2_proxy_config, ping_oauth2_proxy, etc.) mcp-otel (namespace: mcp-servers) — 3 tools for OpenTelemetry trace correlation (correlate_trace, get_error_traces, search_traces) mcp-orchestrator (namespace: mcp-servers) — 5 tools for incident lifecycle management (start_incident, add_finding, resolve_incident, etc.) george-gpt (namespace: mcp-servers) — 13 tools for custom SRE tasks including alerts, PromQL queries, and kubectl/helm remediation (check_alerts, run_promql, investigate, kubectl_remediate, helm_remediate, etc.) mcp-prometheus (namespace: mcp-servers) — (Not currently used; I use run_promql() from george-gpt for Prometheus queries.) My primary function is to investigate and resolve production incidents in the Kubernetes cluster by intelligently calling these tools, correlating data across various observability signals (metrics, logs, traces, Kubernetes state), identifying root causes, and providing actionable remediation steps. I follow comprehensive playbooks for common scenarios like login failures, pod evictions, and service mesh 503 errors.

TESTING

## image: georgeezejiofor/echo-pod:blue-v1  (correct image)
kubectl create ns terranetes
kubectl create deployment echo-pod --image=georgeezejiofor/echo-pod:blue-bad-v1 -n terranetes --replicas=2

TEST RESULTS

user

why is the pod in terranetes namespace not running?

kagent/george-gpt

k8s_get_resources adk-485b17df-a174-4d49-a5da-c663935622d9 Completed

Arguments { "namespace": "terranetes", "resource_type": "pod" }

Results k8s_describe_resource adk-f6786634-19b3-4da0-a1fb-bd87c306e03d Completed

Arguments { "namespace": "terranetes", "resource_name": "echo-pod-779cb697d4-czvgp", "resource_type": "pod" }

Results Name: echo-pod-779cb697d4-czvgp Namespace: terranetes Priority: 0 Service Account: default Node: aks-system-30404036-vmss000000/10.0.0.100 Start Time: Tue, 17 Mar 2026 21:55:51 +0000 Labels: app=echo-pod pod-template-hash=779cb697d4 Annotations: Status: Pending IP: 10.0.0.132 IPs: IP: 10.0.0.132 Controlled By: ReplicaSet/echo-pod-779cb697d4 Containers: echo-pod: Container ID:
Image: georgeezejiofor/echo-pod:blue-bad-v1 Image ID:
Port: Host Port: State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fkdjx (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-fkdjx: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt Optional: false DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 8m57s default-scheduler Successfully assigned terranetes/echo-pod-779cb697d4-czvgp to aks-system-30404036-vmss000000 Normal Pulling 5m59s (x5 over 8m56s) kubelet Pulling image "georgeezejiofor/echo-pod:blue-bad-v1" Warning Failed 5m59s (x5 over 8m56s) kubelet Failed to pull image "georgeezejiofor/echo-pod:blue-bad-v1": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/georgeezejiofor/echo-pod:blue-bad-v1": failed to resolve reference "docker.io/georgeezejiofor/echo-pod:blue-bad-v1": docker.io/georgeezejiofor/echo-pod:blue-bad-v1: not found Warning Failed 5m59s (x5 over 8m56s) kubelet Error: ErrImagePull Warning Failed 3m49s (x20 over 8m55s) kubelet Error: ImagePullBackOff Normal BackOff 3m37s (x21 over 8m55s) kubelet Back-off pulling image "georgeezejiofor/echo-pod:blue-bad-v1"

k8s_get_events adk-cd6d9711-1275-4279-9c30-5beb4d1f50d4 Completed

Arguments { "namespace": "terranetes" }

Results kubectl_remediate adk-6b617523-0d16-4b84-95b3-1bf8de120c84 Failed

Arguments { "args": "patch deployment echo-pod -n terranetes --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value": "georgeezejiofor/echo-pod:blue"}]' ", "dry_run": false }

kagent/george-gpt

The pods in the terranetes namespace are not running because the Kubernetes cluster is unable to pull the specified container image.

Root Cause: The pods echo-pod-779cb697d4-czvgp and echo-pod-779cb697d4-x8bbd are in an ImagePullBackOff state. The error message from the kubelet clearly indicates: Failed to pull image "georgeezejiofor/echo-pod:blue-bad-v1": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/georgeezejiofor/echo-pod:blue-bad-v1": failed to resolve reference "docker.io/georgeezejiofor/echo-pod:blue-bad-v1": docker.io/georgeezejiofor/echo-pod:blue-bad-v1: not found

This means the container image georgeezejiofor/echo-pod:blue-bad-v1 cannot be found on Docker Hub (docker.io).

Impact: The echo-pod application in the terranetes namespace is completely non-functional, as its pods cannot start.

Remediation Steps:

Verify Image Name and Tag: The primary action is to confirm that the image georgeezejiofor/echo-pod with the tag blue-bad-v1 actually exists in your Docker Hub repository. There might be a typo in the image name or tag within the Kubernetes Deployment configuration.

Update the Deployment with the Correct Image: If the image name or tag is incorrect, you need to update the echo-pod Deployment in the terranetes namespace to reference the correct, existing image.

Assuming your deployment is named echo-pod, you can use the following kubectl command to patch the deployment with a corrected image (replace georgeezejiofor/echo-pod:blue with the actual correct image and tag):

kubectl patch deployment echo-pod -n terranetes --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value": "georgeezejiofor/echo-pod:blue"}]'

After running this command, Kubernetes will attempt to pull the new, correct image, and the pods should start running. You can verify this by checking the pod status again using kubectl get pods -n terranetes.

Live Incident Example – GEORGE-GPT Resolved ImagePullBackOff in <2 Minutes

The pods immediately went into Pending → ImagePullBackOff. Query to GEORGE-GPT: why is the pod in terranetes namespace not running? What happened next (real-time tool calls):

k8s_get_resources — listed all pods in terranetes k8s_describe_resource — pulled detailed pod description for one failing pod k8s_get_events — fetched namespace events

GEORGE-GPT's immediate conclusion: The pods … are not running because the Kubernetes cluster is unable to pull the specified container image. Root Cause: … ImagePullBackOff … failed to resolve reference "docker.io/georgeezejiofor/echo-pod:blue-bad-v1": … not found Impact: The echo-pod application in the terranetes namespace is completely non-functional, as its pods cannot start.

Remediation recommendation (auto-generated safe command): George-GPT suggested how to fix the error , it doesn't have any permission to make any change in the aks architecture.

Tool-Backed Reasoning (MCP)

Agents interact with systems through MCP servers (Model Context Protocol).

This enforces:

• structured queries

• controlled access

• consistent outputs

Why MCP

• prevents arbitrary access

• standardizes interactions

• improves reliability of results

Incident Investigation Walkthrough

This section demonstrates the system in action.

Example:

• ingress gateway failure

• traffic disruption

Flow:

1. user submits investigation query

2. agents collect evidence

3. metrics are analyzed

4. logs are inspected

5. Azure activity logs are checked

6. root cause is identified

Security by Design

Security is enforced at every layer.

Read-Only RBAC

Agents:

• cannot create resources

• cannot modify resources

• cannot delete resources

They can only:

• read cluster state

• query telemetry

GitOps Deployment

All changes flow through:

GitHub
   ↓
GitHub App (OIDC)
   ↓
ArgoCD
   ↓
AKS

This guarantees:

• no manual changes

• full traceability

• secure authentication

Identity Correlation

The platform integrates:

• Azure Activity Logs

• Entra ID identities

This enables:

precise attribution of changes — who did what and when.

Traditional vs AI-Driven Observability

Traditional stack:

• dashboards

• alerts

• manual investigation

Your platform adds:

• reasoning layer

• cross-system correlation

• automated investigation

Difference:

from data exploration → to decision support.

Lessons Learned

• Read-only AI systems are safer and more predictable

• Structured tooling improves reliability

• Correlation is the hardest part of observability

Conclusion

Observability tools provide signals, not understanding.

By adding a secure AI-driven reasoning layer, we can turn fragmented data into actionable insight and significantly reduce incident response time in Kubernetes environments.

🤝 Stay Connected

Found this guide helpful? Follow my journey into AI Agent Automation Engineer on LinkedIn! Click the blue LinkedIn button to connect: George Ezejiofor on LinkedIn. Let’s keep building scalable, secure cloud-native systems, one project at a time!

Introduction

Navigating the world of home networking can be a “plug-and-play” experience — until you decide to build a professional-grade lab. When I set out to install OPNsense on an Intel N150 fanless mini-PC, I ran into the classic “locked out” scenarios, interface mismatches, and DNS rebind attacks.

Here is the definitive guide on how I moved from a broken installer to a fully headless, domain-accessible firewall that I can manage from anywhere in the world.

The Hardware Architecture

Component Specifications:

• Device: Intel N150 (4 Cores, 4 Threads)

• Port ETH0 (igc0): LAN (Connected to Mac Studio)

• Port ETH1 (igc1): WAN (Connected to Home ISP Router)

• Network Target: 10.0.1.1/16

• Management: TinyPilot Voyager KVM (headless control)

I Spent 3 Days Locked Out of My Own Firewall — The OPNsense Guide That Would Have Saved Me

You know that feeling when you buy new hardware and think "I'll have this running in an hour"?

Yeah. Three days later I was still locked out of my OPNsense box, toggling cables like some kind of network rain dance, wondering why I didn't just buy a pre-configured Ubiquiti.

The Intel N150 mini-PC sat there silently judging me (it's fanless, so literally silent). Two ethernet ports. One job. Somehow I managed to hit every possible failure mode.

The interface detection bug. The Live mode trap. The IP conflict with my home router. DNS rebind attacks blocking my own domain. The "Block private networks" checkbox that exists specifically to ruin your day.

I hit them all. Here's what I wish someone had told me.

The Hardware

Nothing exotic:

• Device: Intel N150 — 4 cores, fanless, silent

• Port 1 (igc0): LAN, connected to my Mac Studio

• Port 2 (igc1): WAN, connected to home ISP router

• Network: 10.0.1.0/16

• Backup plan: TinyPilot Voyager KVM for console access when (not if) I lock myself out

┌─────────────────┐      ┌─────────────────┐
│  Home Router    │ WAN  │  OPNsense N150  │
│  192.168.1.xxx  │─────▶│  WAN: igc1      │
└─────────────────┘      │  LAN: igc0      │
                         └────────┬────────┘
                                  │ LAN
                         ┌────────▼────────┐
                         │  Mac Studio     │
                         │  10.0.1.xxx/16  │
                         └─────────────────┘

The Failures (Learn From Mine)

Failure 1: The Live Mode Trap

OPNsense boots into Live mode by default. Everything works. You configure interfaces, set IPs, enable DHCP. Reboot. Gone. All of it.

Because Live mode doesn't persist anything. You have to actually run the installer.

What to do: Select "Install" from the boot menu. Login as installer with password opnsense. Actually install to the SSD.

This seems obvious in retrospect. Cost me an entire evening.

Failure 2: The Cable Timing Bug

Here's a weird one. During initial setup, if both ethernet cables are plugged in, interface detection gets confused. Sometimes igc0 becomes igc1. Sometimes neither works.

What to do:

1. Only plug in WAN cable first

2. Assign WAN to igc1

3. Then plug in LAN cable

4. Assign LAN to igc0

5. Now reboot

Three fresh installations before I figured this out.

Failure 3: IP Conflict

OPNsense defaults LAN to 192.168.1.1. My home router is also 192.168.1.1. Same subnet. Routing table confusion. Web UI unreachable.

What to do: From the console, select Option 2 and change LAN to 10.0.1.1/16 (or whatever doesn't conflict with your home network).

Failure 4: Wi-Fi Management Blocked

I wanted to manage the firewall from anywhere in my house without being physically cabled to it. But traffic from my Wi-Fi comes in on the WAN interface, and OPNsense blocks that by default.

What to do:

1. Interfaces → WAN — Uncheck "Block private networks"

   Your home router gives OPNsense a private 192.168.x.x IP. This checkbox blocks all traffic from private ranges. Including your Wi-Fi.

2. Firewall → Rules → WAN — Add a rule:

   • Pass TCP to WAN address on port 443

3. System → Settings → Administration — Set Listen Interfaces to "All"

Failure 5: DNS Rebind Attack Protection

I set up firewall.homelab.com to point to my OPNsense WAN IP. Loaded it in my browser. Blocked.

OPNsense has built-in protection against DNS rebind attacks — it blocks external domains that resolve to internal IPs. Great security. Terrible for homelabs.

What to do: Either disable DNS Rebind checks entirely, or add your domain to "Alternate Hostnames" in System → Settings → Administration.

The Working Configuration

After all the failures, here's what actually works.

LAN Rules (igc0 / 10.0.1.0/16)

WAN Rules (igc1)

NAT Outbound

Hybrid mode. Let OPNsense auto-create masquerade rules for your internal networks.

WireGuard Remote Access

I travel to Nigeria regularly. Needed a way to access my homelab from anywhere.

Install: System → Firmware → Plugins → os-wireguard

Server config:

• Enable WireGuard

• Listen Port: 51820

• Tunnel Address: 10.0.100.1/24

• DNS: 10.0.1.1

Client peer:

• Name: Nigeria_Laptop

• Public Key: (from client device)

• Allowed IPs: 10.0.100.2/32

Don't forget: Firewall rule for UDP 51820 on WAN.

Now I can hit my homelab from 5,000 miles away through an encrypted tunnel.

Ansible Automation

Once it was working manually, I immediately codified everything. If this N150 dies, I'm not spending another three days on it.

- name: Deploy Core Rules
  hosts: opnsense
  tasks:
    - name: LAN to Internet
      oxlorg.opnsense.rule:
        description: LAN_to_Internet
        interface: igc0
        protocol: any
        source_net: 10.0.1.0/16
        action: pass
        sequence: 10

    - name: Wi-Fi Management
      oxlorg.opnsense.rule:
        description: WiFi_Mgmt
        interface: igc1
        protocol: tcp
        destination_port: 443
        action: pass
        sequence: 10

Using the oxlorg.opnsense Ansible collection. The full playbook covers everything — interfaces, DHCP, DNS, WireGuard, static reservations.

One command rebuild:

ansible-playbook -i inventory.yml playbooks/99_full_rebuild.yml --ask-vault-pass

Sanity Checklist

Before you call it done:

• [ ] WAN Interface: "Block private networks" is UNCHECKED

• [ ] WAN Rules: Port 443 management rule exists

• [ ] WAN Rules: Port 51820 WireGuard rule (if using)

• [ ] Admin Settings: DNS Rebind checks disabled or alternate hostname set

• [ ] Admin Settings: Listen Interfaces = All

• [ ] Unbound DNS: Domain override configured

• [ ] NAT Outbound: Hybrid mode, networks masqueraded

Troubleshooting

Can't reach 10.0.1.1? Mac Ethernet set to DHCP? Cable in igc0?

Wi-Fi management broken? "Block private networks" still checked. Or WAN rule missing.

Domain doesn't resolve? DNS Rebind protection. Disable it or add alternate hostname.

WireGuard won't connect? Port 51820 not open on WAN.

Inter-subnet traffic blocked? Your LAN rules have wrong source/destination subnets.

The Result

Four ways to access my firewall now:

Silent fanless box sits in the corner. Accessible from anywhere. Fully automated rebuild if needed.

Worth the three days? Probably. Would I do it again? With this guide, it takes about 45 minutes.

Resources

• OPNsense Documentation

• Ansible OPNsense Collection

• WireGuard

This is part of my homelab series. Next up: Proxmox clustering, Kubernetes on bare metal, and eventually some low-latency trading infrastructure.

Hit me up in the comments if you're stuck. I've probably made your exact mistake.

Deploying applications in a private Azure Kubernetes Service (AKS) cluster is a widely adopted strategy to enhance security by isolating workloads and Kubernetes API server from the public internet. However, one of the biggest challenges DevOps and platform engineers face is how to expose those private services securely to external consumers — whether they're internal teams, partner systems, or customers — without compromising on network security, scalability, or operational agility.

Traditionally, this required complex combinations of internal load balancers, Application Gateway Ingress Controller (AGIC), or service meshes. But now, Azure has introduced a powerful solution: Application Gateway for Containers.

This new capability brings layer 7 load balancing natively to Kubernetes environments using the Kubernetes Gateway API, while offering real-time updates, mutual TLS (mTLS), traffic splitting, and seamless support for private networking. It significantly simplifies ingress management, especially for private AKS clusters where outbound and inbound traffic control is critical.

In this guide, we’ll walk you through how to securely expose your application from a private AKS cluster using Application Gateway for Containers, integrated with cert-manager for automated TLS certificate management. You’ll learn how to deploy the full setup, issue certificates from Let’s Encrypt, and test HTTPS access to your workloads — all within a locked-down, private Kubernetes environment.

🔍 What is Application Gateway for Containers?

🚀 Overview

Azure Application Gateway for Containers is a Kubernetes-native, Layer 7 load balancer designed for secure and scalable ingress in AKS workloads. It extends Azure’s load balancing capabilities by enabling:

• Advanced routing,

• TLS and mTLS termination,

• Real-time updates, and

• Kubernetes Gateway API support.

Unlike traditional load balancers, it operates outside the AKS data plane, managed by an Azure Load Balancer (ALB) Controller running inside the cluster. This architecture is ideal for exposing private AKS apps to external users — without sacrificing security or performance.

🔄 From AGIC to AGC

Application Gateway for Containers (AGC) is the evolution of the Application Gateway Ingress Controller (AGIC). While AGIC relied on ARM calls and had limitations in scale and update speed, AGC offers:

• Native support for Gateway API

• Near real-time configuration updates

• Support for >1400 pods

• Simplified ops with fewer moving parts

It removes the bottlenecks of AGIC by offloading most control logic to Azure-managed components while retaining Kubernetes-native interfaces.

🛠️ Core Components

AGC is composed of three main resources:

• Gateway
  The parent Azure resource that manages the control plane, mapping Kubernetes Ingress/Gateway API resources to actual proxy configuration.

• Frontends
  Define public/private entry points (with FQDNs) for client traffic. Frontends are referenced in Kubernetes Gateway/Ingress specs.

• Associations
  Bind the Gateway to a delegated subnet in the AKS VNet. Each association maps traffic to the correct backend pods. Requires a /24 subnet (≥ 256 IPs).

🔗 Key Dependencies

To deploy AGC, you need:

• 🔒 User-Assigned Managed Identity
  Grants the ALB Controller access to manage Azure resources securely (requires roles like AppGw for Containers Config Manager and Network Contributor).

• 🌐 Subnet Delegation
  A subnet delegated to Microsoft.ServiceNetworking/trafficControllers. This hosts the AGC proxies. Only one gateway per delegated subnet.

• 📡 Private IP Address
  AGC uses this as the listener endpoint for incoming traffic. It’s not a standalone ARM resource but critical for routing.

This modern ingress solution empowers teams to expose apps from private AKS clusters confidently — with TLS/mTLS, autoscaling, traffic splitting, and seamless GitOps or IaC integration via Azure CLI, Terraform, or native K8s YAML.

Prerequisites

Before setting up Azure Application Gateway for Containers to securely expose applications from a private AKS cluster, ensure you have the following:

• ✅ Azure Subscription: An active Azure subscription with Contributor role permissions to create and manage AKS clusters, Application Gateway resources, and networking components. Sign up for Azure.

• 🏗️ Private AKS Cluster: A private AKS cluster deployed using Terraform in Terraform Cloud. Ensure the cluster is running and accessible via kubectl. The cluster should have a user-assigned managed identity and a delegated subnet for Application Gateway integration. Learn about private AKS clusters.

• 🛠️ Azure CLI: Installed (version 2.30 or later) and configured with az login. Use it to manage Azure resources and AKS. Install Azure CLI.

• 🖥️ kubectl: Installed and configured to interact with your AKS cluster. Verify connectivity with kubectl get nodes. Install kubectl.

• 📦 Helm: Installed (version 3.x) for deploying cert-manager and other Helm charts. Install Helm.

• 🔐 Workload Identity Federation – for secure, passwordless authentication to Azure resources without secrets (for cert-manager deployment)

• 🔒 Cert-Manager: Ready to install via Helm or already installed in your AKS cluster. Cert-manager will automate TLS certificate issuance (e.g., from Let’s Encrypt). Basic familiarity with cert-manager’s ClusterIssuer or Issuer resources is recommended. Cert-Manager Documentation.

• 🌐 Domain for TLS: A registered domain dripwithstyle.com for configuring HTTPS with TLS certificates. Ensure you can manage DNS records to point to the Application Gateway’s public IP or DNS name. A subdomain (e.g., echotls.dripwithstyle.com) is typically used for the application.

• 🔑 User-Assigned Managed Identity: A managed identity assigned to the AKS cluster for Application Gateway integration. This enables secure communication between AKS and Azure resources. Learn about managed identities.

• 🌐 Subnet Delegation: A dedicated subnet in your AKS virtual network, delegated for Application Gateway for Containers. This ensures proper networking configuration. Configure subnet delegation.

• 📚 Knowledge Requirements:

  • Familiarity with Kubernetes Ingress and Gateway API concepts to configure routing rules.

  • Basic understanding of Azure networking (e.g., VNets, private IPs, DNS).

  • Awareness of TLS certificate management (e.g., Let’s Encrypt or custom CAs).

Ensure all tools are installed and configured on your local machine or CI/CD environment. Verify Terraform state in Terraform Cloud to confirm the AKS cluster’s configuration, including networking and identity settings.

🚀 Deployment Strategies

When deploying Application Gateway for Containers, Azure offers two distinct strategies for managing the lifecycle of the gateway and its associated resources. The choice depends on how much control you want over the infrastructure and how tightly you want it integrated with Kubernetes.

🧩 1. Bring Your Own (BYO) Deployment (I will use this strategy)

In the BYO model, you manually provision and manage all AGC resources — including the Gateway, Frontend, and Association — using tools like the Azure Portal, CLI, PowerShell, or Terraform.

• You create the Frontend resource in Azure before referencing it in your Kubernetes Gateway definition.

• The lifecycle of these resources is decoupled from Kubernetes — deleting a Gateway in Kubernetes does not delete the Azure Frontend.

• This model gives you full control over infrastructure but requires more manual management.

✅ Best for: Teams that prefer explicit control over Azure resources or already manage infrastructure via Terraform or Bicep.

⚙️ 2. Managed by ALB Controller

In this fully managed model, the Azure Load Balancer (ALB) Controller — running inside your AKS cluster — handles the lifecycle of AGC resources automatically.

• You define an ApplicationLoadBalancer custom resource in Kubernetes.

• The ALB Controller provisions the Gateway, Frontend, and Association resources in Azure.

• When you create a Gateway resource that references the ApplicationLoadBalancer, the controller automatically creates and manages the corresponding Frontend.

• Deleting the Kubernetes Gateway also cleans up the Azure resources.

✅ Best for: Teams that want a Kubernetes-native experience with minimal Azure-side configuration.

🧠 Summary

kubectl get pod -A | grep -E 'alb|cert|wi-webhook'

I use Azure active directory to connect to my jumpbox.

Then Verified the tools on the prerequisites

Verify user assigned identity client id

az identity federated-credential list \
  --identity-name certmanager-uami-terranetes \
  --resource-group MC_rg-terranetes-aks-prod_aks-terranetes-cluster-prod_westeurope \
  --query "[?name=='cert-manager'].{Name:name,Subject:subject,Audience:audience,Issuer:issuer}" -o table

Verify user assigned identity resource federated identity

az identity federated-credential list \
  --identity-name certmanager-uami-terranetes \
  --resource-group MC_rg-terranetes-aks-prod_aks-terranetes-cluster-prod_westeurope \
  --query "[?name=='cert-manager'].{Name:name,Subject:subject,Audience:audience,Issuer:issuer}" -o table

Export AGC values

## Export values for Application Gateway for container
export AGC_RESOURCE_ID=$(az network alb show \
  --resource-group rg-terranetes-aks-prod \
  --name terranetes-aks-alb \
  --query id -o tsv)

export AGC_FRONTEND_NAME="terranetes-aks-alb-frontend"

## verify the exported values
echo "AGC_RESOURCE_ID=${AGC_RESOURCE_ID}"
echo "AGC_FRONTEND_NAME=${AGC_FRONTEND_NAME}"

Deploy echo pod application and services

cat << EOF | kubectl apply -f -
# 1. BLUE Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blue-echo-pod
  labels:
    version: blue
spec:
  replicas: 2
  selector:
    matchLabels:
      version: blue
  template:
    metadata:
      labels:
        version: blue
    spec:
      containers:
      - name: echo-pod
        image: georgeezejiofor/echo-pod:blue-v1
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_COLOR
          value: "BLUE"

---
# 2. BLUE Service
apiVersion: v1
kind: Service
metadata:
  name: blue-echo-service
spec:
  selector:
    version: blue
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

---
# 3. GREEN Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: green-echo-pod
  labels:
    version: green
spec:
  replicas: 2
  selector:
    matchLabels:
      version: green
  template:
    metadata:
      labels:
        version: green
    spec:
      containers:
      - name: echo-pod
        image: georgeezejiofor/echo-pod:green-v1
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_COLOR
          value: "GREEN"

---
# 4. GREEN Service
apiVersion: v1
kind: Service
metadata:
  name: green-echo-service
spec:
  selector:
    version: green
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

EOF

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: echopod-gateway-tls
  namespace: default
  annotations:
    alb.networking.azure.io/alb-id: "/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-terranetes-aks-prod/providers/Microsoft.ServiceNetworking/trafficControllers/terranetes-aks-alb"
    cert-manager.io/cluster-issuer: letsencrypt-dns01-istio
    service.beta.kubernetes.io/port_80_no_probe_rule: "true"
    service.beta.kubernetes.io/port_443_no_probe_rule: "true"
spec:
  gatewayClassName: azure-alb-external
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      hostname: "echotls.dripwithstyle.com"
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "echotls.dripwithstyle.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: terranetes-echopod-tls
            kind: Secret
            group: ""
      allowedRoutes:
        namespaces:
          from: All
EOF

Deploy clusterissuer for Azuredns with dns01

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01-istio
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: george@gmail.com
    privateKeySecretRef:
      name: letsencrypt-dns01-istio-key
    solvers:
      - dns01:
          azureDNS:
            subscriptionID: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"  # Subscription ID
            resourceGroupName: "rg-terranetes-poc"         # dns zone resource group
            hostedZoneName: "dripwithstyle.com"           # This should be the Azure DNS zone name
            environment: AzurePublicCloud
            managedIdentity:
              clientID: "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"      # This should be the client ID of the user-assigned managed identity
EOF

Create argocd certificate

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: terranetes-istio-cert
  namespace: istio-ingress                   # Must match the namespace of the Istio ingress gateway
spec:
  secretName: terranetes-istio-tls            # This is the credentialName in the Gateway resource
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 4096
  issuerRef:
    name: letsencrypt-dns01-istio
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
    - "echotls.dripwithstyle.com"
EOF

Deploy echo-pod gateway (Bring your own deployment strategy)

cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: echopod-gateway-tls
  namespace: default
  annotations:
    alb.networking.azure.io/alb-id: "/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-terranetes-aks-prod/providers/Microsoft.ServiceNetworking/trafficControllers/terranetes-aks-alb"
    cert-manager.io/cluster-issuer: letsencrypt-dns01-istio
    service.beta.kubernetes.io/port_80_no_probe_rule: "true"
    service.beta.kubernetes.io/port_443_no_probe_rule: "true"
spec:
  gatewayClassName: azure-alb-external
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      hostname: "echotls.dripwithstyle.com"
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "echotls.dripwithstyle.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: terranetes-echopod-tls
            kind: Secret
            group: ""
      allowedRoutes:
        namespaces:
          from: All
EOF

Deploy HTTPRoute tls

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: terranetes-echopod-route-tls
  namespace: default
spec:
  parentRefs:
  - name: echopod-gateway-tls
  hostnames:
  - echotls.dripwithstyle.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: blue-echo-service
      port: 80
      weight: 50
    - name: green-echo-service
      port: 80
      weight: 50
EOF

Verify the deployed objects

kubectl get pod,svc,gateway,httproute,certificate -n default | grep echo

Traffic routing for blue and green echo pod services

🎯Conclusion

You’ve successfully deployed a secure, production-grade ingress for your private Azure Kubernetes Service (AKS) cluster using Azure Application Gateway for Containers, integrated with cert-manager for TLS and leveraging Kubernetes Gateway APIs with Infrastructure-as-Code principles.

✅ What’s Been Achieved

• 🔒 Exposed private AKS services securely to external users without compromising network security.

• ⚙️ Automated TLS certificate management with cert-manager for seamless HTTPS.

• 🚀 Built a scalable, low-latency ingress using modern Gateway API standards.

• 🔁 Implemented flexible deployment strategies, including ALB Controller automation.

🧭 What’s Next?

My next project explores advanced cloud-native networking:
🔧 Managing and exposing frontend services for 10 microservices with the Azure-managed AKS Istio add-on, using service mesh patterns and zero-trust security. Stay tuned for updates!

🤝 Stay Connected

Found this guide helpful? Follow my journey into Istio and microservices on LinkedIn! Click the blue LinkedIn button to connect: George Ezejiofor on LinkedIn. Let’s keep building scalable, secure cloud-native systems, one project at a time! 🌐🔧

In modern DevOps environments, secure and centralized access control is critical — especially when managing production-grade Kubernetes clusters with GitOps tools like Argo CD. Rather than relying on local user accounts or static passwords, integrating Single Sign-On (SSO) using enterprise identity providers like Azure Active Directory (Azure AD) ensures that authentication is secure, scalable, and aligned with your organization’s identity governance policies.

In this article, we’ll walk through how to configure Argo CD to authenticate users via Azure AD using OpenID Connect (OIDC), enabling seamless and secure access through the web UI. By the end of this guide, you'll have a setup that supports workload identity, group-based role access, and a flexible RBAC model that maps Azure AD groups to Argo CD roles — allowing you to manage access across multiple teams efficiently and securely.

Whether you're a platform engineer, SRE, or DevOps lead looking to enforce identity-based access control across your GitOps workflow, this guide will give you a production-ready foundation.

🔧 Prerequisites

Before configuring Argo CD with Azure Active Directory (AAD) SSO via the web UI, ensure the following are in place:

✅ Azure Subscription: Active subscription with Contributor (or higher) access to manage AKS, AAD, and networking. Sign up.

🏗️ Private AKS Cluster: Deployed via Terraform Cloud, with user-assigned managed identity and running behind firewall with private enspoints. Accessible via kubectl in the jumpbox within the nework.

🛡️ Istio Internal Ingress: Istio installed with an internal ingress gateway to restrict public access and allow access within the network or via a jumpbox/VPN.

🛠️ Azure CLI: Version 2.30+, authenticated with az login. Used for managing Azure AD apps and AKS.

🖥️ kubectl: Configured to access the AKS cluster. Test with kubectl get nodes.

📦 Helm 3.x: Required to deploy Argo CD, Istio, and cert-manager.

🔐 Workload Identity: Enabled for secretless authentication to Azure resources from Kubernetes workloads.

🔒 Cert-Manager (optional): For automated TLS via Let’s Encrypt or internal CA. Cert-Manager Docs. but i will use openssl because let’sencrypt does not authenticate ACME challenge in internal private dns zone

🌐 Domain & Private DNS zone : And point private dns name to the internal AKS ISTIO ingress IP (e.g., https://argocd.terranetesprivate.com).

🔑 App registration: used to configure AAD SSO/OIDC securely in argocd.

📚 Knowledge Required:

• Basic understanding of Istio Gateway, VirtualService, and AAD SSO/OIDC

• Familiarity with Kubernetes RBAC, TLS, and DNS setup

⚠️ Ensure all tools are installed and your Terraform state reflects the expected AKS and networking configuration.

kubectl get pod -A | grep -E 'istio|cert|wi-webhook'
kubectl get svc -A | grep -E 'istio|cert|wi-webhook'

Configure a new Entra ID App registration

Add a new Entra ID App registration¶

1. From the Microsoft Entra ID > App registrations menu, choose + New registration

2. Enter a Name for the application (e.g. ArgoCD-Terranetes-SSO).

3. Specify who can use the application (e.g. Accounts in this organizational directory only).

4. Enter Redirect URI (optional) as follows (replacing http://argocd.terranetesprivate.com:8085/auth/callback with your Argo URL), then choose Add.

   • Platform: Web

   • Redirect URI: https://argocd.terranetesprivate.com/auth/callback

5. When registration finishes, the Azure portal displays the app registration's Overview pane. You see the Application (client) ID.

Configure additional platform settings for ArgoCD CLI¶

1. In the Azure portal, in App registrations, select your application.

2. Under Manage, select Authentication.

3. Under Platform configurations, select Add a platform.

4. Under Configure platforms, select the "Mobile and desktop applications" tile. Use the below value. You shouldn't change it.

   • Redirect URI: http://localhost:8085/auth/callback

Add credentials a new Entra ID App registration¶

Using Workload Identity Federation (Recommended)

1. Label the Pods: Add the azure.workload.identity/use: "true" label to the argocd-server pods.

2. Add Annotation to Service Account: Add azure.workload.identity/client-id: "$CLIENT_ID" annotation to the argocd-server service account using the details from application created in previous step.

3. From the Certificates & secrets menu, navigate to Federated credentials, then choose + Add credential

4. Choose Federated credential scenario as Kubernetes Accessing Azure resources

5. Enter Cluster Issuer URL, refer to retrieve the OIDC issuer URL documentation

6. Enter namespace as the namespace where the argocd is deployed

7. Enter service account name as argocd-server

8. Enter a unique name

9. Click Add.

I updated the Workload Identity Federation for the App registration with TERRAFORM

Setup permissions for Entra ID Application¶

1. From the API permissions menu, choose + Add a permission

2. Find User.Read permission (under Microsoft Graph) and grant it to the created application:

From the Token Configuration menu, choose + Add groups claim

Associate an Entra ID group to your Entra ID App registration¶

1. From the Microsoft Entra ID > Enterprise applications menu, search the App that you created (e.g. ArgoCD-Terranetes-SSO).

   • An Enterprise application with the same name of the Entra ID App registration is created when you add a new Entra ID App registration.

2. From the Users and groups menu of the app, add any users or groups requiring access to the service.

   mine is terranetes-group

Configure and deploy Argocd to use the new Entra ID App registration with helm

let’s create variables

## create variables for the tenant_id and client_id and object_id for terranetes-group
export TENANT_ID="TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT"
export APP_REGISTRATION_CLIENT_ID="AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
export TERRANETES_GROUP_OBJECT_ID="GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG"

### create argo-cd-values.yaml file with cat command
cat > argo-cd-values.yaml << EOF
## Custom resource configuration
crds:
  install: true
  keep: true
  annotations: {}
  additionalLabels: {}

global:
  podAnnotations:
    azure.workload.identity/use: "true"   
  podLabels:
    azure.workload.identity/use: "true"

controller:
  serviceAccount:
    create: true
    name: argocd-application-controller
    annotations:
      azure.workload.identity/client-id: ${APP_REGISTRATION_CLIENT_ID} # Required for workload identity
    labels:
      azure.workload.identity/use: "true" # Required if using the webhook

server:
  serviceAccount:
    create: true
    name: argocd-server
    annotations:
      azure.workload.identity/client-id: ${APP_REGISTRATION_CLIENT_ID} # Required for workload identity
    labels:
      azure.workload.identity/use: "true" # Required if using the webhook

## Argo CD Configs
configs:
  cm:
    create: true

    # Enable local admin user
    admin.enabled: true

    ## ArgoCD URL 
    url: https://argocd.terranetesprivate.com

    # OIDC configuration
    oidc.config: |
      name: AzureAD
      issuer: https://login.microsoftonline.com/${TENANT_ID}/v2.0
      clientID: ${APP_REGISTRATION_CLIENT_ID} # Required for workload identity
      redirectURIs:
        - https://argocd.terranetesprivate.com:8085/auth/callback
        - https://argocd.terranetesprivate.com/auth/callback

      azure:
        useWorkloadIdentity: true
      requestedIDTokenClaims:
        groups:
          essential: true
          value: "SecurityGroup"
      requestedScopes:
        - openid
        - profile
        - email

  params:
    server.insecure: true

  # Add RBAC configuration to map Azure AD groups to ArgoCD roles
  rbac:
    # Ensure this is set to true if you are defining RBAC policies here
    create: true 
    policy.default: role:readonly
    policy.csv: |
      # Platform Admin Policies
      p, role:org-admin, applications, *, */*, allow
      p, role:org-admin, clusters, get, *, allow
      p, role:org-admin, projects, *, *, allow
      p, role:org-admin, repositories, *, *, allow
      p, role:org-admin, certificates, *, *, allow
      p, role:org-admin, accounts, *, *, allow
      p, role:org-admin, gpgkeys, *, *, allow # Allow GPG key management for admins

      # Azure AD Group Mapping for terranetes-group (Object ID: GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG)
      # The OIDC connector name 'AzureAD' must match 'name' in oidc.config.
      g, AzureAD:${TERRANETES_GROUP_OBJECT_ID}, role:org-admin
    scopes: "[groups, email]" # These scopes are required for RBAC to function correctly
    userInfoGroupsField: memberOf

EOF

Deploy Argocd with helm

## install the argo-cd
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  -f argo-cd-values.yaml \
  --debug

sleep 180 # wait for 3 mins

When you check the pods in argocd namespace, only 1 container will be running on each component.

kubectl get pods -n argocd

But after labelling the argocd namespace with aks istio envoy proxy injection and restart the namespace pods, each pod will be running with two containers (sidecar and main container).

kubectl label namespace argocd istio.io/rev=asm-1-24 --overwrite

## Verify the argocd namespace labels
kubectl get namespace argocd --show-labels

## restart the pods
kubectl rollout restart deployment -n argocd
kubectl delete pod argocd-application-controller-0 -n argocd

To add the callback url it requires https url . In that case, i need to create an tls certificate

I want to use cert-manager with let’sencrypt to automate the certificate issuing and certificate renewal.

But let’sencrypt does not validate ACME challenge in private dns zone, so i decided to use OPENSSL certificate for the POC. But i production i will advise to use private certificate provider like Digicert CA.

# Create directory in your home folder instead of root
mkdir -p ~/argocd-tls

# Generate self-signed certificate
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout ~/argocd-tls/argocd.key \
  -out ~/argocd-tls/argocd.crt \
  -days 365 \
  -subj "/CN=argocd.terranetesprivate.com/O=argocd.terranetesprivate.com"

Create a Kubernetes Secret for the Self-Signed Certificate in the aks-istio-ingress namespace

# Create the secret using the correct path to your files
kubectl create secret tls argocd-tls-secret \
  --cert=./argocd-tls/argocd.crt \
  --key=./argocd-tls/argocd.key \
  -n aks-istio-ingress

## Verify the secret
kubectl get secret argocd-tls-secret -n aks-istio-ingress

Configure AKS Istio Gateway and Virtualservice: in aks-istio-ingress namespace where Istio Ingress Gateway is installed / cert tls secret is created.

echo "INSTALLING ISTIO INGRESS GATEWAY FOR INTERNAL TRAFFIC (private IP)"
cat <<EOF | kubectl apply -f -
# argocd-gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: argocd-internal-gateway-tls
  namespace: aks-istio-ingress                        # Namespace for Istio Ingress Gateway
spec:
  selector:
    istio: aks-istio-ingressgateway-internal  # Selects the internal gateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "argocd.terranetesprivate.com"
    tls:
      httpsRedirect: true    
  - port: 
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: argocd-tls-secret    # Name of the TLS secret created by cert-manager or kubectl and openssl
    hosts:
    - "argocd.terranetesprivate.com"
---
# argocd-virtualservice.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: argocd-virtualservice
  namespace: argocd
spec:
  hosts:
  - "argocd.terranetesprivate.com"
  gateways:
  - aks-istio-ingress/argocd-internal-gateway-tls
  http:
  - route:
    - destination:
        host: argocd-server.argocd.svc.cluster.local
        port:
          number: 80      
EOF

BINGO! 🚀

✅ Conclusion: Elevating Security and Efficiency with Azure AD SSO for Argo CD

Integrating Azure Active Directory (AAD) Single Sign-On (SSO) with Argo CD is not just a technical enhancement—it's a strategic upgrade for any organization embracing GitOps at scale. This integration bridges the gap between secure identity management and seamless developer experience, ensuring that access to your deployment pipelines is both tightly controlled and frictionless.

By aligning Argo CD authentication with your enterprise identity provider, you gain the ability to enforce consistent access policies, reduce operational overhead, and meet compliance requirements—all while empowering your teams to move faster and more securely.

**🔍 Why Azure AD SSO Is a Necessity—**Not Optional

🔐 Strengthened Security Posture
SSO centralizes authentication through Azure AD, enabling advanced security features like Multi-Factor Authentication (MFA), Conditional Access, and Identity Protection. This drastically reduces the risk of credential sprawl and unauthorized access, especially in environments where Argo CD manages critical deployment workflows.

🚀 Seamless Developer Experience
With SSO, developers and operators can log in using their existing corporate credentials—no need to manage or remember separate usernames and passwords. This reduces login friction, minimizes password fatigue, and allows teams to focus on delivering value rather than managing access.

🧩 Simplified Identity Lifecycle Management
As teams grow and change, managing user access manually becomes error-prone and inefficient. Azure AD integration ensures that onboarding, offboarding, and role transitions are handled centrally. When a user leaves the organization or changes roles, their access to Argo CD is automatically updated or revoked—no manual cleanup required.

📊 Centralized Auditing and Compliance
All authentication events are logged in Azure AD, providing a single source of truth for access control. This is essential for meeting regulatory requirements, conducting security audits, and maintaining visibility into who accessed what, when, and from where.

🔧 Foundation for Granular Access Control
This SSO setup lays the groundwork for implementing fine-grained access control in Argo CD using AppProjects. By mapping Azure AD groups to specific roles and projects, you can enforce least-privilege access across teams and environments—ensuring that users only interact with the resources they’re authorized to manage.

Follow me on Linkedin George Ezejiofor to stay updated on cloud-native observability insights! 😊

Happy deploying! 🚀🎉

🌟 In today’s cloud-native environments, Canary Deployment stands out as a powerful technique for achieving zero-downtime releases. By incrementally rolling out new application versions, canary deployments reduce risk and ensure a seamless user experience. When combined with the advanced rollout strategies of Argo Rollouts and the fine-grained traffic management of Istio Service Mesh, you gain precise control over traffic shifts, enabling smooth transitions between versions. This project also has Automatic Rollback capability for the stable version when canary version deployment goes wrong.

🎯 This guide will show you how to implement Canary Deployment using Argo Rollouts’ intelligent strategies alongside Istio’s traffic-splitting capabilities. You’ll learn how to gradually shift traffic between application versions while maintaining full observability and control. By the end of this guide, you’ll have a robust, production-ready setup that deploys new features seamlessly—without impacting your end users.

✅ Prerequisites 🛠️

To successfully implement Zero Downtime Canary Deployment with Argo-Rollouts and Istio Service-Mesh, ensure you have the following:

• 🐳 Kubernetes Cluster: A working Kubernetes cluster set up using KUBEADM on a bare-metal setup, with MetalLB configured for LoadBalancer functionality.

• 💻 kubectl: Install and configure the Kubernetes command-line tool to interact with your cluster.

• 🧩 Helm: The Kubernetes package manager for simplified application deployment and configuration.

• 🔒 Cert-Manager (optional): Installed in the cluster for automated TLS certificate management.

• 🌐 Istio Ingress Controller: Deploy the Istio Ingress Gateway to handle HTTP(S) traffic routing effectively.

• 📂 Namespace Configuration: Create distinct namespaces or use labels to separate stable and canary deployments for clear isolation.

• 🌐 Domain Name: Set up a domain (e.g., terranetes.co.uk) or a subdomains pointing to your LoadBalancer IP address. You can manage DNS using providers like Cloudflare.

• 📧 Let's Encrypt Account: Ready with a valid email address for certificate issuance to enable HTTPS.

• 📡 MetalLB: Configured for bare-metal Kubernetes clusters to manage LoadBalancer services.

• 📈 Kiali: Installed for monitoring Istio's traffic flow and gaining visibility into service dependencies and metrics.

• 📦 Argo Rollouts: Installed to handle advanced Canary Deployment strategies. Use the following

    kubectl create namespace argo-rollouts
    kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
  

• 📡 Basic Networking Knowledge: Familiarity with Kubernetes networking concepts like Ingress, Services, and LoadBalancer mechanisms.

kubectl get pod -A | grep -E 'argo|istio|metal|cert|monitoring'

With these prerequisites ready, you're equipped to dive into setting up Canary Deployments! 🚀

Architecture 📈

Deployments 🚀

You can get certificate deployment from my BlueGreen deployment article

Argo-rollouts deployment 💻

Deploy Clusterissuer with the same method as Bluegreen Deployment.
Deploy certificate for istio-ingress

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: terranetes-istio-cert
  namespace: istio-ingress                    # Must match the namespace of the Istio ingress gateway
spec:
  secretName: terranetes-istio-tls            # This is the credentialName in the Gateway resource
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 4096
  issuerRef:
    name: letsencrypt-dns01-istio
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
    - "grafana.terranetes.co.uk"
    - "prometheus.terranetes.co.uk"
    - "alertmanager.terranetes.co.uk"
    - "kiali.terranetes.co.uk"
    - "argo-rollout.terranetes.co.uk"
EOFCertificate Issued ✅

Certificate Issued ✅

Deploy argo rollout namespace (canary)🏠

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: canary
  labels:
    istio-injection: enabled
EOF

Deploy argo rollout Gateway

cat << EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: terranates-app-gateway
  namespace: istio-ingress
spec:
  selector:
    istio: ingress       # use istio default controller
  servers:

  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true  # Redirect HTTP to HTTPS

  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: terranetes-istio-tls  # Reference the TLS secret
    hosts:
    - "argo-rollout.terranetes.co.uk"    
EOF

Deploy argo rollout Services

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: terranates-app-canary
  namespace: canary
  labels:
    app: terranates-app  # Add this label  
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: terranates-app
    # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:
    # rollouts-pod-template-hash: 7bf84f9696

---
apiVersion: v1
kind: Service
metadata:
  name: terranates-app-stable
  namespace: canary
  labels:
    app: terranates-app  # Add this label  
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: terranates-app
    # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:
    # rollouts-pod-template-hash: 789746c88d
EOF

Deploy argo rollout VirtualServices

cat << EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: terranates-app-vs1
  namespace: canary
spec:
  gateways:
  - istio-ingress/terranates-app-gateway
  hosts:
  - "argo-rollout.terranetes.co.uk"
  http:
  - name: route-one
    route:
    - destination:
        host: terranates-app-stable
        port:
          number: 80
      weight: 100
    - destination:
        host: terranates-app-canary
        port:
          number: 80
      weight: 0
EOF

Deploy argo rollout Terranetes webapp

cat << EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: terranates-app
  namespace: canary
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: terranates-app-canary
      stableService: terranates-app-stable
      analysis:
        startingStep: 2
        templates:
        - templateName: istio-success-rate
        args:
        - name: service 
          value: terranates-app-canary  # ✅ Canary service name
        - name: namespace
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      trafficRouting:
        istio:
          virtualServices:
          - name: terranates-app-vs1
            routes:
            - route-one
      steps:
      - setWeight: 10
      - pause: {}
      - setWeight: 20
      - pause: {duration: 30s}
      - setWeight: 30
      - pause: {duration: 30s}
      - setWeight: 40
      - pause: {duration: 30s}
      - setWeight: 50
      - pause: {duration: 30s}
      - setWeight: 60
      - pause: {duration: 30s}
      - setWeight: 70
      - pause: {duration: 30s}
      - setWeight: 80
      - pause: {duration: 30s}
      - setWeight: 90
      - pause: {duration: 30s}
      - setWeight: 100
  selector:
    matchLabels:
      app: terranates-app
  template:
    metadata:
      labels:
        app: terranates-app
    spec:
      containers:
      - name: terranates-app
        image: georgeezejiofor/argo-rollout:yellow
        ports:
        - name: http
          containerPort: 8080
EOF

Deploy argo rollout AnalysisTemplate

cat << EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: istio-success-rate
  namespace: canary
spec:
  args:
  - name: service
  - name: namespace
  metrics:
  - name: success-rate
    interval: 10s
    successCondition: result[0] < 0.2 or result[1] < 10  # ✅ Handle low traffic
    failureCondition: result[0] >= 0.2
    failureLimit: 3
    provider:
      prometheus:
        address: http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
        query: >+
          (
            sum(irate(istio_requests_total{
              reporter="source",
              destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local",
              response_code!~"2.*"}[2m])
            )
            /
            sum(irate(istio_requests_total{
              reporter="source",
              destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local"}[2m])
            )
          )
          # Add total requests as second metric
          , 
          sum(irate(istio_requests_total{
            reporter="source",
            destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local"}[2m])
          )
EOF

Validate deployments in “canary” namespace

kubectl get all -n canary
# All these should show resources after successful deployment
kubectl get svc -n canary --show-labels
kubectl get rollout -n canary
kubectl get gateway -n canary
kubectl get virtualservice -n canary

Generate Traffic (Essential!): using hey command for macOS users

hey -z 5m -q 10 https://argo-rollout.terranetes.co.uk

Set new Green image for argo rollout

kubectl argo rollouts set image terranates-app terranates-app=georgeezejiofor/argo-rollout:green -n canary

Set new Red image for argo rollout

kubectl argo rollouts set image terranates-app terranates-app=georgeezejiofor/argo-rollout:red -n canary

Set new Blue image for argo rollout

kubectl argo rollouts set image terranates-app terranates-app=georgeezejiofor/argo-rollout:blue -n canary

Set new Yellow image for argo rollout

kubectl argo rollouts set image terranates-app terranates-app=georgeezejiofor/argo-rollout:yellow -n canary

Set new Purple image for argo rollout

kubectl argo rollouts set image terranates-app terranates-app=georgeezejiofor/argo-rollout:purple -n canary

Watch argo-rollouts

kubectl argo rollouts get rollout terranates-app -n canary -w

Visual Testing for Rollouts and Automatic Rollback 😊

Conclusion 🎉

Congratulations! You’ve just unlocked the power of zero-downtime deployments with Argo Rollouts and Istio! 🚀 By combining Argo Rollouts’ intelligent canary strategies with Istio’s granular traffic management, you’ve built a robust system that:

• Reduces Risk 😌: Gradually shift traffic to new versions while monitoring real-time metrics.

• Ensures Smooth User Experience 🌟: No downtime, no disruptions—just seamless updates.

• Automates Rollbacks 🛡️: Detect issues early and revert to stable versions effortlessly.

• Optimizes Traffic Control 🎛️: Istio’s dynamic routing ensures precise traffic splitting.

With this setup, you’re not just deploying code—you’re delivering confidence. 💪 Whether you’re rolling out mission-critical features or experimenting with new updates, this integration empowers you to innovate fearlessly.

Next Project: Observability Stacks 📈

Now that you've mastered canary deployments, it's time to build a powerful observability stack for deeper insights into your applications! 🚀 In this next project, we'll explore tools that provide real-time monitoring, centralized logging, and distributed tracing to help you maintain a reliable and performant system.

Observability Tools We’ll Cover 🛠️

Dive into building a powerful observability stack for deeper insights! We'll explore tools like Prometheus, Grafana, Loki, Jaeger, OpenTelemetry, Kiali, and the promtail for real-time monitoring, logging, and tracing. 🛠️

Stay tuned for hands-on implementations and best practices! 🎯This stack will help you monitor, troubleshoot, and optimize your applications with full visibility into system behavior.

Stay tuned as we explore hands-on implementations and best practices! 🎯

Follow me on Linkedin George Ezejiofor to stay updated on cloud-native observability insights! 😊

Happy deploying! 🚀🎉

Are you overwhelmed by the complexity of deploying production-grade multi-node Kubernetes clusters? 😅 The countless configurations, the manual setup, and the inevitable errors—it’s enough to make anyone feel like giving up! But what if I told you there’s a way to automate the entire process with just one command? 🤯 Just like the way you do terraform apply -auto-approve and terraform destroy -auto-approve .

Imagine deploying a robust, scalable, and production-ready Kubernetes cluster as effortlessly as running a simple script. ✨ With the power of Kubespray and Multipass, I’ve developed an automated solution that does exactly that! 🚀

Here’s how it works:

• Cluster Deployment: A single command, python3 terranetes.py cluster initprovisions a multi-node Kubernetes cluster using Kubespray on Multipass. No manual intervention is needed! 🛠️

• Cluster Destruction: When you're done, simply run python3 terranetes.py cluster terminate and it gracefully destroys the cluster, cleans up resources, and even purges the Multipass instances. 💣✨

• Fresh Start, Need a new deployment with new IPs from the DHCP? No problem! Run the command again, and voilà, your new Kubernetes cluster is deployed and ready to go. Each new deployment gets fresh IPs dynamically assigned from the DHCP, ensuring no conflicts and a seamless experience. 🌐

This approach combines the best of both worlds: the flexibility and reliability of Kubespray with the lightweight convenience of Multipass. Whether you're setting up a home lab or experimenting with multi-node clusters for learning, this solution takes the pain out of the process—and puts the fun back in DevOps! 😎

Ready to see the magic in action? Let’s dive in! 🚀

Prerequisites

Multipass 🌐: It is essential for creating and managing lightweight virtual machines on your local machine. It’s easy to deploy and delete. Memory - 4GB, Disk - 20GB.

Python3 🐍: Python is the backbone of the automation script. Ensure you have Python3 installed.

Ansible 📦: Kubespray relies on Ansible for orchestrating Kubernetes setup across nodes.

Kubespray 🚀: Kubespray is the tool of choice for automating Kubernetes clusters.

Bash Script 📝: This script automates the commands for deploying and destroying the Kubernetes cluster using Kubespray and Multipass.

Why Multipass? 🤔

When choosing a virtualization environment for running Kubernetes nodes, there are several options available like VirtualBox, VMware, Lima, Colima, Vagrant, etc. Each has unique features, but Multipass stands out for its simplicity and efficiency in local development. Here’s why I chose Multipass for this project:

1. Lightweight and Fast ⚡

Multipass is optimized for lightweight, fast virtual machine creation and management.

• Quick Deployment: Launch VMs in seconds with minimal configuration. making it ideal for quickly spinning up and tearing down virtual machines.

• This efficiency is crucial for development and testing environments where time is of the essence. ⚡

• Low Overhead: Multipass VMs consume fewer resources compared to traditional hypervisors like VMware or VirtualBox.

2. Seamless Integration 🌐

Multipass is tightly integrated with Ubuntu, making it an excellent choice for running Kubernetes clusters on Ubuntu-based images.

• Prebuilt Ubuntu Images: Comes with official Ubuntu LTS images, eliminating the need to download or manage custom ISOs.

• Integration with Cloud-Init: Multipass supports Cloud-Init, which allows you to automate the initial configuration of your instances. This feature is particularly useful for automating tasks and ensuring consistency across deployments. 🚀

• Effortless Updates: Automatically downloads and keeps images up-to-date with the latest versions.

3. Command-Line Simplicity 🛠️

Multipass is designed with developers in mind, featuring a clean and intuitive command-line interface.

• Single Command Operations: Create, delete, and manage VMs with simple commands.

• Scripting-Friendly: Ideal for automation workflows, making it easy to integrate with tools like Ansible and Kubespray.

4. Cross-Platform Support 🌍

Multipass is compatible with major platforms, including Linux, macOS, and Windows, ensuring flexibility and portability.

• Unified Experience: Consistent CLI across all platforms.

• Cross-Platform Compatibility: Multipass runs on Windows, macOS, and Linux, providing a consistent experience across different operating systems. This cross-platform compatibility is beneficial for teams working in diverse environments. 🌐

• No Dependencies: Unlike Vagrant, which requires a hypervisor like VirtualBox, Multipass works out of the box with its backend.

5. Cloud-Like Experience ☁️

Multipass mimics the simplicity of provisioning instances in the cloud, bringing that experience to your local machine.

• Ease of Networking: Multipass simplifies networking setup, making it easier to configure and manage network interfaces and IP addresses for your virtual machines. This ease of use is beneficial for setting up and managing Kubernetes clusters. 🌍

• Automation-Friendly: Multipass can be easily integrated into automation scripts and CI/CD pipelines, making it a perfect fit for automated deployments and testing workflows. 🛠️

• Dynamic IP Assignment: Leverages DHCP for dynamic IP allocation, making networking seamless.

• Minimal Configuration: Abstracts away the complexities of VM networking and resource management.

6. Efficient Resource Usage 🖥️

Multipass is optimized for development environments and uses system resources efficiently.

• Multipass is resource-efficient, meaning it uses fewer system resources compared to heavier alternatives like VMware or VirtualBox. This efficiency is crucial for running multiple virtual machines simultaneously without significant performance degradation. 💻

• Customizable Resources: Allocate specific CPU, memory, and disk resources for each VM.

• Low Host Impact: Uses lightweight QEMU as the backend on Linux and macOS, ensuring your host system remains performant.

How Does It Compare to Other Tools?

Why I Chose Multipass for My Project? 💡

For this project, the goal was to deploy and manage multi-node Kubernetes clusters locally with minimal effort. Multipass checked all the boxes:

1. Simplicity: Easy VM provisioning without dealing with hypervisor complexities.

2. Speed: Rapid setup and teardown of VMs.

3. Integration: Works seamlessly with Ubuntu and aligns perfectly with Kubespray’s requirements.

4. Automation: Script-friendly CLI for hands-free deployment and destruction of clusters.

Why Kubespray?

When it comes to deploying Kubernetes clusters, there are numerous tools to choose from, including Kind, Minikube, k3s, MicroK8s, and kubeadm. Each has its strengths and serves specific use cases. So, why choose Kubespray for setting up a production-grade multi-node Kubernetes cluster? Let’s explore! 🚀

1. Production-Grade Setup 🌟

Kubespray is designed to create highly available, production-ready Kubernetes clusters.

• Supports Multi-Node Clusters: While tools like Minikube and Kind are excellent for single-node or small-scale environments, Kubespray shines in setting up robust multi-node configurations.

• Built-In HA: Out-of-the-box high availability (HA) ensures resilience, making it ideal for production workloads.

• With Kubespray, you can easily scale your clusters by adding or removing nodes as needed. This flexibility makes it a preferred choice for dynamic environments.

2. Highly Customizable 🎨

Kubespray provides the flexibility to customize every aspect of the cluster.

• Fine-Tuned Configuration: Modify parameters like container runtime, network plugins, and cloud providers.

• Advanced Options: Includes additional features like network policies, cluster monitoring, and logging by default.

• Support for Various CNIs: Choose from Calico, Cilium, Flannel, and more based on your networking needs.

• Kubespray offers extensive configuration options, allowing you to tailor your Kubernetes setup to specific requirements. This level of customization is often not available in simpler tools like MicroK8s.

3. Idempotent and Reliable 🔁

Kubespray leverages Ansible for orchestration, ensuring idempotency and reliability.

• Ansible Integration: Kubespray leverages Ansible for configuration management, providing a powerful and flexible way to automate complex tasks. This integration enhances the overall automation and management capabilities.

• Error Recovery: If an operation fails, rerunning the playbook picks up where it left off without breaking the cluster.

• Scalability: Easily scale your cluster by adding or removing nodes.

4. Cross-Platform Support 🌍

Unlike other tools that are tied to specific environments, Kubespray works seamlessly across various platforms.

• Cloud Provider Agnostic: Kubespray can deploy clusters across various cloud providers and on-premises environments, providing versatility that other tools might lack.

• On-Premise and Cloud: Supports deployment on on-premise bare metal, cloud VMs, or a hybrid setup.

• Multi-Provider Flexibility: Compatible with AWS, GCP, Azure, OpenStack, and more.

5. Kubernetes Compliance ✅

Kubespray adheres to upstream Kubernetes standards, ensuring compatibility with the latest versions.

• Community and Support: Kubespray has a strong and active community, offering plenty of resources, documentation, and support for troubleshooting and optimization.

• Security Features: Kubespray includes built-in security features and best practices, ensuring your clusters are secure from the start. 🔒

• No Vendor Lock-In: Fully compliant with the Kubernetes API, so you’re not locked into proprietary solutions.

• Community-Driven: Supported by the Kubernetes SIG (Special Interest Group), ensuring regular updates and robust community support.

How Does It Compare to Other Tools?

Why I Chose Kubespray for My Project? 💡

For this project, the goal was to automate the deployment of production-grade, highly available, multi-node Kubernetes clusters in a local environment and also customisable.

• Kubespray provided the perfect balance between flexibility, reliability, and ease of automation.

• Combining it with Multipass made the process lightweight and repeatable with minimal resource overhead.

Why My TerraNetes Over Kubespray? 🚀

While Kubespray is a fantastic tool for deploying production-grade Kubernetes clusters, it requires significant setup and configuration. TerraNetes, on the other hand, simplifies the entire process by automating everything with just one command. Here's a detailed comparison and why TerraNetes stands out:

1. Node Provisioning Made Simple 🛠️

Kubespray: Requires manual provisioning of nodes before you can deploy Kubernetes.

• You need to set up virtual machines or physical servers, configure SSH access, and ensure the nodes meet Kubernetes prerequisites.

• Customization can be time-consuming and prone to errors.

TerraNetes: Handles node provisioning out of the box.

• Automatically spins up and configures virtual machines using Multipass.

• No manual intervention is required to create or manage nodes.

2. Out-of-the-Box Configuration 🎁

Kubespray: Requires detailed configuration for nodes, including:

• Firewall Rules: Ensure necessary ports are open for Kubernetes components.

• Hostnames: Set unique and resolvable hostnames for all nodes.

• Swap Off: Manually disable swap to ensure Kubernetes’s stability.

• Kernel Modules: Enable necessary modules like br_netfilter with modprobe.

TerraNetes: Takes care of all these configurations automatically.

• Ensures all nodes are pre-configured with the required settings.

• No need to worry about forgetting a crucial step in node setup.

3. Ansible Connection Simplified 🔗

Kubespray: Requires SSH access and key management to establish connections between the control node and worker nodes.

• This includes creating SSH keys, distributing them to nodes, and ensuring proper permissions.

TerraNetes: Removes the need for manual SSH setup.

• Automatically configures Ansible connections during the provisioning process.

• New ssh-keys are being created on every deployment and deleted on every termination for security reasons.

• Automatically update the ansible inventory file with new node names and node IPs.

• No manual key distribution or host configuration is necessary.

4. Single Command Deployment 💻

Kubespray: Deployment involves multiple steps:

• Provision nodes, prepare them, configure Ansible inventory, and run the playbooks.

• Each step requires careful attention to detail to avoid errors.

TerraNetes: Simplifies deployment with one command.

• Handles node creation, configuration, and Kubernetes installation in a single execution.

• Saves time and ensures consistency across all environments.

5. Automation and User Experience 🎯

Kubespray: Powerful but designed for experienced users.

• Suitable for advanced scenarios where deep customization is required.

• However, it has a steep learning curve and requires significant effort to set up.

TerraNetes: Built with a focus on automation and ease of use.

• Perfect for users who want a Kubernetes cluster up and running quickly without dealing with the underlying complexities.

• Designed to minimize errors and reduce the barrier to entry for Kubernetes deployment.

6. Time-Saving Advantage ⏳

• Kubespray: Can take hours or even days to configure and deploy, depending on your expertise and environment.

• TerraNetes: Gets you a fully functional Kubernetes cluster in minutes, thanks to its streamlined approach.

Summary of Why TerraNetes Wins 🌟

When to Choose TerraNetes?

• Rapid Prototyping: Need a cluster quickly without worrying about the underlying infrastructure.

• Simplicity: Want to focus on using Kubernetes, not setting it up.

• Automation: Prefer an end-to-end automated solution with minimal manual intervention.

While Kubespray shines in highly customizable and production-grade setups, TerraNetes provides an unmatched level of simplicity and speed. With just one command, it does all the heavy lifting—node provisioning, configuration, and deployment—so you can focus on building and deploying applications. 🎉

Testing the Terranetes project

Conclusion 🎯

The TerraNetes project redefines the Kubernetes deployment experience with it’s simplicity, automation, and efficiency. It revolutionizes the way we deploy and manage production-grade multi-node Kubernetes clusters. Automating the entire process with just a single command simplifies the complexity of node provisioning, SSH configuration, updating node hostnames, firewall setup, and more. This out-of-the-box solution saves time, reduces errors, and makes Kubernetes deployment accessible to everyone, whether you're setting up a home lab or experimenting with multi-node clusters for learning.

With Terranetes, you can focus on what truly matters—building and scaling your applications—while leaving the heavy lifting of cluster management to the automation script. The combination of Kubespray and Multipass, coupled with the streamlined approach of Terranetes, ensures a robust, scalable, and hassle-free Kubernetes experience. 🌐✨

Whether you're a DevOps professional, developer, or Kubernetes enthusiast, TerraNetes empowers you to harness the full potential of Kubernetes in the fastest and easiest way possible. 🚀

Linkedin

🌟 In modern cloud-native environments, ensuring zero-downtime deployments while maintaining robust security is critical. Blue-Green Deployment is a proven strategy that allows teams to switch traffic between different versions of an application seamlessly. Combined with TLS encryption for secure communication, this approach ensures a smooth and secure user experience.
🚀 In this guide, we’ll implement a Blue-Green Deployment in Kubernetes, utilizing Cert-Manager for automated TLS certificate management and Nginx Ingress for traffic routing. By the end of this project, you’ll have a production-ready setup that you can replicate in your own environments.

Tech Stack

🔧 Kubernetes: Cluster orchestration and management.
🔒 Cert-Manager: Automated TLS certificate management.
🌐 Nginx Ingress Controller: Routing HTTP(S) traffic to your services.
📦 Helm (optional): Simplifying deployments.
🛡️ Let's Encrypt: Free TLS certificates for HTTPS.
📡 MetalLB: LoadBalancer for bare-metal Kubernetes clusters.
💻 kubectl: Command-line tool for interacting with Kubernetes.

Prerequisites

✅ Kubernetes Cluster: I will be using MICROK8S on this project on a bare-metal setup with MetalLB for LoadBalancer.
✅ kubectl: Install and configure kubectl to interact with your cluster.
✅ Helm: Install Helm, the Kubernetes package manager, for simplified application deployment and configuration.
✅ Cert-Manager: Ensure Cert-Manager is installed in the cluster for TLS certificate management.
✅ Nginx Ingress Controller: Deploy the Nginx Ingress Controller to handle HTTP(S) traffic routing.
✅ Namespace Configuration: Create separate namespaces or labels for blue and green deployments.
✅ Domain Name: Set up a domain name (or subdomain) terranetes.com pointing to your LoadBalancer IP address. I Used Cloudflare to manage my DNS .
✅ Let's Encrypt Account: Prepare for certificate issuance by having a valid email for Let's Encrypt configuration.
✅ Basic Networking Knowledge: Familiarity with Kubernetes networking concepts, including Ingress and Services.

Architecture

DEPLOYMENTS

Let’s create different namespaces for the project. And also add some environment variables for the project

export CLOUDFLARE_API_KEY="xxxx change your token xxxxxxx"
export EMAIL="changeYOURemail@gmail.com"

echo $CLOUDFLARE_API_KEY
echo $EMAIL

kubectl create namespace blue-green
kubectl create namespace cert-manager

Deploy Certificate components

• Create Cloudflare API Token Secret for cert-manager.

    # Create Cloudflare API Token Secret for cert-manager
    kubectl create secret generic cloudflare-api-token-cert-manager \
      --namespace cert-manager \
      --from-literal=api-token="$CLOUDFLARE_API_KEY"
  
    kubectl get secret cloudflare-api-token-cert-manager -n cert-manager -o yaml
  

• Create a ClusterIssuer with Cloudflare DNS-01 validation.

    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-dns01-nginx
    spec:
      acme:
        server: https://acme-v02.api.letsencrypt.org/directory
        email: $EMAIL
        privateKeySecretRef:
          name: letsencrypt-dns01-private-nginx-key
        solvers:
        - dns01:
            cloudflare:
              email: $EMAIL
              apiTokenSecretRef:
                name: cloudflare-api-token-cert-manager
                key: api-token
  

• Create a Certificate that references the ClusterIssuer letsencrypt-dns01-nginx. I want to create the certificate in blue-green namespace.

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: blue-green-nginx-cert
      namespace: blue-green
    spec:
      secretName: blue-green-nginx-tls                     # Reference the secret name on encoding secretName value
      duration: 2160h # 90 days
      renewBefore: 360h # 15 days
      isCA: false
      privateKey:
        algorithm: RSA
        encoding: PKCS1
        size: 4096
      issuerRef:
        name: letsencrypt-dns01-nginx
        kind: ClusterIssuer
        group: cert-manager.io
      dnsNames:
        - "blue.terranetes.com"
        - "green.terranetes.com"
  

Green Environment is my default live environment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: terranetes-nodegreen
  namespace: blue-green
  labels:
    app: terranetes-nodegreen
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: terranetes-nodegreen
      version: green
  template:
    metadata:
      labels:
        app: terranetes-nodegreen
        version: green
    spec:
      containers:
      - name: terranetes-nodegreen
        image: georgeezejiofor/terranetes-nodegreen:green-v1
        imagePullPolicy: Always
        ports:
        - containerPort: 3000
        env:
        - name: VERSION
          value: "green"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: terranetes-nodegreen-svc
  namespace: blue-green
spec:
  selector:
    app: terranetes-nodegreen
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000

Blue Environment is my NEW live environment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: terranetes-nodeblue
  namespace: blue-green
  labels:
    app: terranetes-nodeblue
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: terranetes-nodeblue
      version: blue
  template:
    metadata:
      labels:
        app: terranetes-nodeblue
        version: blue
    spec:
      containers:
      - name: terranetes-nodeblue
        image: georgeezejiofor/terranetes-nodeblue:blue-v1
        imagePullPolicy: Always
        ports:
        - containerPort: 3000
        env:
        - name: VERSION
          value: "blue"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
---
apiVersion: v1
kind: Service
metadata:
  name: terranetes-nodeblue-svc
  namespace: blue-green
spec:
  selector:
    app: terranetes-nodeblue
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000

This services terranetes-nodeblue-svc and terranetes-nodegreen-svc acts as a Router or Switch to different environments.

The services are also deployed as a cluster IP. Hence can only be access within the cluster. I’m going to expose the service from the Loadbalancer of my ingress Controller. Also Update the Ingress resource with TLS configuration.

Deploy Ingress resource to expose both services

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blue-green-ingress
  namespace: blue-green
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns01-nginx
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - blue.terranetes.com
        - green.terranetes.com
      secretName: blue-green-nginx-tls
  rules:
    - host: blue.terranetes.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: terranetes-nodeblue-svc  
                port:
                  number: 80
    - host: green.terranetes.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: terranetes-nodegreen-svc  
                port:
                  number: 80
---

Initial Traffic Routing

Currently, the traffic is split between blue.terranetes.com (blue app) and green.terranetes.com (green app). This setup routes users based on the hostname. Now let’s switching traffic from green environment (Live) to blue environment (New). And we are doing this switching on ingress resource. This is acting as out router .

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blue-green-ingress
  namespace: blue-green
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns01-nginx
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - blue.terranetes.com
        - green.terranetes.com
      secretName: blue-green-nginx-tls
  rules:
    - host: blue.terranetes.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: terranetes-nodeblue-svc
                port:
                  number: 80
    - host: green.terranetes.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: terranetes-nodeblue-svc   # update blue svc here
                port:
                  number: 80

You will notice that even green.terranetes.com url will access (blue app).

• https://blue.terranetes.com/

• https://green.terranetes.com/

We just switch the green app users from green environment to blue environment we can easily rollback with this strategy by switching the service back to green svc on green.terranetes.com .

Rolling Back the Switch

You can roll back by modifying the Ingress again to restore the original routing:

• blue.terranetes.com -> blue app

• green.terranetes.com -> green app

Optional: Weighted Traffic Splitting

If you want to gradually switch users from green to blue (or vice versa), consider implementing canary deployment or weighted routing using tools like:

• NGINX annotations (if supported).

• Istio or Traefik for advanced traffic management.

This allows for:

• Gradual routing (e.g., 80% green, 20% blue).

• Monitoring the behavior of users before full migration.

Conclusion 🎉

Through this guide, we’ve successfully implemented a Blue-Green Deployment strategy on Kubernetes with robust TLS encryption, utilizing Cert-Manager and Nginx Ingress. This architecture ensures zero-downtime deployments, seamless traffic switching, and enhanced security, making it a reliable choice for production environments.

By switching users between green and blue environments effortlessly, we’ve demonstrated the power of dynamic traffic management. Whether it’s for releasing new features or mitigating issues with instant rollbacks, this approach minimizes risk and enhances user experience.

Additionally, the option to incorporate weighted traffic splitting or canary deployments provides further flexibility for gradual rollouts, enabling better control and monitoring during transitions.

Key Takeaways 🗝️

• Ease of Switching: ✨ Modify the Ingress resource to direct traffic instantly.

• Enhanced Security: 🔒 Automated TLS certificates ensure secure communication.

• Rollback Ready: 🔄 Revert traffic with minimal effort.

• Scalability: 📈 Extend the setup to support more complex routing patterns like weighted traffic.

With this setup, you’re equipped to deploy applications confidently, ensuring both reliability and user satisfaction. Ready to try this out in your environment? 🚀

Happy Deploying! 🌟