# Harden & Lockdown RKE2 Cluster with a 4-Layer DevSecOps Stack

## Introduction

Before you move furniture into a new house, you install the locks, the cameras, and the gates. Kubernetes is no different, yet engineers constantly build clusters inside-out, treating security as a cosmetic bolt-on to be handled "later."

Not this time. The order here is entirely deliberate. You don't move into a finished house and *then* start wondering about the security system—you wire the alarms while the building is still empty.

With my 29-node RKE2 infrastructure fully framed across three distributed Proxmox datacenters, the platform is technically alive. But it is also completely vacant. Before a single tenant workload steps foot inside—long before MLflow, Kubeflow, Tekton, or Harbor’s first private project—the security stack goes live.

When a homelab scales past a handful of nodes and begins hosting your actual identity providers, encrypted secrets engines, private registries, and core data pipelines, the blast radius of a compromise becomes real. Security transforms from a theoretical checklist into an absolute prerequisite.

This marks the definitive boundary where this project stops behaving like a handcrafted science project and starts operating like a zero-trust production platform. It also marks the moment I transition completely to a **100% GitOps workflow via ArgoCD**. The days of ad-hoc terminal manipulation and local `kubectl apply` commands are over. From here on out, every single policy configuration is a version-controlled commit.

This article documents the exact mechanics of how I wired four distinct, cloud-native security engines into a unified defensive front covering the full workload lifecycle:

### **The Four Guards**

Here's what I wired together:

| **Guard** | **Engine** | **Job** |
| --- | --- | --- |
| 🚪 The Bouncer | Kyverno | Blocks bad manifests at admission |
| 🔒 The Locks | KubeArmor | Watches syscalls at the kernel — and can block them |
| 📹 The CCTV | Falco | Detects and alerts on abnormal behavior |
| 🔍 The Inspector | Trivy | Continuously scans for CVEs and misconfigurations |

Each works independently. If one fails, the others don't notice. That's the point.

### **What Makes This Different**

This episode covers the full workload lifecycle:

*   **Before** workloads enter the API (admission control)
    
*   **While** workloads run at the kernel tier (runtime audit, enforcement on demand)
    
*   **When** workloads behave anomalously (behavioural detection)
    
*   **Long after** workloads were deployed (continuous scanning)
    

This isn’t a pristine reference diagram copied from a vendor's whitepaper. These are real, unvarnished operational lessons, architectural deadlocks, and configuration missteps encountered while making these engines work together cleanly. My hope is that sharing these traps saves you time on your own infrastructure journey.

### Where I Am in the Build

This security layer is part of a larger infrastructure series documenting the evolution of the cluster:

*   **Episode 1 — The Foundation**  
    Infrastructure provisioning with Terraform across multiple Proxmox datacenters
    
*   **Episode 2 — The Framing**  
    Building the HA RKE2 control plane using Ansible, kube-vip, and Cilium
    
*   **Episode 3 — The Security System**  
    Hardening the platform using Kyverno, KubeArmor, Falco, and Trivy Operator
    

The infrastructure exists.

Now the platform gets secured before anything important is allowed to run inside it.

The repository layout for this stack lives inside `kubernetes-addons/security-stack/` with an intentionally clean directory structure:

```yaml
security-stack/
├── README.md                                   ← Walkthrough + disaster recovery runbook
│
├── kyverno/                                    ← The Bouncer: admission control
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── kyverno-kubearmor-falco-trivy-values.yaml
│   └── policies/                               ← shipped as its OWN ArgoCD app
│       ├── kustomization.yaml
│       ├── charts/kyverno-policies-3.8.0/      ← vendored upstream PSS baseline
│       ├── security-policies.yaml              ← my custom hardening rules
│       ├── compliance-policies.yaml
│       ├── kubeflow-user-namespace-governance.yaml
│       ├── gpu-node-scheduling.yaml
│       ├── job-ttl-after-finished.yaml
│       ├── cleanup-finished-jobs.yaml
│       ├── cleanup-stale-machineconfig-jobs.yaml
│       └── rook-csi-apparmor-unconfined.yaml   ← the Ceph CSI scar, as code
│
├── kubearmor/                                  ← The Locks: runtime LSM (audit today)
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── kubearmor-falco-trivy-values.yaml
│   ├── example-policies.yaml
│   ├── webhook-skip-terminating-hook.yaml      ← the wedged-pod workaround
│   └── charts/kubearmor-operator-v1.6.18/      ← vendored operator chart
│
├── falco/                                      ← The CCTV: runtime detection
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── falco-trivy-values.yaml
│   ├── custom-rules.yaml                       ← scoped exceptions live here
│   └── servicemonitor.yaml
│
└── trivy/                                      ← The Inspector: continuous scanning
    ├── kustomization.yaml
    ├── namespace.yaml
    ├── trivy-values.yaml
    └── servicemonitor.yaml
```

Two things in that tree are worth pointing at before I go further, because both are scars rather than design: `kubearmor/webhook-skip-terminating-hook.yaml` and `kyverno/policies/rook-csi-apparmor-unconfined.yaml` only exist because KubeArmor's AppArmor annotations wedged terminating pods and confined my Ceph CSI plugins. I get to those later. Each stack also carries its own `README.md`.

Five ArgoCD child Applications, one single master ApplicationSet, and two distinct sync waves. The structural walls of my cluster haven't shifted an inch since Episode 2; what is about to change is what I allow to get inside, and what gets flagged the second it steps out of line.

## The Threat Surface

Let’s be completely transparent: a modern homelab is no longer a casual testing environment where safety defaults can be skipped. This 29-node environment hosts `HashiCorp Vault instances, enterprise Keycloak identity providers, MLflow registries, Kubeflow modeling pipelines, and private Harbor registries`. Every single one of these software components holds explicit corporate identities, cryptographic keys, machine learning models, or proprietary container images.

The blast radius of a single container escape or internal compromise is incredibly high. More importantly, this lab is my production testing sandbox; if I cut corners on security configurations here, I build bad architectural habits that will inevitably leak into the enterprise architectures I design for clients.

> The cluster threat vectors break down into four critical layers:

**Misconfigured Manifests (The Illegal Entry):** Workloads containing unsafe specifications that should never be written to etcd. Think of a deployment requesting `runAsUser: 0`, a pod requesting host namespaces `hostNetwork: true`, images pointing to mutable tags like `:latest`, or a generic network policy leaving wide-open public egress. These should be caught and stopped dead at the API gateway during kubectl apply.

**Malicious Post-Admission Execution (The Broken Lock):** Pods that legitimately pass validation checks but once scheduled onto a node, attempt to violate boundaries—such as writing directly to `/etc/shadow`, attempting to run an unexpected `exec /bin/sh` inside a production database pod, or mapping block devices. The local Linux kernel needs to refuse these operations natively.

**Subtle Runtime Anomalies (The Stealth Intruder):** Exploits that bypass standard rule boundaries entirely. `Fileless malware executions`, `unauthorized sudo executions` `inside an active application container,` or an `application process spawning unexpected binary payloads`. These system calls must be observed, parsed, and logged instantly.

**The Zero-Day Decay (The Expired Clearance):** Images that are fully verified, pristine, and compliant when pushed to the cluster on Friday, but become `highly vulnerable on Monday morning because a new critical CVE was disclosed over the weekend`. The cluster must continuously audit its own passive inventory.

No single security engine can mitigate this entire lifecycle. That is the core tenet of the defense-in-depth model: I install four distinct guards that share zero dependency lines, hook into entirely different layers of the OS and Kubernetes lifecycle, and run side-by-side without interference.

## The Four Guards

Architecture Here is the exact layout of how my security tooling intercepts threat vectors across the entire deployment and runtime lifecycle:

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/32650262-8504-42a7-a5b5-00085faf597c.png align="center")

The dotted line is deliberate. Everything drawn with a solid arrow is running in my cluster today; the alert fan-out is one chart flag away and is the one piece I have not turned on yet. I would rather draw the gap than pretend the box is filled.

By segregating these duties, I protect against systemic failure. If the admission webhook is temporarily bypassed or down during a system upgrade, the kernel layer is still watching — and, once policies land, still enforcing. If the kernel layer has no rule for a specific zero-day vector, the behavioral CCTV logging system still records and exposes the activity.

**The Bouncer at the Door — Kyverno :**  
Kyverno runs as my primary admission controller. Every single API interaction targeting my cluster passes directly through its webhook filters before the specification can be saved to etcd. If a defined policy evaluates to a failure, the request is outright denied. If a policy specifies an auto-remediation rule, the specification is seamlessly mutated in flight.

My architecture runs Kyverno Chart `3.8.0` isolated completely within the `kyverno` namespace, consisting of five core deployments:

`kyverno-admission-controller` — The primary engine processing real-time validation and mutation webhooks.

`kyverno-background-controller` — Manages asynchronous generate policy reconciliation loops.

`kyverno-cleanup-controller` — Garbage cycles dead PolicyReport and EphemeralReport metrics.

`kyverno-reports-controller` — Aggregates policy data cluster-wide.

`kyverno-policy-reporter` — The reporting visualization plane (disabled in my default values to preserve RAM).

While clean on paper, syncing this setup via GitOps under ArgoCD presented a massive hurdle. Out of the box, the `3.8.0` chart immediately breaks synchronization pipelines due to mutating resource values. To establish an entirely hands-off automated sync loop, I had to introduce four specific structural fixes directly into the ArgoCD configuration:

```yaml
# Inside my master ApplicationSet / Application definitions:
spec:
  ignoreDifferences:
    # 1. Neutralize the Kustomize CRD naming hash mutation
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      jsonPointers:
        - /metadata/annotations/kustomize.config.k8s.io~1needs-hash
    # 2. Prevent permanent diffs on ClusterPolicies defaulting validation modes
    - group: kyverno.io
      kind: ClusterPolicy
      jsonPointers:
        - /spec/validationFailureAction
    # 3. Strip dynamic API mutations from webhook manifests
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      jsonPointers:
        - /spec/conversion/webhook/clientConfig/caBundle
```

Additionally, I forced `forceFailurePolicyIgnore: true` across all helm configurations. This ensures that during a major cluster upgrade or node restart where the Kyverno pods are initializing, the Kubernetes control plane switches to an `Ignore` mode rather than locking up the API server entirely.

One major day-two operational trap to memorize (`feedback_kyverno_chart_upgrade_job_leftovers`): during every chart upgrade, a temporary hook job named `Job/kyverno-migrate-resources` is run alongside associated service accounts and cluster roles. ArgoCD marks these resources as unmanaged orphans (`sync=None requiresPruning=True`). You can manually execute a safe `kubectl delete` once you confirm the job completed its task; however, **never trigger a blind global** `argocd sync --prune`, as it will inadvertently purge active sibling metadata from unassociated applications.

Here is the Bouncer's actual roster, and what happens to a walk-in who ignores the dress code:

```yaml
kubectl get clusterpolicy
kubectl run nginx --image=nginx
```

📸 **SCREENSHOT:** the output of both commands — 36 `ClusterPolicies` all `READY=True`, then the plain `kubectl run nginx` bounced before it ever reaches a node.

Worth reading that second command's output carefully, because it shows **two guards, not one**:

```yaml
Error from server (Forbidden): pods "nginx" is forbidden:
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
runAsNonRoot != true ... seccompProfile ...
```

That rejection says `PodSecurity`, not `Kyverno` — it is Kubernetes' **native Pod Security Admission**, the free floor built into the API server, catching the most naive possible mistake before Kyverno is even consulted. The 36 ClusterPolicies sitting above it are what PSA *can't* do: mutate a spec in flight, generate a NetworkPolicy into every new namespace, verify a cosign signature, pin GPU pods to GPU nodes.

I keep both deliberately, and the layering is the point — the cheapest guard catches the cheapest mistake. A plain `kubectl run` is exactly the kind of thing a tired operator does at 2am, and it never gets far enough to need the expensive machinery.

* * *

**The Locks on Every Door — KubeArmor :**

Kyverno decides *who gets in*. KubeArmor governs *what they can do once they're inside*. This is the distinction most homelabs miss: a pod can pass every admission check — correct `securityContext`, no host namespaces, a pinned image digest — and still, the moment it's running, try to read `/etc/shadow`, spawn `/bin/sh` inside a production database, or map a block device. Admission control is blind to all of it, because admission already happened.

KubeArmor closes that gap at the only place that can't be argued with: **the Linux kernel itself**. It enforces per-pod policies through the kernel's Linux Security Modules (AppArmor + BPF-LSM), so a blocked action isn't *reported* — it's *refused*, natively, before the syscall completes. That is the capability. What I actually have turned on is a step short of it, and I'll show you exactly where I stopped.

My architecture runs the **KubeArmor operator v1.6.18** in the `kubearmor` namespace, which stands up three pieces:

`kubearmor` (DaemonSet) — the per-node enforcement engine that talks to the kernel LSM.

`kubearmor-relay` — aggregates the realtime block/audit event stream from every node into one place.

`kubearmor-controller` — the admission + annotation controller that wires each pod to its AppArmor profile.

Policies are per-workload and live right next to the thing they protect — KubeArmor watches every namespace. Locking a pod down is deliberately boring:

```yaml
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: no-shell-in-my-app
  namespace: my-ns
spec:
  selector:
    matchLabels: { app: my-app }
  process:
    matchPaths:
      - path: /bin/sh
        action: Block
      - path: /bin/bash
        action: Block
```

With that applied, an attacker who gets code execution inside `my-app` reaches for a shell and the kernel hands them nothing.

**Except I haven't applied it.** Here is the honest state of my cluster, and it is the most useful thing in this section:

```yaml
kubectl get kubearmorpolicy -A
No resources found
```

Zero policies. Not one. And the operator's default posture backs that up:

```yaml
kubectl -n kubearmor get kubearmorconfig -o yaml | grep Posture
  defaultCapabilitiesPosture: audit
  defaultFilePosture: audit
  defaultNetworkPosture: audit
```

So what *is* KubeArmor doing on my 29 nodes right now? **Watching, in enormous detail.** The relay aggregates a live stream of every process exec, file access and network connection, resolved to pod and namespace:

```bash
kubectl -n kubearmor logs -l kubearmor-app=kubearmor-relay -f
```

📸 **SCREENSHOT:** the `kubearmor-relay` stream — a live feed of resolved connections (`remotehost=pod/harbor/harbor-nginx-...`, `svc/rook-ceph/rook-ceph-rgw-...`), alongside `kubectl get kubearmorpolicy -A` returning **No resources found**. Visibility without enforcement, which is exactly what audit posture looks like.

That gap is deliberate, and it is the correct order of operations. **You do not deploy an LSM in enforce mode on a cluster you have not first watched.** Audit posture tells you what normal looks like — which binaries actually exec, which paths actually get written, which pods actually talk to which — and *that* is what a least-privilege policy is written from. Ship `action: Block` on day one and you will spend your week explaining to yourself why CNPG can't start.

I know this because KubeArmor has already blocked things on this cluster — just not attackers. The baseline AppArmor profile it generates for **every** pod carries its own deny-list (`deny mount`, `deny /sys/...` writes), and that alone was enough to break two critical subsystems. Which brings me to the traps.

Here's the honest part. KubeArmor is the guard that will bite *you* before it ever bites an attacker, because it doesn't only lock down *your* workloads — by default it generates a restrictive AppArmor profile for **every** pod, including the critical infrastructure DaemonSets that keep the cluster alive. Three traps cost me real time:

**Trap 1 — KubeArmor vs. Cilium's eBPF (**`feedback_cilium_kubearmor_apparmor_block`**).** After any change that rolls a BPF-using DaemonSet, some nodes wedged with:

```plaintext
mkdir /sys/fs/bpf/tc: permission denied
```

KubeArmor's default-deny profile was refusing Cilium's own eBPF mount. And it *survives reboots* — the restrictive profile is reapplied on every pod start, so a reboot "fix" lasts exactly until the next `cilium-agent` restart. The fix is to opt the DaemonSet out with a pod-template annotation:

```yaml
spec:
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined
```

For Cilium this is now baked into the Ansible RKE2 bootstrap manifest (`rke2-cilium-config.yaml`) so a fresh install never rediscovers it. Any other BPF-touching DaemonSet (kube-vip, NFD) can need the same treatment.

**Trap 2 — the terminating-pod deadlock (**`feedback_kubearmor_apparmor_annotation_stuck_pod_k8s134`**).** On newer Kubernetes, the controller's AppArmor-annotation mutation can leave pods stuck `Terminating` indefinitely — the annotation it wants to apply races the pod's own teardown. The release valve is blunt but reliable: scale `kubearmor-controller` to `0`, let the stuck pods drain, then scale it back. A `PostSync` hook that skips terminating pods keeps it from recurring.

**Trap 3 — don't confine the storage plumbing (**`feedback_ceph_csi_rook_apparmor_rbd_module`**).** KubeArmor's profile will happily confine the Rook-Ceph CSI plugin pods, and when it does, RBD volume mounts start failing cluster-wide — a storage outage wearing a security costume. The CSI plugins get the `unconfined` treatment for the same reason Cilium does: infrastructure that manipulates the kernel directly cannot run under a default-deny LSM profile.

The lesson across all three: **KubeArmor is a scalpel, not a blanket.** Lock down *tenant* workloads aggressively; leave the infrastructure that talks to the kernel alone.

* * *

**The CCTV on the Ceiling — Falco :**

Locks stop the break-ins you *anticipated*. But security's oldest truth is that the attack you didn't write a rule for is the one that gets you. That's what the cameras are for. **Falco doesn't block anything** — it watches everything, and the second something moves that shouldn't, it shouts.

Where KubeArmor enforces a known-bad list at the kernel, Falco taps kernel **tracepoints via eBPF** and pattern-matches the entire syscall firehose against a ruleset. A fileless payload executing from memory, an unexpected outbound connection from a pod that should only ever talk to Postgres, a `sudo` inside a container that has no business having one — none of those may trip a specific KubeArmor lock, but all of them light up the CCTV.

I run the **Falco chart 8.0.2** as a DaemonSet in the `falco` namespace on the `modern_ebpf` driver — one sensor per node, no kernel module to compile. Today, detections go exactly one place: `stdout`, which means the camera feed is `kubectl logs`:

```yaml
kubectl -n falco logs -l app.kubernetes.io/name=falco -f
```

**And that is the honest weak point in this stack.** A camera nobody watches is a camera that catches nothing after the fact. `stdout` is fine while I'm actively tailing it and useless at 3am — the detection fires, scrolls past, and ages out of the log buffer. Falco is the one guard here whose value collapses without somewhere to *send* the alert.

Routing it is a chart flag, not a rebuild — Falco ships **Falcosidekick** as a subchart, so there is no second Helm install:

```yaml
# falco/falco-trivy-values.yaml
falcosidekick:
  enabled: false        # ← the honest current state
```

Flip that to `true`, add a backend (Slack, Discord, Loki, PagerDuty), and the fan-out is done. The only reason it isn't already on is that the webhook is a credential, and I am not putting one in Git — so it waits for the same treatment everything else in this homelab gets: Terraform writes it to Vault, VSO syncs it in as a `Secret`. That is the next commit on this stack, and I would rather publish the gap than describe a pipeline I have not wired.

📸 **SCREENSHOT:** a live Falco detection in `kubectl -n falco logs -f` — rule name, pod, namespace, the offending command — the CCTV catching someone mid-motion.

Two traps here, both about the sensor itself:

**The kernel floor.** The `modern_ebpf` driver needs `CAP_PERFMON` + `CAP_BPF` and a **kernel ≥ 5.8**. On a compliant node it just works; drop in an older distro and Falco `CrashLoopBackOff`s on startup with a driver-load failure. The fallback is the legacy `ebpf` driver (or the `kmod` driver, which then wants kernel headers on the node) — set via values. The homelab's Ubuntu 24.04 image is well clear of the floor, but it's the first thing to check when a *new* node's Falco pod won't come up.

**Tuning the sensitivity, not silencing the camera.** A fresh Falco install is noisy — legitimate operations trip generic rules. The wrong reflex is to delete the rule. The right one is to add a scoped *exception* in `falco/custom-rules.yaml`, commit, and push; ArgoCD reconciles within 30 seconds and **Falco hot-reloads its config with no pod restart**. You narrow the camera's blind spot deliberately, in version control — you never unplug it.

* * *

**The Inspector on Patrol — Trivy :**

The first three guards all police the *present*: what's entering, what's running, what's misbehaving right now. Trivy polices the thing none of them can see — **time**. An image can be pristine, signed, and fully compliant when it's pushed on Friday, and be a critical CVE waiting to happen by Monday morning, because a vulnerability was disclosed over the weekend and *nothing about the image changed*. The lock is fine. The camera sees nothing. The image just quietly went bad.

So the **Trivy operator (0.32.1)**, in the `trivy-system` namespace, walks the beat: it continuously re-scans the cluster's entire passive inventory and writes every finding as a Kubernetes custom resource — no database, no PVC, no UI. Six report types cover the surface:

| Report CR | What it audits |
| --- | --- |
| `VulnerabilityReport` | Image CVEs, per workload, auto-rescanned |
| `ConfigAuditReport` | Workload misconfiguration (Pod security) |
| `RbacAssessmentReport` | Over-permissioned RBAC |
| `InfraAssessmentReport` | Node / control-plane CIS checks |
| `SbomReport` | A full SBOM for every pulled image |
| `ClusterComplianceReport` | NSA / CIS Benchmark posture, cluster-wide |

Because it's all just CRs, your SIEM is `kubectl`:

```yaml
# Every workload with a CRITICAL CVE, right now
kubectl get vulnerabilityreport -A -o json | \
  jq '.items[] | select(.report.summary.criticalCount > 0) |
      {ns:.metadata.namespace, name:.metadata.name, crit:.report.summary.criticalCount}'

# Cluster-wide compliance posture — note the REAL report names
kubectl get clustercompliancereport
#   k8s-cis-1.23
#   k8s-nsa-1.0
#   k8s-pss-baseline-0.1
#   k8s-pss-restricted-0.1
kubectl describe clustercompliancereport k8s-nsa-1.0
```

📸 **SCREENSHOT:** the `jq` output above with its CRITICAL counts, and `kubectl describe clustercompliancereport k8s-nsa-1.0` — the Inspector's findings board.

And since the whole point of this stack is that I report what it actually finds rather than what I wish it found: **the worst offender on my cluster is my own registry.** Harbor's components come back at **26–33 CRITICAL** apiece — `harbor-portal` at 33, `harbor-core`, `registry` and `jobservice` at 26 each — followed by my CNPG Postgres pods at 14, Elasticsearch at 8, and `vault-transit` at 6.

That is an uncomfortable result to publish, and it is exactly why the Inspector earns its place. Harbor is the thing that *scans everything else on the way in*, and it is carrying the heaviest CVE load in the cluster. Nothing in the first three guards would ever have told me that — the manifests are valid, the syscalls are ordinary, the behaviour is normal. Only the thing that polices *time* finds it.

Worth being precise about what those numbers mean: a CRITICAL count is not a breach, and most of these are unreachable code paths in base images. The signal is the **trend and the ranking**, not the absolute number — and the ranking is telling me which upstream images to pin, rebuild, or replace first.

Pair the `SbomReport` output with a Kyverno `verifyImages` policy and the Bouncer and the Inspector start working *together*: Trivy tells you what's actually in every image, Kyverno refuses anything unsigned at the door. That's a real supply-chain story, not a checkbox.

The one trap worth naming: **private images silently fail to scan.** Trivy's `k8schain` reads `imagePullSecrets` from the workload's ServiceAccount — so anything in an auth-required registry (some of Harbor's projects) won't scan until the namespace SA carries the pull secret:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: my-ns
imagePullSecrets:
  - name: harbor-tekton-builder-creds
```

Miss it and you get the most dangerous kind of failure — a *silent* one. The reports look clean not because the images are clean, but because Trivy never got to look.

* * *

## The Honest Head-to-Head — Why These Four, and Not the Alternatives

First, the framing most security comparisons get backwards: **these four tools are not competing with each other.** "Should I use Falco or Kyverno?" is the wrong question — it's like asking whether a building needs a door lock *or* a smoke alarm. Security isn't a *tool*, it's a *pipeline*, and each guard owns a different stage of it:

*   **Build time** → **Trivy** scans the image for CVEs before it ships.
    
*   **Admission time** → **Kyverno** refuses the manifest (and cosign-verifies the signature) before it ever reaches etcd.
    
*   **Runtime, detect** → **Falco** watches syscalls and alerts on the abnormal.
    
*   **Runtime, enforce** → **KubeArmor** refuses the malicious syscall at the kernel.
    

So the real question is never "which one" — it's "for *this layer*, which tool?" Here's that answer, honestly, layer by layer:

| Layer | My pick | The alternatives | Why I picked mine |
| --- | --- | --- | --- |
| **Admission / policy-as-code** | **Kyverno** | OPA/Gatekeeper, Kubewarden, native PSA | YAML-native policies (no Rego to learn) + `mutate` + `generate` |
| **Runtime enforcement (LSM)** | **KubeArmor** | raw AppArmor/SELinux, Tetragon | K8s-native CRDs over the kernel LSM; per-workload least-privilege |
| **Runtime detection** | **Falco** | Tetragon, Tracee, Sysdig Secure | CNCF-graduated; biggest rule ecosystem; plugin model |
| **Vuln / misconfig scanning** | **Trivy** | Grype, Clair, Snyk, Docker Scout | All-in-one (CVE + IaC + secrets + SBOM); free; already in Harbor |

Where the alternatives **genuinely win** — because every one of my picks has a real rival:

*   **OPA/Gatekeeper vs Kyverno.** Gatekeeper's **Rego** is a genuine policy *language* — more expressive than Kyverno's YAML for complex cross-resource logic, and the incumbent in heavily-regulated enterprises. I chose Kyverno because I'd rather write policies in the same YAML as everything else, and because Kyverno's `generate` (auto-stamp a NetworkPolicy / ResourceQuota / alias Service into every new namespace) and `verifyImages` (cosign — the same signatures my Tekton Chains pipeline produces) are first-class. That's load-bearing in my Kubeflow multi-tenancy. If I needed Rego's raw expressiveness, Gatekeeper would win. **Kubewarden** (WASM policies in any language) is the elegant dark horse; a smaller community kept it off my cluster.
    
*   **Native Pod Security Admission** is *free and built in* — three levels, zero install. If all you need is "no privileged pods," use PSA and skip a policy engine entirely. I run *both*: PSA sets the floor, Kyverno does everything PSA can't (mutate, generate, image verification, custom logic).
    
*   **Tetragon vs Falco (and KubeArmor).** This is the sharpest call in the stack, and the most honest thing I can say is: **I run Cilium, so Tetragon is already right there** — eBPF-based, and it can both *detect and enforce* in one tool, potentially collapsing two of my guards into one. I stayed with Falco + KubeArmor because Falco's rule ecosystem and CNCF maturity are unmatched for *detection*, and KubeArmor's CRD model is friendlier than hand-driving Tetragon `TracingPolicies` for *enforcement*. A leaner cluster could genuinely do this with Tetragon alone — it's on my honest "re-evaluate" list.
    
*   **Grype / Clair / Snyk / Docker Scout vs Trivy.** All fine scanners. **Snyk** has the best developer DX (and a bill); **Grype** is fast and SBOM-native; **Clair** is the registry classic. Trivy won because it's *all-in-one* — CVEs **and** IaC misconfiguration **and** leaked secrets **and** SBOM in one binary — it's free, and **my Harbor already scans every push with Trivy natively.** One scanner, build-time and registry-time, no new tool to learn.
    

The honest cost of running all four, stated plainly: it's real overhead — two DaemonSets (KubeArmor + Falco), Kyverno's five controllers, and the Trivy operator — and **KubeArmor is the finickiest thing in my whole cluster.** Its AppArmor annotations have wedged terminating pods, blocked Cilium's own BPF mounts, and confined CSI plugins — each of those is a scar in my notes. Falco will drown you in alerts until it's tuned. Kyverno's immutable-ConfigMap policy has bitten me more than once. If you want *one* layer to start, start with **Kyverno** — admission is the highest-leverage gate — and add the others as your threat model demands. Defense-in-depth is a spectrum, not all-or-nothing. But the four together are what turns *"I hope nothing's wrong"* into *"I can prove nothing's wrong."*

## Bringing the Four Guards Together

Step back and look at what's now standing watch over an empty house:

| Layer | Guard | Mode | When it acts |
| --- | --- | --- | --- |
| Admission | **Kyverno** | gate / mutate | *before* the workload enters etcd |
| Runtime audit | **KubeArmor** | LSM, `audit` posture | *while* it runs, at the kernel |
| Runtime detect | **Falco** | tracepoint match | *when* it behaves anomalously |
| Continuous | **Trivy** | scheduled scan | *long after* it was deployed |

Four engines, **zero shared dependencies** — no common operator, no shared CRDs, no single point of failure. If Kyverno's webhook is down mid-upgrade, the kernel layer is still watching. If a zero-day slips past every lock, the cameras still record it. If everything passes today, the Inspector still finds the CVE that lands tomorrow. Defense-in-depth isn't four tools; it's four tools that *don't know about each other*, so no single failure blinds the whole system.

And notice what isn't here: a dashboard to log into, an SSO redirect to configure, a UI to gate. The entire security posture is `kubectl get` against custom resources. **Security you query, not security you click.** For a GitOps platform, that's the point — every policy, every rule, every exception is a commit, reviewable and revertible, with no console drift.

## What's Next? (Don't Miss Out!)

The locks are set. The cameras are live. The inspector is on patrol. The house is finally safe to move into — and that's exactly what happens next.

With the security system armed *first*, the tenants can finally arrive: the private Harbor registry, Tekton pipelines, MLflow, Kubeflow, the data platform — every one of them landing in a cluster that rejects a bad manifest at the door, records every syscall at the kernel, alerts on anything anomalous, and never stops scanning its own inventory. That was the whole point of doing this in the empty house.

And the two honest gaps stay on the board where I can see them: **Falco's alerts still only reach** `stdout`, and **KubeArmor is still in** `audit` **posture with zero policies**. Neither is a surprise and neither is an accident — one needs a webhook in Vault, the other needs a few weeks of audit data before enforcement is anything other than an outage generator. Writing them down is how they stay next actions instead of quietly becoming the permanent state.

That's the next episode: **moving the workloads in — safely.** Follow along so you don't miss how the platform goes from *secured and vacant* to *secured and busy*. 🔐

📸 **SCREENSHOT (closer):** the ArgoCD `security-stack` ApplicationSet — all five child apps across two sync waves, `Synced` + `Healthy`. The whole security system, GitOps-reconciled, in one view.