Tekton, Cosign, Kyverno: A Signed CI/CD Supply Chain on RKE2

Introduction

The Workshop Where Trust Is Manufactured

Here's a confession I'll make out loud: I was running code I'd never read, built by people I've never met, pulled from a registry I don't control—and deploying it with a straight face. docker pull something:latest, kubectl apply, ship it. I called it a supply chain. It was really a leap of faith with extra steps.

Not in this cluster.

By Episode 3, the house had its locks, its cameras, and its inspector—a four-layer security system that refuses a bad manifest at the door and a bad syscall at the kernel. But a guarded house is only ever as safe as what you carry into it. And the single most dangerous thing you can carry into a Kubernetes cluster is an image you cannot prove you built.

So before anything ships, it goes through the workshop.

This is Episode 4 — the CI/CD stack, and it runs on one non-negotiable rule: nothing executes in this cluster that wasn't built here, scanned here, and signed here. Every image is assembled on my own line, put through a quality gate, stamped with a cryptographic maker's mark, and stored in my own warehouse. And the security system from Episode 3? Its front door—Kyverno—refuses to admit anything that doesn't carry that mark. The workshop and the security system are two ends of the same handshake.

The supply-chain Interactive diagram (Git → Tekton → SonarQube → kaniko → Chains → Harbor → Kyverno), or a Harbor repo showing an image tag with its .sig + .att artifacts.

The Stations

Here's what's on the workshop floor:

Station	Engine	Job
🏭 The Assembly Line	Tekton	Builds the image, in-cluster, from source
🔬 The Quality Inspector	SonarQube	Fails the build if the code doesn't pass the gate
🧪 The Safety Test	Trivy	Fails the build if the pushed image carries a CRITICAL or HIGH CVE
✍️ The Maker's Mark	Tekton Chains	Signs every artifact (cosign) + attaches SLSA provenance
📦 The Warehouse	Harbor	Stores the image, its signature, its provenance, its CVE scan and its SBOM
🚪 The Checkpoint	Kyverno verifyImages	Admission rejects anything unsigned — and pins what it admits to the signed digest

Each is independent. But wired in sequence they produce something a public registry never can: an image with a provable birth certificate.

Where I Am in the Build

This is the fourth episode of a homelab that stopped being a project a while ago:

• Episode 1 — The Foundation. Terraform across three Proxmox datacenters.

• Episode 2 — The Framing. HA RKE2 control plane via Ansible, kube-vip, Cilium.

• Episode 3 — The Security System. Kyverno, KubeArmor, Falco, Trivy.

• Episode 4 — The Workshop. Tekton, SonarQube, Tekton Chains, Harbor — the subject of this piece.

Two tools got removed on the way here, and the removals are as deliberate as the additions: Jenkins (Tekton + Chains + Argo Events cover every trigger without a snowflake controller) and Nexus (Harbor is a purpose-built OCI registry with native cosign + Trivy baked in). Fewer moving parts, every one of them cloud-native.

The stack lives in kubernetes-addons/cicd-stack/, deployed as a single ArgoCD ApplicationSet:

cicd-stack/
├── README.md
├── tekton/
│   ├── tekton-operator/         ← the Operator + TektonConfig (one CR, whole estate)
│   ├── tekton-pipeline/         ← reference build-and-push Pipeline + PipelineRuns
│   └── tekton-chains/           ← cosign keypair + cosign.pub for verifiers
├── harbor/                      ← OCI registry (+ Trivy scan, native Keycloak OIDC)
│   ├── charts/
│   └── vault-integration/       ← admin + registry creds via Vault → VSO
├── sonarqube/                   ← code-quality gate (SAML2 SSO)
│   ├── charts/
│   └── vault-integration/
├── github-action-runner/        ← self-hosted GitHub runners (ARC)
│   ├── controller/
│   └── runner-set/
├── cicd-pipeline-e2e-example/   ← the whole chain, end to end, as one example
│   ├── app/                     ← trivial Go service + its Dockerfile
│   ├── pipeline/                ← e2e-build-scan-sign + a PipelineRun to trigger it
│   ├── policy/                  ← the Kyverno verifyImages gate (Enforce)
│   └── deploy/                  ← what ArgoCD syncs once the image is signed
└── external-dns/                ← Cloudflare DNS sync (scaffolded)

A quick word on SSO — three tools, three patterns

One detail worth calling out, because it trips people up: the three UIs each authenticate differently, dictated by what each tool natively supports.

• Tekton Dashboard → oauth2-proxy reverse-proxy (no native auth).

• Harbor → native OIDC (Harbor speaks it directly).

• SonarQube → native SAML2 (Community edition has no OIDC).

All three federate to Keycloak (with Google + GitHub behind it), but you don't get to pick one pattern and reuse it—you meet each tool where it lives. (And if the SonarQube/Harbor SAML login ever starts redirect-looping, it's almost always a stale IdP cert after a Keycloak key rotation—re-sync it, don't rebuild the realm.)

Three UIs with SSO login with Google and Github — the Tekton Dashboard, Harbor, and SonarQube — nine ArgoCD child applications, and a supply chain that ends with a cryptographic proof. I'll walk the floor.

Why Build Your Own

Let's be blunt about the threat, because it's the reason the whole workshop exists.

When you pull nginx:latest or some-vendor/app:v2, you are trusting: the author, every dependency the author pulled, the CI system that built it, the registry that stored it, and the network path in between—none of which you can see, and any of which can be compromised without you ever knowing. The famous supply-chain attacks of the last few years didn't break in through the front door. They walked in inside a trusted image.

Microsoft Supply Chain Attack (February 2023)

In February 2023, a software supply chain attack also affected Microsoft. The attack exploited a vulnerability in the Jfrog Artifactory, a binary repository manager that Microsoft uses to distribute and store its software components.

The attackers accessed Jfrog Artifactory and injected malicious code into some of Microsoft’s software components, allowing them to access Microsoft’s network while stealing source code and other confidential information.

A homelab that hosts Vault, Keycloak, private model registries, and real data pipelines can't hand-wave that away. So the answer isn't "scan harder." The answer is provenance: don't run what you can't trace. Build the image yourself, from source you control, on a line you can audit. Prove—cryptographically—that the thing running in production is the exact thing your pipeline produced, and reject everything else at the door.

That's a chain of custody. And a chain is only as strong as its weakest link, so every station below hardens one link:

• The Assembly Line proves where it was built (in-cluster, from your Git).

• The Inspector proves the code met a bar (quality gate).

• The Maker's Mark proves it wasn't tampered with after the build (signature + SLSA attestation).

• The Warehouse proves it hasn't rotted (continuous CVE scan).

• The Checkpoint enforces all of the above at runtime (unsigned → rejected).

The honest head-to-head — assemble best-of-breed, or buy the integrated platform?

There's a fair objection to everything above: GitLab already does all of this in one product. So do GitHub, Azure DevOps, and JFrog. Why bolt together seven tools when a single platform ships SCM + CI + registry + scanning + deploy pre-wired, one login, one vendor to call? That's the real decision at the workshop level — not "Tekton vs Jenkins" (that's the Assembly-Line post), but assemble a best-of-breed supply chain, or buy an all-inclusive one. The field splits three ways:

1. Integrated DevOps platforms — GitLab (SCM + CI + registry + SAST/DAST/container scan + deploy in one product), GitHub (Actions + Packages + Advanced Security + Environments), Azure DevOps, JFrog Platform (Artifactory + Xray). One product, one UI, one auth, batteries included.

2. Managed cloud-native chains — AWS (CodePipeline + ECR), GCP (Cloud Build + Artifact Registry + Binary Authorization). Wired into cloud IAM.

3. Assembled best-of-breed (this workshop) — Tekton + SonarQube + Trivy + Harbor + Chains + Kyverno + ArgoCD. Best-in-class per stage, K8s-native, self-hosted, composable — but you are the integrator.

	Integration	Best-in-class / stage	Self-host on your K8s	Supply-chain depth (sign→verify)	Lock-in	You integrate
This workshop	Self-hosted	✅ each	✅	✅✅ cosign + SLSA + admission verify	none	✅ (the tax)
GitLab	✅✅ pre-wired	⚠️ "good enough"	✅ (CE/EE)	⚠️ signs, weaker on-cluster admission	medium	❌
GitHub	✅✅	✅ SCM	❌ (GHES $$)	⚠️ attestations, off-cluster	medium	❌
Azure DevOps	✅	⚠️	❌	⚠️	medium	❌
JFrog	✅ artifact-centric	✅✅ registry/scan	✅	✅ Xray	high ($)	partial

Where the integrated platforms genuinely win:

• GitLab is the strongest all-in-one, full stop. One product, one UI, one auth, SCM through deploy — and it self-hosts. For a team that wants CI/CD to just work without becoming systems integrators, GitLab is the right answer and I'd recommend it without flinching. My only counter: it's a monolith where each stage is "good enough" rather than best-in-class, and you live on GitLab's rails.

• GitHub (Actions + GHAS) — unmatched SCM, marketplace, Dependabot, and native build attestations. If your code already lives on GitHub, the integrated path is the path of least resistance. It leans cloud (self-hosted GHES is a bill), and the trust story lives off your cluster.

• Azure DevOps — mature and enterprise-solid in Microsoft shops (Boards + Repos + Pipelines + Artifacts); aging, but cohesive.

• JFrog Platform — if artifacts are your center of gravity, Artifactory + Xray is the gold standard for binary management and deep dependency scanning. Commercial, and heavier than Harbor for a homelab.

Why assemble for this cluster: every station is best-in-class and Kubernetes-native — the whole thing runs on my own RKE2 with no external runners, no per-seat SaaS bill, and no cloud — it's composable (swap SonarQube out without touching the rest), there's zero lock-in, and — the part integrated platforms rarely match on-prem — the supply-chain depth: cosign signatures + SLSA provenance from Tekton Chains, enforced at admission by Kyverno verifyImages. The cluster refuses to run an image it can't prove it built. That last, load-bearing link is the one most platforms don't give you as rigorously inside your own cluster.

The honest cost is precisely what GitLab sells against: I am the systems integrator. Seven tools is seven things to deploy, wire, and keep talking — Tekton's robot creds into Harbor, Chains' signing key, Kyverno's cosign.pub, ArgoCD watching all of it. GitLab hands you that pre-wired; I hand it to myself, and that assembly is the tax. I pay it for best-of-breed, self-hosting, and a provenance story I control end to end — which, for a homelab hosting Vault, Keycloak, and real data pipelines, is the entire point. If I were shipping a product on a deadline with a small team, I'd probably just buy GitLab and move on.

The Supply Chain — seven stages

▶ Open the interactive version — click any node to trace what depends on it, or follow one of the three red rejection paths.

In words, the line runs:

#	Stage	Engine	Produces	Fails the run if…
1	Trigger	GitHub Actions on a self-hosted ARC runner	a PipelineRun	—
2	Quality gate	SonarQube, in-pipeline before the build	pass/fail	the quality gate fails (proven: exit 3 on planted violations)
3	Build	Tekton + kaniko v1.23.2	image + digest	the build fails
4	Vulnerability gate	Trivy 0.58.0 against the pushed digest	pass/fail	a CRITICAL or HIGH CVE is present
5	Sign + attest	Tekton Chains (x509, simplesigning + in-toto)	.sig + .att in Harbor	—
6	Label	the pipeline's own finally task	Harbor signed label	Chains never published a signature
7	Admission verify	Kyverno verifyImages	admit / reject	the image is unsigned or signed by the wrong key

Notice the shape: it's a loop that closes on itself. The workshop produces a signed image; GitOps deploys it; the security system checks the workshop's own signature before it runs. Nothing about that trust is imported.

And every one of those gates has been shown to say no, not just yes — which is the only evidence that any of them are real. A gate you've never seen refuse something is indistinguishable from a gate that admits everything. The two negative tests are in the walkthrough below.

---

The Assembly Line — Tekton :

Tekton is the line itself—Kubernetes-native CI, where every build step is a container and every pipeline is a CRD. There's no long-lived build server to patch, no plugin hell; a PipelineRun spins up pods, does its work, and disappears.

I run it through the Tekton Operator: a single TektonConfig CR is the whole control surface. From that one resource the operator reconciles Pipelines, Triggers, the Dashboard, Chains, the Pruner, and the ManualApprovalGate into the tekton-pipelines namespace. One CR, the entire Tekton estate—declarative, versioned, self-healing.

The working pipeline is e2e-build-scan-sign, and it runs in supply-chain order:

1. prepare-source — stages main.go + Dockerfile into the build workspace. Source arrives as a content-hashed ConfigMap, not a git-clone: this repo is private, and an unauthenticated clone 404s.

2. sonar-scan — the SonarQube quality gate, with qualitygate.wait=true so the scanner blocks on the server's verdict instead of posting an analysis and exiting 0.

3. trivy-fs-scan — a filesystem pre-check for secrets and Dockerfile misconfiguration. This is not the vulnerability gate. More on that in a moment, because it is the most instructive trap in the whole pipeline.

4. kaniko — builds the OCI image without a Docker daemon (no privileged socket to escape from) and pushes harbor.georgehomelab.com/library/<app>:<tag>, writing the digest to IMAGE_DIGEST.

5. trivy-image-scan — the real gate: trivy image --severity CRITICAL,HIGH --exit-code 1 against the pushed digest.

6. write-image-url — emits IMAGE_URL alongside IMAGE_DIGEST, so Chains knows exactly which artifact to sign.

7. finally: label-signed — waits for Chains to publish the signature, then labels the artifact in Harbor. It is a finally task for a reason, covered in the Maker's Mark section.

kubectl create -f cicd-pipeline-e2e-example/pipeline/pipelinerun.yaml
kubectl -n tekton-builds get pipelinerun -w

Why the filesystem scan is not the gate. Both steps are Trivy, both carry --exit-code 1, and only one of them can actually fail a build. I proved it by pinning an EOL golang:1.23-alpine base image and running it:

prepare-source     → exit 0
trivy-fs-scan      → exit 0        # ← blind to it
kaniko             → exit 0        # vulnerable image is pushed, by design
trivy-image-scan   → exit 1        # Total: 22 (HIGH: 21, CRITICAL: 1)
write-image-url    → never ran

trivy filesystem reads dependency manifests and lockfiles. A stdlib-only Go program has none, so it reports num=0, exits 0, and looks exactly like a pass — while 22 CVEs sit compiled into the binary. Only trivy image against the pushed digest sees them. A pipeline carrying just the filesystem scan would look identical to this one on every green run and gate nothing.

The CRITICAL was CVE-2025-68121 in crypto/tls. Putting the toolchain back to golang:1.27-alpine went green again with a byte-identical digest — kaniko --reproducible means the same source rebuilds to the same bytes, which is what makes a signature worth anything.

Scanning after the push is deliberate, and worth being explicit about: a failing image does land in Harbor. But the step still gates deployment, because a failed step fails the TaskRun, so IMAGE_URL is never written, Chains never signs it, and the digest never reaches a manifest. The unsigned failure then gets refused a second time at admission. Two independent gates catch it.

The Tekton Dashboard mid-run — a PipelineRun with its Tasks going green, or the kaniko step streaming a build.

Two traps I paid for:

The /workspace/source permission wall. Tekton's prepare initContainer pre-creates the shared workspace as uid 65532. Your Task containers run non-root and then can't write into their own workspace—permission denied, mid-build, for no obvious reason. The fix is to grant the one capability that unblocks it (or match the uid):

securityContext:
  capabilities:
    add: ["DAC_OVERRIDE"]     # or: runAsUser: 65532

The Dashboard's login lie (feedback_oauth2_proxy_authheader_breaks_kube_passthrough). The Tekton Dashboard has no native auth, so it sits behind an oauth2-proxy (Keycloak realm tekton-dashboard). The overlay must set PASS_AUTHORIZATION_HEADER=false and SET_AUTHORIZATION_HEADER=false. Flip them to true and oauth2-proxy forwards the Keycloak JWT to the Dashboard, which hands it to the kube-apiserver as a bearer token—the apiserver rejects it (Keycloak isn't a registered OIDC provider at the apiserver), and the UI shows 401 Error loading <kind> after a visibly successful login. The meanest kind of bug: the one that looks like it worked.

---

The Quality Inspector — SonarQube :

A build that compiles isn't a build worth shipping. SonarQube is the inspector standing between "it built" and "it's allowed to leave the workshop": if the code fails the quality gate—coverage, bugs, security hotspots, code smells past a threshold—the run fails and no image gets signed. The gate isn't advisory; it's wired into the line.

Writing this section is what got the gate fixed. The draft originally said the scan ran inline, then that it ran from the workflow — both were wrong, and the truth was that e2e-build-scan-sign simply had no SonarQube step at all. It had been dropped because it failed both PipelineRuns this cluster had ever had. The failures turned out to be a usage error, not a gate saying no: sonar-scanner ran without sonar.projectKey, which exits 1 or 2 before kaniko is reached. That reads identically to a quality gate rejection in the PipelineRun view, which is why it stood for three months.

It is back now, as step 2, before the build — there is no reason to compile and push code that has already failed review.

And the second attempt failed more quietly than the first. My step passed -Dsonar.sources=/workspace/build, an absolute path. sonar.sources resolves relative to sonar.projectBaseDir, so the scanner matched nothing and said so in a line nobody reads:

0 languages detected in 0 preprocessed files
0 files indexed
QUALITY GATE STATUS: PASSED

Green build, gate passed, nothing analysed. I only caught it because I planted deliberate violations in main.go and watched them sail through. With -Dsonar.projectBaseDir=/workspace/build -Dsonar.sources=. the same source indexes 2 files, the gate returns FAILED, the step exits 3, and kaniko never runs. Reverting the violations puts it back to green with a byte-identical digest.

Note which failure was better: the original exited 2 and stopped the line. Mine exited 0 and shipped. A gate that fails loudly is a working gate; a gate that passes without looking is worse than no gate at all, because it buys you confidence you have not earned. The only way to tell them apart is to make the thing fail on purpose.

The scan Task talks to SonarQube over the cluster-internal URL (http://sonarqube-sonarqube.sonarqube.svc:9000) with a scan-only token from Vault—it never touches the SAML2 SSO that gates the human UI. Machines use tokens; people use Keycloak. (SonarQube Community has no native OIDC, so the human path is SAML2, which is its own small saga—see the SSO note below.)

Two traps here that cost real time:

The CrashLoop with no error (feedback_sonarqube_chart_gotchas). SonarQube's most baffling failure mode is a pod that restarts forever with liveness-probe failures and nothing useful in the logs. The cause is almost always a missing monitoringPasscode (SONAR_WEB_SYSTEMPASSCODE)—without it the liveness endpoint never returns healthy, so the pod is killed and restarted in perpetuity. The Vault path secret/homelab/sonarqube carries a monitoring_passcode field for exactly this.

The migration dirty-flag deadlock (feedback_golang_migrate_dirty_trap). On an interrupted first boot, the schema-migration table can be left with dirty=true, and every subsequent start refuses to proceed. The release valve is holdApplicationUntilProxyStarts (so Istio's sidecar is up before the migration runs) plus a one-time UPDATE schema_migrations SET dirty=false. Also: don't double-declare SonarQube's ServiceAccount—the chart already makes one, and declaring it again in the kustomization gives ArgoCD a render conflict.

A SonarQube project's Quality Gate — the green "Passed" (or a red "Failed" that killed a pipeline), with the coverage/bugs/hotspots breakdown.

---

The Maker's Mark — Tekton Chains :

This is the station that turns "I built it" into "I can prove I built it, and prove it hasn't changed since." Tekton Chains watches every TaskRun, and the moment a build produces an image, it:

1. Signs it with cosign — a .sig artifact, keyed to the private half of a keypair that lives only in the signing-secrets Secret inside the cluster.

2. Attaches an in-toto SLSA provenance attestation — a .att artifact that records what was built, from what, by which pipeline. A machine-readable birth certificate.

The keypair is owned by Vault, not by a bootstrap script. It lives at secret/homelab/tekton/cosign and VSO projects it into the signing-secrets Secret that the Chains controller mounts. The Vault role is bound to the tekton-chains-controller ServiceAccount alone — deliberately not to tekton-builder, the identity kaniko runs as. Folding the key into the robot credential that already has push rights would collapse "can push" and "can sign" into one blast radius, which is the exact property signing exists to keep apart. The public half is committed so anyone — including the admission controller — can verify without holding the secret.

Verifying offline is a two-liner:

cosign verify --insecure-ignore-tlog \
  --key tekton/tekton-chains/cosign.pub \
  harbor.georgehomelab.com/library/hello-supply-chain:v1

cosign verify-attestation --insecure-ignore-tlog --type slsaprovenance \
  --key tekton/tekton-chains/cosign.pub \
  harbor.georgehomelab.com/library/hello-supply-chain:v1

The trap that cost me 101 days: chains.tekton.dev/signed: "true" is not evidence of anything. The signing-secrets Secret shipped by the Tekton operator is created empty, and Chains stamps that annotation on completed runs regardless. Every run for over three months carried signed=true while the signer was failing with no valid private key found, looked for: [x509.pem, cosign.key] — a warning in the controller log and nowhere else. The run is green, the annotation says signed, and there is no signature.

Only cosign verify against the public key counts. A related detail: because the key is a volume mount read once at signer startup, and the Tekton operator reverts kubectl rollout restart on its own Deployments within about two seconds, the only way to load a rotated key is to delete the pod.

That lesson is also why the finally task exists. Chains signs a TaskRun only after it completes, so a step inside the build cannot see the signature yet — labelling there would assert something unverified, which is the same bug in a new costume. Instead the finally task polls Harbor for the signature accessory and only then writes the label. No signature inside five minutes and it fails the run: the build, scan and push all succeeded, but an artifact admission will refuse is not a green pipeline.

(--insecure-ignore-tlog because Chains' transparency-log integration isn't wired to a public Rekor here—signatures are verified against the key, not a public log.)

The trap (feedback_harbor_chart_gotchas, last one): Chains' k8schain reads imagePullSecrets from the TaskRun's ServiceAccount, not the Chains controller's. If the controller logs UNAUTHORIZED reading a manifest it's trying to sign, it's not a Chains bug—the tekton-builder SA in tekton-builds needs the Harbor robot secret attached:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-builder
  namespace: tekton-builds
imagePullSecrets:
  - name: harbor-tekton-builder-creds

The cosign verify output returning Verified OK, and/or a Harbor tag's "Signatures" column showing the .sig + .att sitting next to the manifest.

---

The Warehouse — Harbor :

Every workshop needs a warehouse it controls, not a public shelf anyone can restock. Harbor (harbor.georgehomelab.com) is that warehouse: a purpose-built OCI registry that stores the image, displays its cosign signature and SLSA attestation inline, runs a native Trivy scan on every push, and gates human access with native Keycloak OIDC (callback /c/oidc/callback; the harbor-admin Keycloak group maps to Harbor system admin). Machines authenticate as robot accounts—tekton-builder in the library project, push+pull, one-shot secret vaulted.

A warehouse only shows you what you've asked it to look for. My artifact page displayed a signed, scanned, pipeline-built image as: not scanned, no SBOM, no labels. Nothing was broken. The library project carried exactly one setting — public: true — and both auto_scan and auto_sbom_generation default to off, so Harbor's own Trivy had never run on a single artifact in it, despite the chart shipping that scanner and registering it as default.

Two API calls fixed it permanently, and they now live in a PostSync hook Job beside the OIDC one, because this configuration lives in Harbor's database, not the chart — a rebuilt Harbor would silently regress to the same blank page:

curl -u admin:$PW -X PUT -H 'Content-Type: application/json' \
  -d '{"metadata":{"auto_scan":"true","auto_sbom_generation":"true"}}' \
  "$H/api/v2.0/projects/1"

Worth saying plainly: Harbor's scan is not a duplicate of the pipeline's. The pipeline gate proves an image was clean when it was built. Harbor re-scans on its own schedule and catches CVEs disclosed after publication — something a build-time gate structurally cannot do.

The signature, meanwhile, had been there the whole time. Harbor 2.x renders cosign signatures as accessories, behind the expand arrow on the artifact row — there is no "signed" column any more; that was Notary v1. An artifact that looks unsigned in the list may be perfectly signed, and ?with_accessory=true on the API is how you find out.

Harbor is also, hands down, the fussiest chart in this whole stack. Three traps enforced by the chart itself (feedback_harbor_chart_gotchas):

1. secret_key must be EXACTLY 16 characters — Harbor core has a hardcoded length check, and the same value has to appear under three Secret keys (secret, secretKey, HARBOR_SECRET_KEY). Sixteen. Not fifteen, not seventeen.

2. The htpasswd value must keep its username: prefix — the chart strips trailing newlines but does not add the prefix, so you generate it with htpasswd -nbBC 10 harbor_registry_user <password> and leave the user: on the front.

3. The PostgreSQL StatefulSet needs updateStrategy: Recreate — leave it on the default RollingUpdate and every upgrade deadlocks, because the new pod can't bind the RWO PVC the old pod still holds.

None of these are in the quickstart. All three will cost you an evening if you don't know them going in.

A Harbor repo view — an image tag with its Trivy CVE summary and the Signatures column populated. The warehouse shelf, labelled and sealed.

---

The Signed Blue/Green images deployed via Argocd & exposed with istio service-mesh

Closing the Loop — where the Workshop meets the Security System

Here's the payoff, and the reason Episodes 3 and 4 are really one system.

Everything above produces a signed image. But a signature nobody checks is just decoration. So the last link isn't in the workshop at all—it's the front door from Episode 3: a Kyverno verifyImages ClusterPolicy that, at admission, cryptographically checks every image against cosign.pub and rejects anything unsigned or signed by the wrong key.

git push → Tekton builds → SonarQube gate → Trivy gate → Chains signs →
Harbor stores → ArgoCD deploys → Kyverno verifies the signature → runs
                                                     │
                                        unsigned? ───┴──▶ REJECTED at the door

The policy does two things at admission, and they have to be switched on together: it verifies the signature, and with mutateDigest: true it rewrites :v1 to the verified @sha256:…. Git still says :v1; what actually runs is the exact bytes that were signed. I learned the ordering the hard way — verifyDigest: true demands a digest on the reference, and with mutateDigest still false a tag never acquires one, so Enforce rejected my own signed image with missing digest. The signature was fine the entire time; the report message just reads almost identically to a signature failure.

One more line earns its place, for anyone running this under GitOps:

annotations:
  pod-policies.kyverno.io/autogen-controllers: none

Without it Kyverno autogenerates a Deployment-scoped copy of the rule that validates the pod template, where the reference is still a tag. Under Enforce that denies every write to the Deployment — including ArgoCD's server-side apply dry run — so the Application parks in ComparisonError while the pods stay happily Healthy, and the error names validate.kyverno.svc-ignore rather than anything that sounds like ArgoCD. Pods are the right enforcement point anyway: an unsigned Deployment rolls out zero of them, because every Pod the ReplicaSet creates is refused.

And the assertion that matters, the one that proves the door is a door — I pushed an image to the same repository outside the pipeline, so Chains never saw it:

Error from server: admission webhook "validate.kyverno.svc-ignore" denied the request:
  verify-hello-supply-chain-signature:
    verify-cosign-signature: 'failed to verify image
      harbor.georgehomelab.com/library/hello-supply-chain:unsigned:
      .attestors[0].entries[0].keys: no signatures found'

That's the whole thesis made real: the cluster doesn't trust an image because it came from a registry. It trusts it because it can prove the image came from this workshop. Provenance in, provenance enforced—a closed loop with no imported faith.

Akubectl apply of an unsigned public image being rejected by the Kyverno verifyImages policy — the checkpoint turning away an artifact with no maker's mark.

What's Next? (Don't Miss Out!)

The workshop is running. Source goes in one end; a scanned, signed, provenance-stamped image comes out the other—and the front door won't admit anything that didn't. The cluster now manufactures its own trust instead of importing it.

Which means the platform is finally ready for the thing all of this was for: real workloads. With a security system that guards the house and a workshop that proves everything built for it, the next episodes move the tenants in—the MLOps platform, the data lakehouse, the private services—every one of them running on images this cluster built and signed itself.

That's the next chapter: from a secured, self-provisioning platform to a busy one. Follow along so you don't miss how the empty, hardened house finally fills up. 🏭

The ArgoCD cicd-stack ApplicationSet — Tekton, Harbor, and SonarQube all Synced + Healthy. The whole workshop, GitOps-reconciled, in one frame.

Subscribe and follow me on [LinkedIn] to catch my upcoming articles.