# NVIDIA GPU Operator on RKE2: Passthrough to kubeflow (The Sports Car in the Garage)

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/3317728b-4367-489f-aa24-af841c364438.gif align="center")

*Episode 7 of the homelab build series. The foundation is poured, the walls are up, the alarms are wired, the storage floor is down and the safe is locked. Now for the fun room: three NVIDIA GPUs, one in each machine, so GPU capacity stays highly available when any one of them is down, managed by the NVIDIA GPU Operator and wired into Kubeflow.*

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/85a2ce5e-8354-4485-89a9-fe4d1c877b79.png align="center")

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/4398cd72-a66b-428a-952a-be01d6ff7545.png align="center")

`nvidia-smi` side by side on the three GPU nodes, one per datacenter.

## Introduction

*Three GPUs. Three datacenters. Zero schedulable.*

I had three NVIDIA cards in three machines, and Kubernetes swore I had none. I asked for one GPU in a test pod and got this:

```yaml
0/21 nodes are available: 21 Insufficient nvidia.com/gpu.
```

The Proxmox hosts could see them. `lspci` could see them. `nvidia-smi` on the host lit up. But to the cluster, three cards' worth of Ampere silicon were parked, idle and **invisible**. That's the most expensive kind of idle there is.

And the whole MLOps platform from the earlier episodes (Kubeflow, Katib, KServe, the training pipelines) was built to train and serve real models. You can't schedule what the scheduler doesn't know exists.

Fitting the cards took a Saturday. Getting the kernel module, the container runtime, the device plugin and the scheduler to agree took a lot longer, and taught me thirteen different ways a GPU can hide from Kubernetes.

Here's where it ended up. Three RTX A1000s, one in each "datacenter", which here means one in each of three mini PCs on a shelf. Lose a whole machine and two cards are still there. The NVIDIA GPU Operator builds the driver, wires up the runtime and advertises the cards on its own.

**And the part I'm proud of:** when the whole lab loses power, all three cards come back and pass their CUDA self-test without me touching anything.

This is how I got there, every trap along the way, and the parts that still aren't done.

And at the end there's a real job on a real card: a 7-billion-parameter language model answering at about 30 tokens a second, against 4 on the CPU beside it.

## Where this sits in the build

*   **Foundation:** Terraform on Proxmox.
    
*   **Framing:** Ansible building the five-master HA RKE2 cluster.
    
*   **Security:** Kyverno, KubeArmor and Trivy, with Falco parked for now to save CPU.
    
*   **Workshop:** Tekton, Harbor, SonarQube and Chains.
    
*   **Storage floor:** Rook-Ceph and CloudNativePG.
    
*   **The safe:** HA HashiCorp Vault.
    
*   **The sports car:** this one.
    

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/f0d93574-2b6e-49f4-b950-52cc737b8841.gif align="center")

This is where the cluster stops being general-purpose Kubernetes and gains an accelerator pool. Before the operator went in, the three GPU workers were scheduled like any other node, by CPU and memory, with no idea a GPU was attached.

The repo side is small: one namespace, one chart, one values file, and one `ClusterPolicy` the operator builds everything from.

```yaml
kubernetes-addons/0-boostrap/gpu-operator/
  namespace.yaml             # gpu-operator namespace, privileged Pod Security
  gpu-operator-values.yaml   # annotated Helm values; every trap fix lives here
  dcgm-custom-metrics.yaml   # extra DCGM fields for the dashboard
  nvidia-dcgm-dashboard.yaml # Grafana dashboard
  examples/                  # multi-GPU best-practice manifests
```

## The hardware in the garage

**The cards: three NVIDIA RTX A1000s.** Ampere `GA107GL`, 8 GB of GDDR6, 2,304 CUDA cores, about 50 W. It draws all its power from the PCIe slot, with no extra connector, which is exactly why it fits where a bigger card wouldn't. The node label says it plainly:

```shell
$ kubectl get node rke2-worker-dc1-gpu-a \
    -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product} {.metadata.labels.nvidia\.com/gpu\.memory}MiB'
NVIDIA-RTX-A1000 8188MiB
```

**The hosts: three Minisforum MS-01s.** Intel i9-13900H, 96 GB of DDR5, and, rarely for a mini PC, one real half-height PCIe slot wired to the CPU: x16 in length, x8 electrically, which is plenty for this card. That slot is the whole reason this build exists.

| Datacenter | Proxmox host | GPU worker | GPU | VRAM |
| --- | --- | --- | --- | --- |
| dc1 | proxmox-dc-1 | rke2-worker-dc1-gpu-a | RTX A1000 | 8 GB |
| dc2 | proxmox-dc-2 | rke2-worker-dc2-gpu-a | RTX A1000 | 8 GB |
| dc3 | proxmox-dc-3 | rke2-worker-dc3-gpu-a | RTX A1000 | 8 GB |

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/b1604544-65ff-4870-9f0a-40eced2e7bde.png align="center")

Three cards, three machines: PCIe passthrough from Proxmox host to RKE2 worker.

That's 24 GB of GPU memory across the fleet. One 8 GB card runs a 4-bit 7B model with room to spare; I measure that further down. It's great for inference, fine for small fine-tunes, and marginal for training from scratch, which is a rented-A100 job anyway.

**Why one card per datacenter instead of three in one box?** The same answer as storage: failure domains. If a machine goes down, GPU work still has two healthy places to land. In a homelab, rack density is worth nothing and HA is worth a lot.

**Why bother with Kubernetes when the GPU is right there on the host?** Three reasons. Each project gets its CUDA version from an image tag instead of eight CUDA installs fighting each other. The runtime becomes a pinned image, so results are reproducible. And the scheduler is the only sane way to say "this job gets one card, that one gets another".

## PCIe passthrough: from Proxmox to the VM

Each card is bound to `vfio-pci` on its Proxmox host and passed into exactly one VM. An Ansible playbook handles the nouveau blacklist, the vfio binding, IOMMU and the reboot. All three cards sit at the same address, `0000:01:00`, which keeps the Terraform pleasingly uniform.

The passthrough itself is Terraform with the bpg provider, through a `gpu_pci_hosts` map per datacenter. One gotcha deserves its own sentence: **raw** `hostpci` **needs the** `root@pam` **Proxmox user.** A normal API token can't assign raw PCI devices, and the error doesn't tell you that.

```yaml
cd terraform/environment/prod-rke2/bpg
terraform plan  -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu
# 0 to add, 2 to change, 0 to destroy   (in place: +machine=q35, +hostpci0)
terraform apply -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu
```

`machine=q35` and `hostpci0` only take effect after a full power cycle of the VM, not a soft reboot.

The best moment of the whole build came here. Once dc1 worked, I passed the cards through on dc2 and dc3 and rebooted them. The card appeared, the node got labelled, and the operator rolled the entire stack onto both nodes by itself. The first node took me twelve hours. The other two took about ninety seconds each, and I didn't touch them.

Three hardware notes that earned their place:

*   **Driver version is pinned** to `570.133.20`. The version stays put. What has to follow the kernel is the module, which the driver container rebuilds against whatever kernel the VM is running.
    
*   **No vGPU, no MIG.** The A1000 supports neither.
    
*   **Secure Boot never gets a say.** The GPU VMs boot with SeaBIOS, not UEFI, so the unsigned NVIDIA module loads without any key enrolment. The Proxmox hosts themselves do have Secure Boot on; it doesn't matter, because the module loads inside the VM. If your VMs use UEFI with Secure Boot, you'll have to sign the module or switch it off there.
    

## What the GPU Operator actually does

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/e0dc3e28-b63c-4f4a-a0bc-cea023185382.png align="center")

What the GPU Operator runs: a tower where every floor waits for the one below.

One Helm release of `nvidia/gpu-operator` (chart v26.3.2) runs a set of DaemonSets from a single `ClusterPolicy`. Sticking with the garage:

| Component | In plain terms | What it really does |
| --- | --- | --- |
| **Operator** | the garage manager | watches the `ClusterPolicy` and keeps everything below in shape |
| **NFD (Node Feature Discovery)** | the inspector | scans hardware and labels nodes that have an NVIDIA PCI device |
| **GFD (GPU Feature Discovery)** | the spec sheet | adds product, memory, count, driver and CUDA labels |
| **Driver** | the mechanic | compiles and loads `nvidia.ko` against the running kernel, inside a container |
| **Container Toolkit** | the licence plate | registers the `nvidia` runtime with containerd |
| **Device Plugin** | the parking attendant | advertises `nvidia.com/gpu` to the kubelet |
| **DCGM Exporter** | the dashboard | per-GPU metrics for Prometheus |
| **Validator** | the MOT | runs a CUDA test on every card and reports the result |

**The dependency chain is the thing to remember.** NFD (**Node Feature Discovery**) labels the nodes that have a GPU. Everything else only runs where that label is. So the driver, toolkit and device plugin land on the three GPU workers and nowhere else. No NFD label, no driver. No driver, no GPU.

And every other piece waits for the driver. When the bottom of that tower is broken, everything above it sits in `Init`. That gave me the rule that saved the most time: **debug from the driver up.** There's no point staring at the device plugin while the driver is still crash-looping.

What I deliberately don't run: the MIG manager (the A1000 has no MIG), the vGPU manager (that's for licensed shared vGPU), the sandbox and Kata pieces (the card is passed through once, at Proxmox), GPUDirect Storage (no hardware for it), and a standalone DCGM (the exporter brings its own, and two would fight over one socket).

The stack lives in the `gpu-operator` namespace with **privileged** Pod Security. The driver, toolkit and device plugin load kernel modules and touch host paths, and a restricted namespace blocks every one of them.

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/844fc896-8719-4188-aff5-d27516263428.png align="center")

`kubectl get pods -n gpu-operator` mid-cascade: a wall of `Init:0/1` and one very sad `nvidia-driver-daemonset`. The "before" shot.

## From PCIe slot to scheduled pod

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/01b53254-6666-41e5-8fcd-9247fbf42d70.png align="center")

From PCIe slot to scheduled pod: once per node, then once per pod.

Once per node:

1.  The worker boots with the GPU present as a PCI device.
    
2.  NFD (**Node Feature Discovery**) labels the node.
    
3.  The driver pod compiles `nvidia.ko` for the running kernel and loads it.
    
4.  The toolkit registers the `nvidia` runtime with containerd.
    
5.  The device plugin tells the kubelet the node has `nvidia.com/gpu: 1`.
    
6.  GFD adds `nvidia.com/gpu.product=NVIDIA-RTX-A1000` and friends.
    
7.  The DCGM exporter starts and metrics flow.
    

Then once per pod:

1.  A workload asks for `nvidia.com/gpu: 1`.
    
2.  Kyverno adds the taint toleration and the GPU node rule (more on that in the next section).
    
3.  The scheduler finds a GPU node with a free card and binds the pod.
    
4.  The NVIDIA runtime injects the device and driver libraries.
    
5.  `nvidia-smi` inside the container sees its card.
    

All of it is automatic once the operator is in. That's the whole argument against a hand-rolled Ansible role that builds kernel modules per node and breaks on every kernel upgrade.

## Keeping GPU pods on GPU nodes

Two bits of metadata mark these nodes, and you want both:

*   **A label** so workloads can find them: `nodepool=gpu`, plus everything NFD and GFD add.
    
*   **A taint** so ordinary workloads can't drift onto them and starve a GPU job: `nvidia.com/gpu=true:NoSchedule`.
    

```bash
$ kubectl get node rke2-worker-dc1-gpu-a -o jsonpath='{.spec.taints}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"true"}]
```

Normally that means every GPU pod needs a toleration, a node selector and a resource limit, and forgetting any one gives you a confusing failure. On this cluster a Kyverno policy, `gpu-pods-to-gpu-nodes`, does the first two for you. Any pod that asks for `nvidia.com/gpu` is mutated to tolerate the taint and require a `nodepool=gpu` node. So a GPU pod here needs exactly one line:

```yaml
resources:
  limits:
    nvidia.com/gpu: 1
```

The operator's own DaemonSets carry the toleration in the values file, and NFD's master also needs a control-plane toleration:

```yaml
daemonsets:
  priorityClassName: system-node-critical
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

nfd:
  master:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Equal
        value: ""
        effect: NoSchedule
```

### Prove it works

The quickest proof is NVIDIA's own CUDA sample. It adds two vectors on the card and prints one line if the whole chain held: driver, runtime, device plugin, scheduler.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
  namespace: default
spec:
  restartPolicy: OnFailure
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
      resources:
        limits:
          nvidia.com/gpu: 1
```

Look at what isn't there: no toleration, no node selector, no `runtimeClassName`. The only GPU line is the limit. The `securityContext` blocks have nothing to do with the GPU; my `default` namespace enforces the restricted Pod Security Standard, so every pod there needs them.

```plaintext
$ kubectl apply -f cuda-vectoradd.yaml
$ kubectl get pod cuda-vectoradd -o wide
NAME             READY   STATUS      RESTARTS   AGE   NODE
cuda-vectoradd   0/1     Completed   0          16s   rke2-worker-dc1-gpu-a

$ kubectl logs cuda-vectoradd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
```

Sixteen seconds from apply to `Completed`, image pull included. The pod landed on a GPU worker without being told to, and you can see what Kyverno wrote into it on the way in:

```bash
$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.tolerations}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Equal","value":"true"}]

$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.affinity.nodeAffinity}'
{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"nodepool","operator":"In","values":["gpu"]}]}]}}
```

Delete the pod when you're done. It has finished, so it isn't holding the card, but it's clutter.

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/015d08a7-c091-4578-899d-0d1d8d53e618.png align="center")

`kubectl logs cuda-vectoradd` ending in `Test PASSED`. The hero shot.

## The runtime: getting containerd to know about GPUs

containerd doesn't know what a GPU is. Something has to register an `nvidia` runtime that injects the device and driver libraries into containers. The classic way is to patch containerd's `config.toml`, and **RKE2 regenerates that file from a template**, throwing your patch away on every restart.

Two things make it work here. CDI is on, so the toolkit writes a device spec at `/etc/cdi/nvidia.yaml` that containerd reads at container start:

```yaml
cdi:
  enabled: true
  nriPluginEnabled: false
```

And the toolkit is pointed at RKE2's own containerd rather than the default one:

```yaml
toolkit:
  env:
    - name: CONTAINERD_CONFIG
      value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock
    - name: CONTAINERD_RUNTIME_CLASS
      value: nvidia
    - name: CONTAINERD_SET_AS_DEFAULT
      value: "true"
```

The subtle bit is that `CONTAINERD_CONFIG` points at the **generated** `config.toml`, not the `.tmpl`. The toolkit re-applies its setup every time it starts, so when a reboot regenerates the config without nvidia, the toolkit simply puts it back. **That's what makes it heal itself across reboots.** Point it at the template and the live config sits there without a GPU runtime until someone restarts `rke2-agent` by hand.

Because `nvidia` is the default runtime, workloads here don't need `runtimeClassName: nvidia`. On a cluster without that setting they do, and that one line is what confuses people copying manifests between clusters.

## Sharing the cards

Each A1000 advertises exactly one `nvidia.com/gpu`, so by default one pod owns the whole card.

| Mode | What it does | Isolation | Here |
| --- | --- | --- | --- |
| **Exclusive** | one pod owns the card | full | ✅ what runs today |
| **Time-slicing** | several pods take turns on the card | compute only, **memory is shared** | available, not switched on |
| **MIG** | hardware partitions with their own memory | full | ❌ the A1000 has no MIG |
| **MPS** | processes share one GPU context | partial | available, not used |

The node labels confirm it: `nvidia.com/gpu.replicas=1` and `sharing-strategy=none` on all three.

Time-slicing is a ConfigMap plus a patch to the `ClusterPolicy`:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  a1000: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4        # one A1000 looks like four schedulable GPUs
```

```bash
kubectl patch clusterpolicies.nvidia.com cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"a1000"}}}}'
```

**Create the ConfigMap first, then patch.** I ran the patch on its own once, and the device plugin's replacement pod sat in `Init` with `configmap "time-slicing-config" not found`, which took that node's card from one schedulable GPU to zero until I cleared the reference again. The DaemonSet replaces one pod at a time, so the rollout stalled there and the other two cards kept working.

Before you turn it on, remember it's oversubscription, not partitioning:

*   **No memory isolation.** Four pods share 8 GB. Four 2 GB models fit; a fourth 3 GB model dies with CUDA out-of-memory.
    
*   **No fault isolation.** One pod that wedges the card affects its neighbours.
    
*   **Sharing, not guarantees.** Right for notebooks and bursty inference, wrong for anything with a latency target.
    

My plan is to time-slice dc1 and dc2 and leave dc3 whole for jobs that need all 8 GB.

## How reliable is it?

Here's what the GPU tier survives, and the evidence for each:

| What goes wrong | What happens | How I know |
| --- | --- | --- |
| A whole machine goes down | Its card is gone, and so is any job that was running on it; new GPU pods land on the other two | One card per machine (dc1, dc2, dc3), each advertising `nvidia.com/gpu: 1` |
| The whole lab loses power | Every node rebuilds its GPU stack and re-tests its card, hands-off | After the last power cut, the toolkit, device plugin, DCGM exporter and validators came back on all three nodes by themselves, and the CUDA validator **Succeeded** on every card |
| RKE2 regenerates containerd's config on reboot | The toolkit re-applies the `nvidia` runtime | `CONTAINERD_CONFIG` points at the generated file, not the template |
| Someone forgets the GPU toleration | Kyverno adds it, plus the GPU node affinity | `gpu-pods-to-gpu-nodes` mutation policy |
| A notebook is abandoned with a card | It should be stopped after 24 hours idle and the card freed | Config only: `ENABLE_CULLING=true`, `CULL_IDLE_TIME=1440`. I have not seen it fire. The lab is powered off overnight, and two notebooks sat on cards until I stopped them by hand |
| A driver or runtime problem | The operator stops and reports it | `ClusterPolicy` state `ready`; validators gate everything above the driver |

And the honest limits. "Highly available" here means GPU *capacity*: a job running on a card dies with its machine and has to start again elsewhere. The operator itself is a single replica; if it's down, running GPU pods carry on and nothing new gets configured. A kernel upgrade on a GPU node still needs a driver rebuild (see the traps), there are no GPU alerts wired yet, and nothing caps how many cards one namespace can take. Those last three are on the list at the end.

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/1060c09f-baab-41c3-8a58-f873868b8ef1.png align="center")

`kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator`: three validators, one per datacenter, all `Completed`.

## The traps, from the driver up

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/100476b5-ac22-4a6f-876e-88fdbc8cbefa.png align="center")

Thirteen ways a GPU hides from Kubernetes, by layer, from the driver up.

Thirteen of them, grouped by layer, in the order you should debug them. Most of them bit me. A few I only know from other people's write-ups, and those say so. Every piece waits on the driver, so a broken foundation makes everything above it look broken too.

### The driver

**The driver gets OOM-killed, exit 137.** The log dies right after an `nvidia-drm` warning, then `nvidia-smi: not found`, and it crash-loops. It looks like a driver bug. It isn't: **the driver container compiles the kernel module**, running `gcc` against the kernel headers, and a 512Mi limit kills it mid-compile.

```yaml
driver:
  resources:
    limits:
      cpu: "2"
      memory: 4Gi     # 512Mi OOMs the kernel-module compile
```

`failed to install nvidia kernel module: distro 'ubuntu24.04' not supported`**.** The driver image ships pre-built modules for a fixed list of distro and kernel pairs, and mine wasn't on it. Build from source instead:

```yaml
driver:
  kernelModuleType: auto      # open modules first, proprietary as fallback
  usePrecompiled: false       # build from source, about 3 minutes on first boot
```

`nvidia: Unknown symbol __register_chrdev` **after a host kernel upgrade.** The cached module was built for the old kernel. Force a rebuild:

```bash
kubectl -n gpu-operator rollout restart daemonset/nvidia-driver-daemonset
```

The real lesson is a runbook entry, not a config change: **a GPU node's kernel upgrade is a special operation**, with a driver restart on the checklist. A plain power cut doesn't need it, because the kernel hasn't changed.

**Missing kernel headers.** The driver builds against the running kernel, so a kernel upgraded without `linux-headers-$(uname -r)` fails the build. The Ansible role for the GPU pool installs them, and the error is plain in the driver pod log.

### The validator

`nvidia-validator: command not found`**, exit 127.** Version skew, and the error tells you nothing. Older charts shipped the validator as a separate image; the 26.x chart puts the binary inside the main operator image. Values written for one layout against a chart of the other produce a missing binary. Pin the chart (`v26.3.2` here), deploy it from the NVIDIA repo rather than a stale vendored copy, and check what's actually running with `helm list -n gpu-operator`, not what your values file assumes.

### The toolkit and runtime

`no runtime for "nvidia" is configured`**.** The RKE2 trap the generic docs miss. The toolkit looks for `/run/containerd/containerd.sock`; RKE2 runs its own containerd at `/run/k3s/containerd/containerd.sock`. The fix is the `toolkit.env` block above.

`Failed to initialize NVML: Driver/library version mismatch` inside a pod, while the host shows the same version. The toolkit had patched `config.toml` directly and RKE2 regenerated it. CDI or the generated-config setting fixes it.

**KubeArmor's AppArmor profile blocks the toolkit.** The same family of trap that once bit Cilium here: the toolkit can't write what it needs, containerd is silently never configured, and pods report the `nvidia` RuntimeClass as missing. If you hit it, run the toolkit container unconfined. Today it runs fine under its own KubeArmor-generated profile.

**The toolkit writes the CDI spec and containerd never reads it.** Pods schedule, but `nvidia-smi` hangs. CDI needs containerd 1.7 or newer, so check what your RKE2 release ships before you switch it on. I didn't hit this one: my containerd is 2.x.

**Missing** `runtimeClassName: nvidia` on a cluster where nvidia isn't the default runtime. The pod schedules, runs with no GPU injected, and looks exactly like broken hardware. Not an issue here because of `CONTAINERD_SET_AS_DEFAULT`, which is why it's worth knowing: the requirement depends on the cluster.

### The device plugin and scheduling

`Insufficient nvidia.com/gpu` **while three healthy GPUs sit idle.** The driver and toolkit were running, but the device plugin was `Pending`:

```bash
0/21 nodes are available: 21 node(s) had untolerated taint
{nvidia.com/gpu: true}, that the pod didn't tolerate.
```

No device-plugin pod means nothing tells the kubelet about the card. The upstream chart ships the right toleration, but a trimmed values file copied from a blog about an untainted cluster quietly drops it. Same trap one layer down: NFD's master without its control-plane toleration never labels anything, so nothing else schedules either.

**NFD takes a minute on first install.** The operator comes up before NFD's first scan, so for about a minute it looks like the operator is running and nothing is happening. It isn't broken. Don't start changing things.

### Telemetry

**The DCGM exporter gets OOM-killed** right after logging `DCGM successfully initialized!`. It loads its whole field set into memory and 256Mi isn't enough:

```yaml
dcgmExporter:
  resources:
    limits:
      memory: 1Gi
```

**All of these fixes live in** `gpu-operator-values.yaml`.

## Who holds a card, and how to get it back

This is the part I'd most have wanted to read, because it's invisible until a GPU job won't schedule.

`nvidia.com/gpu` **can't be oversubscribed.** Without time-slicing, a pod with `nvidia.com/gpu: 1` owns that physical card for its whole life. Three cards means three GPU pods at once, full stop.

That makes long-lived notebooks the quiet problem. At one point two Kubeflow notebooks were each holding a whole A1000, one of them for three and a half months, while asking for 0.6 and 1.2 CPUs. A notebook that asks for less than one CPU and holds an entire GPU is the shape of the problem. It also quietly capped a Katib sweep at one GPU trial at a time. And they cost more than cards: together they used over half of their namespace's CPU quota, enough to make a daily training pipeline time out because its last job could never be scheduled.

**The claim is on the pod**, not the Notebook, the StatefulSet or the volume. So stopping a notebook frees the card. Kubeflow's "stop" sets an annotation, the controller scales to zero, and the pod goes away:

```bash
kubectl -n <profile> annotate notebook <name> \
  kubeflow-resource-stopped="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite
```

I sampled the free-GPU count every ten seconds while doing it:

```bash
[01] sts replicas=0   pod=Terminating   FREE GPUs=1
[02] sts replicas=0   pod=gone          FREE GPUs=2
```

Look at the first line. The notebook was "stopped" by every API you'd think to check, and the card was **still claimed**, because the pod hadn't finished terminating. `replicas=0` isn't proof a card came back. On a cluster where KubeArmor can wedge pods in `Terminating`, that matters. The reliable check counts from the pods and cross-checks the scheduler:

```bash
# what pods claim
kubectl get pods -A -o json | jq '[.items[] | select(.status.phase=="Running" or .status.phase=="Pending")
  | .spec.containers[].resources.limits["nvidia.com/gpu"] // "0" | tonumber] | add'

# what the scheduler has allocated on a node
kubectl describe node <gpu-node> | awk '/Allocated resources/,/Events/' | grep nvidia.com/gpu
```

As I write this, both GPU notebooks are stopped and all three cards are free.

**Culling is on as a safety net.** The notebook controller stops notebooks that have been idle for 24 hours. It stops, it never deletes: the volume and every file survive, and restarting is one click. What's lost is in-memory kernel state. A running cell counts as busy, so a long training run won't be culled. And the setting is global and not GPU-aware, so it catches abandoned notebooks; it doesn't manage contention. The better lever is not giving notebooks a GPU by default.

⚠️ One more surprise: `nvidia.com/gpu` isn't in the profile namespaces' ResourceQuota, and a quota only limits what it names. So GPU use per namespace is unbounded even though CPU and memory are capped.

## Putting the GPUs to work

Making `nvidia.com/gpu` schedulable was the prerequisite, not the goal.

### Who on the platform can ask for a card

Nothing on this platform *needs* a GPU to run. Every Kubeflow component is a CPU service; what changes is the work users launch through them. Here's each door to a card, and whether anything has walked through it yet:

| Platform piece | How it asks for a card | Has it used one here? |
| --- | --- | --- |
| **Kubeflow Notebooks** | a GPU dropdown in the spawner form | Yes, and too well: two notebooks held cards while idle |
| **Katib** | `nvidia.com/gpu` in the trial template | No. Both sweeps ran on CPU |
| **Kubeflow Trainer** | `resourcesPerNode` on a `TrainJob` | No. The runtimes are installed (`torch-distributed`, three torchtune LLM fine-tunes); no job has run |
| **KServe** | a GPU limit on the predictor | No. The one live model is XGBoost on CPU |
| **Kubeflow Pipelines** | a GPU limit on a single step | No. The daily pipeline is all CPU |
| **Spark** | executor GPU settings plus the RAPIDS plugin | No, and not planned: these jobs are small |
| **A plain pod** | one line, `nvidia.com/gpu: 1` | Yes: the LLM benchmark below |

Everything else in the data platform (Kafka, Airflow, MLflow, the databases) never touches a card and shouldn't.

So the honest summary is two real users so far: notebooks, which took cards and did nothing with them, and one benchmark pod, which took a card and drove it at 98%.

### Hyperparameter search with Katib

Katib runs an `Experiment` that launches trials as pods, each reporting a metric back through an injected sidecar. Getting it to run at all taught me more than the GPUs did.

**It couldn't run in any Kubeflow profile namespace.** An Experiment sat at `Created` with **zero** trials, and the UI only said "Couldn't find any successful Trial." The real error was two levels down, on a ReplicaSet:

```bash
pods "<exp>-tpe-..." is forbidden:
  minimum memory usage per Container is 16Mi, but request is 10Mi
```

Katib asks for 10Mi of memory for its suggestion pods, and the Kyverno-generated LimitRange in every profile namespace has a 16Mi floor. Eleven of twelve algorithms had no explicit resources, so eleven of twelve couldn't schedule. Fixing those exposed the same problem on the trial pods, because the metrics sidecar is injected by a webhook and its default never shows up in the template. The fix is to raise Katib to meet the floor, not lower the floor:

```yaml
runtime:
  suggestions:
    - algorithmName: tpe
      image: ghcr.io/kubeflow/katib/suggestion-hyperopt:v0.19.0
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
  metricsCollectors:
    - kind: StdOut
      image: ghcr.io/kubeflow/katib/file-metrics-collector:v0.19.0
      resources:
        requests:
          cpu: 50m
          memory: 32Mi
        limits:
          cpu: 200m
          memory: 128Mi
```

Three more gotchas from getting a sweep to finish:

*   `katib-config` **is mounted with** `subPath`, which never receives updates. Restart the Katib controller after every config change; waiting does nothing.
    
*   **Never put a multi-line** `python3 -c` **script in a trial template.** The stdout collector rewrites the command into a shell string, and the inline script doesn't survive. Pass a file path and flags instead.
    
*   **A sub-second trial can finish before the collector attaches**, which shows up as random failures with an empty observation. Pace the objective and set `backoffLimit: 1`.
    

With all of that fixed, a 12-trial TPE sweep finished 12 out of 12 and found the known optimum, on CPU, because for a demonstration sweep a GPU changes nothing (more on that below). And with no time-slicing, the real ceiling for GPU trials is three at a time, one per card. A `parallelTrialCount` higher than that just leaves trials `Pending`.

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/7ce46241-5394-48dd-a893-350cf6916135.png align="center")

The Katib trial leaderboard with the best trial highlighted.

### Training and serving

A GPU training job in Kubeflow Trainer V2 looks like this. I haven't run this one yet, so treat it as the shape, not a tested manifest. With one card per machine, two workers have to land on two different machines:

```yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: resnet-finetune
  namespace: terranetes
spec:
  runtimeRef:
    name: torch-distributed
  trainer:
    numNodes: 2
    resourcesPerNode:
      limits:
        nvidia.com/gpu: 1
```

For serving, a KServe `InferenceService` with `minReplicas: 0` gives the card back when the model is idle and cold-starts it on demand. That works *because* the GPU claim is per pod: scale to zero, the pod goes, the card is free. With three cards, scale-to-zero is the difference between one always-on model and a fleet of models sharing three GPUs.

My live models today don't need a GPU at all. The football match predictor is an XGBoost model on tabular data, and it trains and serves on CPU in seconds. The first real GPU TrainJob and a properly served LLM are both on the list at the end.

### The first real job: a 7B model on one card

For three months these cards passed their self-test and did nothing else. Prometheus has the receipts: 0% utilisation and 0 MiB of GPU memory on all three, the whole time. So before writing another word about what an A1000 can do, I made one do it.

The job is deliberately plain. One pod, one card, Ollama running Mistral 7B in 4-bit (`Q4_K_M`, a 4.1 GB download), the same prompt every time with a fixed seed. The whole request for hardware is one line, `nvidia.com/gpu: 1`. Kyverno added the toleration and the node rule, the scheduler put it on dc2, and `nvidia-smi` inside the container saw the card.

```yaml
NAME          SIZE      PROCESSOR    CONTEXT
mistral:7b    5.8 GB    100% GPU     4096
```

Then the same model on the same node with the GPU switched off, four threads against the pod's four-core limit:

|  | Tokens per second | Notes |
| --- | --- | --- |
| **RTX A1000** | **about 30** (four runs: 29.9 to 31.6) | 5,031 MiB of the 8,188 MiB used, 98-99% utilisation |
| **CPU, 4 threads (i9-13900H)** | **about 4.2** (two runs: 4.1 and 4.3) | same model, same prompt, same seed |

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/7aa8f4b7-f46f-4f2c-9782-3cdd24184914.png align="center")

Mistral 7B on one RTX A1000 against the CPU beside it: about 30 tokens a second against about 4.2.

Seven times faster, on a 50-watt card that fits in a mini PC. A 400-token answer took 13 seconds on the GPU. At the CPU's rate it would take over a minute and a half. That is the difference between a tool you'd use and one you wouldn't.

Three things the run taught me that the spec sheet didn't:

*   **It runs hot.** The card idles at about 60 °C in the MS-01 and sat at 88 to 90 °C within a minute of load, a few degrees over its own 86 °C target, with the driver reporting a software thermal slowdown. Speed drifted from 31.6 to 29.9 tokens a second across the runs. It never reached the 96 °C hard slowdown, but a small case and a single-slot cooler have a ceiling, and sustained training will live at it.
    
*   **It can't tell you its power draw.** `nvidia-smi` reports `power.draw` as `[N/A]` on this card under passthrough. So the missing power metric isn't an exporter setting I forgot.
    
*   **Threads matter more than cores.** My first CPU run was left to pick its own thread count. It saw 20 cores, started 20 threads inside a four-core limit, and was still going ten minutes later. Pinned to four threads it produced 60 tokens in about fifteen seconds. If you benchmark CPU inside a container, set the thread count to the limit.
    

The manifest and the exact commands are in `examples/llm-benchmark-pod.yaml`. It's a benchmark, not a service: the pod holds a whole card while it exists, so I deleted it when the numbers were in, which is this article's own advice.

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/b3501523-7b07-4e91-be23-e50d3dac5dc9.jpg align="center")

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/99f23523-50b5-4b9a-b303-3132ef7c7716.png align="center")

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/83715d77-7f52-4bdd-a28f-4994f721ce07.png align="center")

![](https://cdn.hashnode.com/uploads/covers/67014345dbb510bc35d60f47/09097233-2ea5-4266-a6f8-ffb6c8b269f3.png align="center")

The Grafana DCGM dashboard for dc2 across the run: utilisation jumping from 0 to 98%, memory to 5 GiB, temperature from 60 to the high 80s. The first non-zero lines those panels have ever drawn.

## Multi-GPU, and does a GPU change the result?

With more than one card, the internet will tell you to reach for tensor parallelism and model sharding. On three 8 GB cards over PCIe with no NVLink, most of that advice is wrong.

| Strategy | Splits | Use when | On these cards |
| --- | --- | --- | --- |
| **Data parallel (DDP)** | the batch, full model per GPU | the model fits in 8 GB | ✅ the default |
| **Sharding (FSDP, ZeRO)** | parameters and optimiser state | the model won't fit one GPU | ⚠️ prefer offloading to CPU RAM |
| **Tensor parallel** | individual layers | needs NVLink | ❌ slower over PCIe than not doing it |
| **Pipeline parallel** | layer stages | very deep models, many GPUs | ⚠️ rarely worth it here |

Use single-GPU tricks first (bf16, gradient checkpointing, smaller batches), then DDP for throughput, and only then sharding. For inference, don't shard: quantise. A 7B model in int4 fits on one card, and splitting it across two PCIe cards is strictly worse.

**Does a GPU change the result, or just the speed?** It changes how fast you get a result, not what the result is. With a fixed number of trials, a GPU gets you the same answer sooner. With a fixed amount of time, faster trials mean more trials, and more trials is how a search finds a better optimum. That gain comes from trial count, not from the GPU. And it can make results slightly worse: GPU reductions aren't always deterministic, so training often isn't bit-for-bit reproducible.

So putting a CPU-bound job on a GPU is theatre, and it costs a card something else needed.

## Monitoring

The DCGM exporter publishes per-GPU metrics, and a ServiceMonitor puts them in Prometheus, with NVIDIA's dashboard in Grafana. The ones worth watching:

*   `DCGM_FI_DEV_GPU_UTIL`: utilisation
    
*   `DCGM_FI_DEV_FB_USED` and `DCGM_FI_DEV_FB_FREE`: memory used and free
    
*   `DCGM_FI_DEV_GPU_TEMP`: temperature
    
*   `DCGM_FI_DEV_PCIE_REPLAY_COUNTER`: link errors on the PCIe slot
    

Two I want and don't have. `DCGM_FI_DEV_XID_ERRORS` isn't in my exporter's field list yet. `DCGM_FI_DEV_POWER_USAGE` isn't either, and adding it won't help: this card doesn't report power draw under passthrough at all.

The most useful panel is utilisation by pod:

```promql
sum(DCGM_FI_DEV_GPU_UTIL{Hostname="rke2-worker-dc2-gpu-a"}) by (exported_namespace, exported_pod)
```

Mind the label. On my setup `pod` is the exporter's own pod, and the workload is in `exported_pod`. Group by `pod` and every card looks like it's being used by the DCGM exporter.

What's missing is any alerting. The metrics are there, but nothing fires on a hot card, full memory or an XID error yet. And the metric I want most isn't on any dashboard: **allocated versus used**. A card that's 100% allocated and 0% used is the notebook problem, and utilisation alone makes it look exactly like a free card.

NVIDIA has since open-sourced something aimed at exactly that gap: the [GPU Usage Monitor](https://github.com/NVIDIA/gpu-usage-monitor), an Apache 2.0 Helm chart that pairs the DCGM exporter with kube-state-metrics, Prometheus and Grafana, and ships dashboards for allocated against used, by namespace and by pod, plus GPU pods stuck `Pending`. I haven't installed it. I already run every one of those four pieces, so what I'd want from it is the dashboards, not a second Prometheus. And its platform filter is named for datacenter cards (Hopper, Blackwell), so I don't yet know how it treats an A1000. It's on the list.

## What it costs

Three used RTX A1000s for about £600, in slots the MS-01s already had. No extra power. The card is rated at 50 W; I can't give you a measured figure, because it doesn't report one. Cooling is the honest cost: nothing extra was fitted, and under load the card sits at its thermal limit.

The operational cost is mostly kernel upgrades on GPU nodes, which need the driver restart. Adding a GPU node is now Terraform the passthrough, label and taint it, and let the operator do the rest.

## What's Next?

The sports car runs, survives a power cut and comes back on its own. What's left, roughly in order:

*   **Try NVIDIA's GPU Usage Monitor dashboards** against the Prometheus I already have, to get allocated versus used on a screen.
    
*   **Export XID errors**, then **wire up GPU alerts:** temperature (now that I know it reaches 90 °C), memory nearly full, XID errors, and a card allocated but unused for hours.
    
*   **Turn on time-slicing** on dc1 and dc2, and keep dc3 whole.
    
*   **Put** `nvidia.com/gpu` **in the profile ResourceQuota**, so one namespace can't take every card.
    
*   **Run a real GPU TrainJob** across two machines, and **serve that LLM properly** behind KServe with scale-to-zero, not from a benchmark pod.
    
*   **Automate the driver restart** after a GPU node's kernel upgrade, so it stops being a checklist item.
    

If you've got a GPU sitting in a box that Kubernetes can't see, I hope this saves you the thirteen traps on this list.

[**Follow me on LinkedIn**](https://www.linkedin.com/in/george-ezejiofor-89615a8a/)

*Author: George Ezejiofor*