Building a Hands-On Learning Platform on a Home Server — Putting Kubernetes Inside an Unprivileged Pod

Once you have a Kubernetes cluster standing in your house, the next question is usually the same one. What do I do with it?

I decided to build a learning platform. A learner opens a course and hits start, a container dedicated to that person comes up immediately, a terminal in the browser attaches to a real Linux shell, and at each step a grade button makes the server inspect the actual state inside the pod and report pass or fail. When the lab finishes or the time limit runs out, the container disappears.

Looking back after building it, the hard part was not the features. It was isolation.

The problem is that you have to hand out root

Because this is a hands-on lab, the learner has to be root inside the container. You learn by running useradd, running chmod, digging around the filesystem. That part is not negotiable.

But that root must not get outside the container. This cluster sits on my home network, and on that same network are the router, the NAS, the registry, and the other nodes.

The first version of the lab pod spec looked like this.

spec:
  containers:
    - name: lab
      image: registry.internal/labhub/lab-linux:v1
      command: ["sleep", "infinity"]
      resources:
        requests: { cpu: 100m, memory: 256Mi }
        limits:   { cpu: "1",  memory: 1Gi }

No securityContext. No network policy either. I checked whether it actually gets through.

# from inside the lab pod
cat < /dev/null > /dev/tcp/192.168.219.1/80    # router admin page
cat < /dev/null > /dev/tcp/10.96.0.1/443       # Kubernetes API server

Both opened. On top of that, since I had not turned off automountServiceAccountToken, the default service account token was mounted inside the pod. Which means a learner who felt like it could call the cluster API.

Tier 1 — open the network per lab

Cilium decides policy by identity, not by IP. So opening toEndpoints to nothing but kube-dns cuts the pod off from every other pod in the cluster, and opening no toEntities at all cuts it off from the nodes and the API server. toCIDRSet applies only to destinations outside the cluster, so that is where you carve out the private ranges.

At first I opened internet 80/443 for every lab. Then it occurred to me that the Linux, Git, and database labs have everything they need inside the image and do not need a single byte of internet. A door with no reason to be open is better closed.

So I split it into three profiles.

Profile	What it opens	Labs that use it
Default (no label)	DNS only	Linux, Git, FDE, DB, Kubernetes
`net=internet`	+ public internet 80/443	labs that need apt or pip
`net=registry`	+ the registry domain only	container labs

The default is the narrowest. A pod with no label automatically gets DNS and nothing else.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: lab-egress-base
spec:
  endpointSelector:
    matchLabels:
      app: lab-session
  egress:
    # Name resolution only. Attach the L7 DNS parser and Hubble records who asked for which domain.
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules:
            dns: [{ matchPattern: "*" }]

The profile that opens the internet carves out every private range.

    - toCIDRSet:
        - cidr: 0.0.0.0/0
          except:
            - 10.0.0.0/8          # private A + pod CIDR + service CIDR
            - 172.16.0.0/12       # private B
            - 192.168.0.0/16      # home network, router, NAS
            - 169.254.0.0/16      # link-local + cloud metadata
            - 100.64.0.0/10       # carrier CGNAT
      toPorts:
        - ports:
            - { port: "80",  protocol: TCP }
            - { port: "443", protocol: TCP }

Restricting this to ports 80/443 is deliberate too. apt, pip, and git clone work, but outbound SSH brute forcing, SMTP spam, and port scanning do not. A lab environment must not become a springboard for attacking other people.

Ingress is blocked outright. The browser terminal is an exec path through the API server and the kubelet, so it does not use the pod network and nothing needs to be open for it. The single exception is web preview. For a learner to view an app they started inside the lab pod, the backend has to reach that port, so only the designated port coming from the backend pod is allowed. As a side effect, this rule also blocks learners from attacking each other's pods.

After applying it, I knocked again.

router 192.168.219.1:80       → blocked (timeout)
NAS 192.168.219.109:5001      → blocked
registry :80                  → blocked
Kubernetes API 10.96.0.1:443  → blocked
neighboring lab pod           → blocked
DNS                           → allowed

What matters is that the block is a timeout, not a Connection refused. It means packets are being dropped silently, and that is what a firewall looks like when it is doing its job.

Tier 2 — grant kernel privileges per lab

Blocking the network does not change the fact that root inside the container is still root. This is where Linux capabilities come in.

At first I gave the same nine to every lab, even though the Linux lab needing SETUID for useradd is no reason for the Kubernetes lab to get the same privileges. So I changed the default to "nothing at all" and split it into profiles that add only what a lab actually requires.

CAP_PROFILES = {
    # The default. Not a single capability.
    # The Kubernetes, DB, FDE, and Git labs all get by on this — creating
    # files you own and starting processes needs no special privilege.
    "none": [],
    # labs that handle files owned by someone else
    "files": ["CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID"],
    # labs that create and switch between users
    "users": ["CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID",
              "SETUID", "SETGID", "KILL"],
    # labs that run rootless containers
    "container": ["CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL",
                  "SETUID", "SETGID", "SETPCAP", "SYS_CHROOT"],
}

Some things go into no profile at all.

NET_RAW — the key to packet sniffing, ARP spoofing, and port scanning. Withholding it is the counterpart to the tier 1 network policy. ping still has to work, though, and the net.ipv4.ping_group_range sysctl handles that. It opens ICMP datagram sockets, so ping works without a raw socket. And that sysctl is on the safe list the kubelet allows by default, so you can simply write it into the pod spec.
SYS_ADMIN — effectively root itself. Mounting and namespace manipulation ride along with it.
SYS_PTRACE — reads the memory of other processes.
SYS_MODULE — the host kernel is a shared resource.

And allowPrivilegeEscalation gets turned off. Leave it on and no matter how far you pare capabilities down above, a single setuid binary brings them all back. This setting applies no_new_privs, which neuters setuid itself.

There is one side effect: sudo stops working. But you are already root inside the container so the labs are unaffected, and syntax checks like visudo -c still run, so the certification labs are fine too.

Tier 3 — hand out no credentials

serviceAccountName: lab-nobody      # an account with not a single RoleBinding
automountServiceAccountToken: false # do not mount the token at all
enableServiceLinks: false           # do not leak other service addresses as env vars

The last line is the one people forget. By default Kubernetes injects the address of every service in the same namespace as environment variables. For a lab pod that is useless, and all it does is hand over the internal layout.

But then how do you run a Kubernetes lab

This is where I got stuck. Teaching Kubernetes means handing the learner a cluster, but kind runs nodes as containers and k3s has to run a kubelet, so both need privilege. The moment you grant privilege, all three tiers built so far become meaningless. Escaping a privileged container is not hard, and on the other side of the escape is the home network.

The answer was kwok.

kwok runs etcd, the API server, the controller manager, and the scheduler as ordinary processes. Then the kwok controller simulates fake nodes and carries pods all the way to Running. With no real container runtime there is not a grain of privilege needed.

kwokctl create cluster --name lab --runtime binary --wait 180s

--runtime binary is the crux. kwokctl defaults to bringing the components up as containers via docker or podman, and this pod has no container runtime. The binary runtime just runs the binaries as processes.

I ran it for real in a pod with every capability dropped.

3 nodes Ready
39 CRDs registered
startup time 11.4s

What works and what does not in this environment deserves an honest accounting.

Works — CRUD on every resource, deployment scaling, rollout and undo, scheduling (node selectors, affinity, taints, drain), RBAC verification, ResourceQuota and NetworkPolicy objects, kubectl explain, kustomize, helm, etcd snapshot save and restore, applying third-party CRDs.

Does not work — kubectl exec into a workload pod, real log output, port forwarding, actual CNI traffic, data sitting on a volume. With no real containers, that follows.

As it happens, most of what the CKA and CKAD exams ask on paper falls in the first list. The second list is covered with theory and quizzes instead of hands-on work.

Traps I stepped in

I pointed the readiness probe at a business API

The deploy stopped. New pods kept coming up but never went Ready, the rollout timed out, the pipeline failed. Not one error in the logs.

The cause was the probe.

readinessProbe:
  httpGet: { path: /api/courses, port: 8000 }   # business API

During a refactor I renamed /api/courses to /api/paths. The probe got a 404 and the pod never went Ready — while the application was working perfectly well.

The lesson is simple. Probes belong on dedicated endpoints and nowhere else.

startupProbe:                       # be generous on first boot
  httpGet: { path: /healthz, port: 8000 }
  periodSeconds: 5
  failureThreshold: 36              # up to 3 minutes
readinessProbe:
  httpGet: { path: /healthz, port: 8000 }
livenessProbe:
  httpGet: { path: /healthz, port: 8000 }
  failureThreshold: 5

/healthz checks only the process, while /readyz goes as far as the DB connection. The reason liveness carries no DB check is that restarting the application because the DB wobbled for a moment only makes things worse.

The incident did have one good side to it: production kept the old version. The pipeline did its job.

A moving tag and imagePullPolicy

I rebaked the lab image and nothing changed. No matter how many times I rebuilt, the files inside the pod were exactly as before. The build log said "Pushed".

The default imagePullPolicy in Kubernetes is Always only when the tag is latest, and IfNotPresent otherwise. I was using v2, a moving tag, so the node reused the image it had pulled once, forever. No error is logged anywhere.

image: registry.internal/labhub/lab-k8s:v2
imagePullPolicy: Always     # mandatory for a moving tag

The straight answer is to use immutable tags. But the curriculum keeps growing and the images get rebaked often, so given that the registry sits on the same network I went with Always.

kwok ignored the cache

A lab pod cannot reach anywhere except DNS. But kwokctl downloads the component binaries from the internet when it starts. If the cache is empty, cluster creation simply fails.

So I pre-downloaded them at image build time, and it made no difference.

{"level":"INFO","msg":"Download","uri":"https://github.com/etcd-io/etcd/.../etcd-v3.5.11-linux-amd64.tar.gz"}
{"level":"ERROR","msg":"Failed to setup config","err":"... i/o timeout"}

kwok uses the default versions it picks itself (what I had downloaded was a different version), and the cache path mirrors the URL structure exactly.

/root/.kwok/cache/https/dl.k8s.io/release/v1.30.4/bin/linux/amd64/kube-apiserver

Miss on either the version or the path and it fails to find the cache and downloads again. Instead of guessing URLs, I switched to creating a cluster once during the build and then deleting it.

RUN kwokctl create cluster --name warmup --runtime binary --wait 180s && \
    kwokctl delete cluster --name warmup && \
    rm -rf /root/.kwok/clusters

kwok fetches what it needs its own way and puts it in its own path, so there is no room to miss. The cluster state (PKI and etcd data) is deleted and only the cache stays. Startup time dropped from 1 minute 52 seconds to 11.4 seconds.

A session whose pod had died held its slot forever

I limited each learner to one lab open at a time, because the moment someone closes the window and forgets about it, a pod is wasted.

But the database saying "running" does not mean the pod is alive. If a node restarted, or it got OOM killed, or someone deleted it by hand, the pod is gone and only the record remains. That learner can then never open a new lab again.

I fixed it to check that the pod is alive before reusing a session, and if it is dead, to close the record out and create a new one. I added one more direction to the reaper as well.

Expired — for sessions out of time, delete the pod and close the record
Orphan — a pod that is alive but absent from the database gets deleted
Ghost — a session the database calls running but whose pod is gone gets its record closed

The third is what I added this time. It has to exclude anything created within the last minute, though, because a pod created moments ago may not be visible in the API yet.

Three layers that keep resources from leaking

Lab pods are expensive. Even when a learner just closes the window, nothing should be left behind.

Every session gets a hard expiry. It is decided at creation time and the remaining time shows on screen. A notice appears at five minutes left, and extension is possible but only up to three times the original.
One session per person. Opening a new lab requires finishing the existing one first. The UI says which lab is open and lets you jump straight to it or end it.
A reaper runs every 60 seconds. It cleans up along the three directions above, and an ops screen continuously shows whether the database and the actual pods agree.

The namespace has a ResourceQuota and a LimitRange on it. There are ceilings on pod count and on total CPU, memory, and ephemeral storage, and count/services: "0" is in there so lab pods cannot create Services or PVCs. The ephemeral storage ceiling matters most: without it someone can fill the disk and kill the node.

Pinning security regressions down with tests

This isolation is scattered across several files: the pod spec, the network policies, the namespace labels. Regress any one of them and the isolation is broken — and the service runs perfectly well while broken. So without tests, nobody finds out.

I built a check that runs on every push.

✅ the default profile is empty (0 capabilities)
✅ NET_RAW is granted in no profile
✅ SYS_ADMIN is granted in no profile
✅ privilege escalation is blocked
✅ the SA token is not mounted
✅ the internet profile excludes 192.168.0.0/16
✅ ingress is blocked
✅ the ingress exception is limited to the backend
✅ PSA has not been lowered to privileged
✅ no API keys are hardcoded in the source
...
passed — all 38 checks green

One thing I learned. The first version of the check searched the source for the string NET_RAW, and it false-positived on a comment that read "do not grant NET_RAW". It now parses the actual profile dictionary with the Python AST and looks at the values. A checker has to aim precisely at what it means to check — obvious once said, hard to see before you live it.

Deployment goes through git

At first CI pushed straight into the cluster with kubectl set image. Which means finding out what is running requires asking the cluster, rollback depends on somebody remembering the old tag, and there is no telling apart what was hand-edited from what was deployed.

Now gitops/ in the repository is the truth.

gitops/
  base/    shared across environments — no image tag, no namespace
  dev/     namespace, session TTL, LoadBalancer exposure
  prod/    namespace, replicas 2, SSO secret, public URL

The image tag is a single images.newTag line in the overlay. CI edits only that line and commits, and ArgoCD syncs on its own. Rollback is one commit reverting that line. Touch anything with kubectl and selfHeal undoes it, so no drift survives.

There was one trap. I defined the Service in the base manifest as ClusterIP only, and ArgoCD turned the dev LoadBalancer into a ClusterIP, cutting off access. When moving to GitOps you have to transcribe what is already in the cluster, accurately, before anything else. It is safer to assume that whatever you fail to transcribe disappears.

Where it stands now

I measured how long a lab pod takes to come up.

Linux lab      session created in 15s → usable immediately
Kubernetes     session created in 6s → environment ready in 15s (kwok cluster startup)

The Kubernetes lab needs extra time for the cluster to come up even after the pod goes Running. Without surfacing that, a learner hits grade in the meantime and concludes they got it wrong. So the prep script drops a marker file as it finishes, the backend reads it and reports whether prep is still in progress, and the UI locks the grade button until it is ready.

Limits and what is left

A few things worth writing down honestly.

The container lab runs real containers with rootless podman, but the storage driver is vfs and it is slow. overlayfs needs mount privileges and fuse-overlayfs needs a device file, and I can give neither. With the small images the labs use the difference is not very noticeable, but building a lab around large images is hard.

The virtualization lab runs QEMU under software emulation. With no /dev/kvm there is no choice. A small Alpine VM boots, but slowly. Why KVM is fast is covered as theory, and the hands-on part is designed around disk image manipulation and snapshots.

tcpdump does not work, because NET_RAW is never granted. Teaching packet capture needs another approach, and since I have no good answer yet it is covered with theory and quizzes. The network diagnostics lab is designed around ss, dig, getent, and /proc/net.

Wrapping up

Building a hands-on environment on a home server, the biggest constraint was not performance but isolation. And that constraint is exactly what made the design better. Because I could not grant privileges I went looking and found kwok, and the result is a cluster that comes up in 11 seconds. Because I could not open the internet I baked everything needed into the images, and the result works completely offline.

If you have a habit of solving problems by widening permissions, going the other way once is not a bad experiment.