I Break It — A Chaos Lab Where the Hypothesis Comes First

The breaking room — five failures injected, one design fixed

Goal

You run five chaos experiments on a real k3s inside a temporary VM. You measure the steady-state baseline, write the hypotheses for all five experiments together before breaking anything, kill a Pod, squeeze the CPU, and starve the memory, then reflect what you learned in the design and take the same attack again. This is a 65-minute experiment for intermediate learners who know basic kubectl queries and the structure of a Deployment.

Why it matters

Injecting a failure is itself one command. The hard part is making that experiment produce something that is still there next week. So the product of this lab is not a failure but a note. You write the hypothesis before the experiment, measure inside the experiment window, and write the conclusion as the result of comparing the two. A prediction that misses is not a failure — a refuted hypothesis is the most expensive piece of information that experiment can earn, and writing it down as refuted is the correct answer.

Judgment table (names to use in the conclusion)

So that people do not read the same number differently, you first fix the rules for naming.

Availability: ok if the success rate is 0.98 or higher, degraded if 0.5 or higher, down below that
Latency: unmeasured if there are fewer than 5 successful latency samples, slower if p95 is at least twice the baseline p95, otherwise same
Verdict: confirmed if both values in the hypothesis equal both observed values, refuted if even one differs

Measurement tool

python3 /opt/fixtures/chaos_lab.py does only three things. It neither causes the failure for you nor fixes it for you.

load <초> <원시.json> (seconds, raw results file) — Sends two streams of load through the Service. For availability, 10 requests per second to the static path /, and for latency, 2 requests per second to /cgi-bin/work, which uses CPU. Start it in the background and cause the failure inside that window.
record <국면> <원시.json> <기록.json> (phase, raw results file, record file) — Saves the load results together with the cluster at that moment. It refuses to record if the structural conditions of the requested phase do not actually hold.
grade — Used by the grader.

Safe scope

You use only the API https://127.0.0.1:6443 inside the temporary VM and the labhub-chaos namespace. Never run this on a production cluster or in someone else's environment. The provided app uses the Recreate strategy to leave only one cause — that does not mean it is a recommended value for zero-downtime operation. The app image is pinned by digest.

This lab is set at 65 minutes, which is longer than the default session (60 minutes). Extend the time in advance when you start. When the session ends, the VM and the record files disappear and cannot be restored. The record files are editable experiment notes, not tamper-proof evidence.

Steps

Apply /opt/fixtures/chaos-lab-app.json, wait for the rollout to finish, and then run a 20-second load with nothing touched to record the baseline. Measure with python3 /opt/fixtures/chaos_lab.py load 20 /root/chaos-lab/raw-01.json and save it with python3 /opt/fixtures/chaos_lab.py record steady /root/chaos-lab/raw-01.json /root/chaos-lab/01-steady.json.
Read the success rate and p95 from the baseline file and write the steady state in steady_state, and write the hypotheses for the five experiments (kill-one · kill-ha · cpu-squeeze · mem-squeeze · hardened) in /root/chaos-lab/hypothesis.json. Each experiment needs four things: availability, latency, why you expect it (why), and when you will stop (abort_if). Set availability_ratio_min to at least 0.95 and no more than the measured baseline value, and p95_ms_max to at least the baseline p95 and no more than 3 times it.
With one replica, start the load in the background and delete that one and only Pod inside the measurement window. When the window closes, save it with python3 /opt/fixtures/chaos_lab.py record kill-one /root/chaos-lab/raw-03.json /root/chaos-lab/03-kill-one.json. Leave the resource limits and preStop as in the baseline.
Raise the replicas to 3, and once all three are Ready, delete exactly one Pod inside the load window in the same way. Save it with python3 /opt/fixtures/chaos_lab.py record kill-ha /root/chaos-lab/raw-04.json /root/chaos-lab/04-kill-ha.json. Change nothing except the number of replicas.
Lower only the cpu limit of the shop container to 100m (memory stays at 96Mi, replicas stay at 3). After the rollout has finished and all three are Ready, run a 20-second load and save it with python3 /opt/fixtures/chaos_lab.py record cpu-squeeze /root/chaos-lab/raw-05.json /root/chaos-lab/05-cpu-squeeze.json.
Restore the cpu limit to 500m while lowering the memory limit to 40Mi. The app grabs 48 MiB all at once at startup, so the container dies from OOM. Once restarts are visible, run a 20-second load and save it with python3 /opt/fixtures/chaos_lab.py record mem-squeeze /root/chaos-lab/raw-06.json /root/chaos-lab/06-mem-squeeze.json.
Restore the resource limits to the normal values (cpu 500m · memory 96Mi), and this time add shutdown handling. Use a preStop hook to buy at least 3 seconds and raise terminationGracePeriodSeconds to 10 or more. When the rollout finishes, repeat the same attacks as in steps 3 and 4 and save it with python3 /opt/fixtures/chaos_lab.py record hardened /root/chaos-lab/raw-07.json /root/chaos-lab/07-hardened.json.
Name the numbers of the five records according to the judgment table in the instructions and write them in /root/chaos-lab/conclusion.json. Each experiment needs observed_availability, observed_latency, a verdict (confirmed or refuted) compared with the hypothesis, and what you will change based on this result (action). The final grading looks at the earlier records and the current cluster together — the fixed design must still be alive.

Notes

Look at the target and its state separately with kubectl -n labhub-chaos get pods,svc,endpointslices -o wide.
kubectl -n labhub-chaos get rs shows the traces of past deployments. The final grading looks at this.
Check the termination reason and exit code in kubectl -n labhub-chaos describe pod <이름> (replace the placeholder with the Pod name).
Start the load in the background (&), cause the failure, wait for the window to close with wait, and then record.
If you need to measure the baseline again, you may go back to an earlier step. But do not edit hypothesis.json.
For limit values, you must look not only at limits but also at the relationship with requests. A limit smaller than the request is rejected.

Record "normal" as numbers before breaking anything

Apply /opt/fixtures/chaos-lab-app.json, wait for the rollout to finish, and then run a 20-second load with nothing touched to record the baseline. Measure with python3 /opt/fixtures/chaos_lab.py load 20 /root/chaos-lab/raw-01.json and save it with python3 /opt/fixtures/chaos_lab.py record steady /root/chaos-lab/raw-01.json /root/chaos-lab/01-steady.json.

Running, Ready, and actual request success are different facts. Measure after there is 1 Ready endpoint. If a new Pod appears inside the measurement window, it is not accepted as a baseline.

Write the five hypotheses first

Read the success rate and p95 from the baseline file and write the steady state in steady_state, and write the hypotheses for the five experiments (kill-one · kill-ha · cpu-squeeze · mem-squeeze · hardened) in /root/chaos-lab/hypothesis.json. Each experiment needs four things: availability, latency, why you expect it (why), and when you will stop (abort_if). Set availability_ratio_min to at least 0.95 and no more than the measured baseline value, and p95_ms_max to at least the baseline p95 and no more than 3 times it.

Your predictions do not have to be right. Writing a wrong prediction up as refuted later is the correct answer in this lab. The meaning of the labels is in the judgment table in the instructions. Do not edit this file from here on — the final grading checks whether it was written before the first experiment record.

Kill a single-replica service

With one replica, start the load in the background and delete that one and only Pod inside the measurement window. When the window closes, save it with python3 /opt/fixtures/chaos_lab.py record kill-one /root/chaos-lab/raw-03.json /root/chaos-lab/03-kill-one.json. Leave the resource limits and preStop as in the baseline.

Start the load first and kill the Pod a few seconds later so the impact is captured inside the window. If you start measuring after the kill, you measure numbers from after the replacement Pod is already ready. The record is accepted only if the Pod is Ready again when the window closes.

Raise replicas to three and repeat the same attack

Raise the replicas to 3, and once all three are Ready, delete exactly one Pod inside the load window in the same way. Save it with python3 /opt/fixtures/chaos_lab.py record kill-ha /root/chaos-lab/raw-04.json /root/chaos-lab/04-kill-ha.json. Change nothing except the number of replicas.

You can explain the result only if you change one variable at a time. Measure after confirming all three are Ready. Even if the success rate does not recover to the baseline level, just record it as it is — that is the result of this experiment.

Squeeze the CPU — a failure that slows things down instead of killing

Lower only the cpu limit of the shop container to 100m (memory stays at 96Mi, replicas stay at 3). After the rollout has finished and all three are Ready, run a 20-second load and save it with python3 /opt/fixtures/chaos_lab.py record cpu-squeeze /root/chaos-lab/raw-05.json /root/chaos-lab/05-cpu-squeeze.json.

Resource limits live in the Pod template, so changing them brings up new Pods. If you measure in the middle of the rollout, the effect of the deployment and the effect of the limit get mixed. Also remember that a limit cannot be smaller than the request.

Starve the memory — this time it dies

Restore the cpu limit to 500m while lowering the memory limit to 40Mi. The app grabs 48 MiB all at once at startup, so the container dies from OOM. Once restarts are visible, run a 20-second load and save it with python3 /opt/fixtures/chaos_lab.py record mem-squeeze /root/chaos-lab/raw-06.json /root/chaos-lab/06-mem-squeeze.json.

This rollout does not finish. Do not wait for rollout status; look at the restart count or the termination reason. How the exit code and reason differ from the CPU case is the heart of this experiment.

Reflect what you learned in the design and take the same attack again

Restore the resource limits to the normal values (cpu 500m · memory 96Mi), and this time add shutdown handling. Use a preStop hook to buy at least 3 seconds and raise terminationGracePeriodSeconds to 10 or more. When the rollout finishes, repeat the same attacks as in steps 3 and 4 and save it with python3 /opt/fixtures/chaos_lab.py record hardened /root/chaos-lab/raw-07.json /root/chaos-lab/07-hardened.json.

preStop runs before the TERM signal, and the signal goes out only when the hook finishes. Endpoint removal propagates in between. The grace period countdown starts before the hook, so it must be more generous than the hook. Removing the probes or adding more replicas to make the numbers look better is an act of dodging the contract of this experiment.

Compare hypothesis and measurement and close the experiment note

Name the numbers of the five records according to the judgment table in the instructions and write them in /root/chaos-lab/conclusion.json. Each experiment needs observed_availability, observed_latency, a verdict (confirmed or refuted) compared with the hypothesis, and what you will change based on this result (action). The final grading looks at the earlier records and the current cluster together — the fixed design must still be alive.

A refuted hypothesis is not a failure. If it differs from your expectation, writing refuted is the right answer. If you write only plausible-looking numbers and name them carelessly, the grader recomputes with the same rules and filters them out. The five actions must be different from one another.