Test report¶

• Date: 2026-09-02 to 2026-09-03
• Scope: one isolated x86_64 Kubernetes worker
• Result: native, launcher, Kata, snapshot, concurrency, agent, observability, and jailer paths passed with documented limitations

This report intentionally omits cluster names, internal addresses, registry names, bastion details, credentials, and unrelated workloads so it can be published independently.

Environment¶

Artifact integrity¶

The official Firecracker release checksum passed before installation.

e2fsck -fn completed successfully on the base rootfs. The Kata archive hash matched before and after the offline transfer. Its GitHub artifact attestation was not verified because the workstation's Sigstore verifier did not initialize, so the digest is reported only as transfer-integrity evidence.

Native results¶

The timing is a smoke observation, not a benchmark. It includes the old quickstart guest's init path and was not repeated under controlled load.

Kubernetes launcher results¶

Kata + Firecracker RuntimeClass results¶

The Kata 4.1.0 static package was downloaded on a connected workstation, transferred through the Kubernetes API, and installed with its runtime-rs Firecracker configuration. Firecracker 1.16.1 from the first phase supplied the VMM and jailer binaries.

A dedicated 20 GiB data / 2 GiB metadata sparse loopback thin pool was created on the root filesystem. No raw disk or pre-existing LVM volume group was used. containerd's built-in devmapper plugin changed from its original unconfigured error state to ok during the test.

The four-second startup and memory samples came from one smoke run. They are not benchmark claims. Container-level metrics did not represent the host VMM's RSS, so a production capacity study must observe shim and VMM cgroups separately.

Compatibility findings¶

The first two sandbox attempts intentionally remain documented because they identify reproducible release/configuration problems:

1. default_maxvcpus = 0 expanded to the 56 host CPUs. Firecracker rejected the VM. A lab copy of the configuration bounded default_maxvcpus to 2.
2. The released Firecracker runtime-rs configuration set dial_timeout_ms = 45000 without a compatible reconnect_timeout_ms. The runtime rejected it. The dial_timeout_ms = 2000 and reconnect_timeout_ms = 60000 workaround from upstream issue #13484 allowed the sandbox to start.

No orphan Firecracker process remained after either failed attempt. These overrides must be re-evaluated when upgrading Kata rather than copied forward blindly.

Retained environment validation¶

After the rollback path had been verified, the node was deliberately enabled again as a dedicated Firecracker test worker. A systemd oneshot service now attaches the sparse data and metadata files, creates fc-devpool, and is a required predecessor of containerd.

The dependency was tested with no Kata Pod running:

1. containerd and the pool service were stopped;
2. dmsetup and both loopback lookups returned empty;
3. starting containerd automatically started the pool service;
4. fc-devpool returned and the devmapper plugin reported ok.

Destroying the earlier pool while retaining containerd's content store left two stale containerd.io/gc.ref.snapshot.devmapper labels. The first retained workbench therefore failed with snapshot does not exist. With no devmapper snapshot or Pod active, those labels were cleared and containerd restarted. The next workbench creation unpacked both sandbox and workload images into the new pool and became Ready.

A live containerd restart was then run while that workbench remained active. The Pod UID and Firecracker PID were unchanged, container restart count stayed at zero, guest uptime increased continuously, and a file under the Pod's emptyDir remained readable.

Selected follow-up experiment results¶

1. Full snapshot and restore¶

The single restore was about 49 times faster to ICMP readiness than the single cold baseline. This is not a benchmark: repeated immutable disk clones and multiple controlled samples were not run.

2. Kata concurrency¶

At peak, 11 Firecracker processes including the retained workbench used about 1.45 GiB aggregate RSS. After batch deletion there were no extra VMMs or devmapper snapshots.

3. Agent applications¶

OpenClaw 2026.8.2, DSH 0.1.2-alpha.4, Hermes 0.21.0, and Codex 0.150.1 all ran as Ready, zero-restart Kata/Firecracker Pods.

The test worker could not directly route to the model endpoint and the Kata TAP path could not use the normal Service VIP. A temporary relay and direct Pod addressing were used for qualification, then the relay was removed. Credentials were injected from Kubernetes Secrets and are absent from the manifests, screenshots, and report.

5. Observability¶

At one observation point, the four payload containers used 1,094 MiB in total, while five host Firecracker processes including the retained workbench used approximately 2.68 GiB aggregate RSS. This confirms that payload-only Kubernetes metrics undercount the complete runtime footprint. The Adapter reported zero active leases after application tasks.

7. Native jailer baseline¶

A separate 2-vCPU, 256-MiB microVM was launched with jailer 1.16.1 and retained for follow-up testing. It used UID/GID 65534, a per-instance chroot, a new PID and mount namespace, CPU shares of 64, and a nofile limit of 128. After VM start, every observed VMM thread had seccomp mode 2; the main process had NoNewPrivs=1 and zero effective capabilities.

Final node state¶

• the worker is online and schedulable, with a dedicated sandbox.aik8s.run/kata-fc=true:NoSchedule taint preventing ordinary workloads from using it;
• the CNI is Ready on the worker;
• RuntimeClass/kata-fc-lab remains installed and selects only labelled test nodes;
• namespace firecracker-lab and its long-running kata-fc-workbench Pod remain active for interactive testing;
• the four agent application Pods remain Ready on the dedicated worker for inspection, but the temporary model-network relay has been removed;
• the native jailed security-smoke microVM remains active for security follow-up;
• containerd, CRI, devmapper, kubelet, and the persistent pool service are active; the pool service is enabled for boot and ordered before containerd;
• the original containerd configuration remains backed up with its recorded SHA-256, while the active configuration contains the kata-fc handler;
• the base Firecracker lab, Kata files, and compressed offline archive remain on disk for repeatable tests;
• temporary artifact-transfer resources were removed;
• no credentials or private endpoints were written into this documentation.

Follow-up work¶

1. Replace the loopback thin pool with dedicated production storage, persistent activation, monitoring, and recovery procedures.
2. Track Kata issue #13484 and remove the timeout workaround after a verified upstream fix.
3. Validate persistent volumes, NetworkPolicy, resource pressure, abnormal kubelet/containerd failures, and concurrent deletion.
4. Mirror pinned Kata, Firecracker, guest, and workload artifacts through the organization's approved supply-chain verification path.
5. Replace the disposable Ubuntu 18.04 quickstart guest with a maintained, reproducibly built image.
6. Extend the bounded ⅕/10-Pod smoke to repeated 50/100-Pod concurrency and compare it with CubeSandbox using equivalent readiness criteria.
7. Fix Service VIP reachability from the Kata TAP network and replace the temporary model relay with a production egress path.