Test Report¶

• Date: 2026-09-02 through 2026-09-03
• Scope: one isolated x86_64 Kubernetes worker
• Result: native, Launcher, Kata, snapshot, concurrency, Agent, observability, and jailer paths all passed; limitations recorded

For independent public release, this report intentionally omits cluster names, internal addresses, image registry names, jump-host details, credentials, and unrelated workloads.

Test Environment¶

Artifact Integrity¶

Before installation, the official Firecracker release packages were verified via checksums.

The base rootfs successfully completed e2fsck -fn. The Kata archive had identical hashes before and after offline transfer. Because Sigstore verifier initialization failed on the workstation, GitHub artifact provenance was not verified; the digests here therefore serve only as transport-integrity evidence.

Native Run Results¶

These timings are smoke observations, not benchmark results. They include the legacy quickstart guest initialization path and were not repeated under controlled load.

Kubernetes Launcher Results¶

Kata + Firecracker RuntimeClass Results¶

The Kata 4.1.0 static package was downloaded on a networked workstation, transferred via the Kubernetes API, and installed using its runtime-rs Firecracker configuration. The Firecracker 1.16.1 from the first-phase installation provides the VMM and jailer binaries.

A dedicated 20 GiB data / 2 GiB metadata sparse loopback thin pool was created on the root filesystem; no raw disks or pre-existing LVM volume groups were used. During testing, the containerd built-in devmapper plugin transitioned from the initially unconfigured error state to ok.

The four-second startup and memory sample come from a single smoke run and do not constitute benchmark conclusions. Container-level metrics do not represent host VMM RSS, so production capacity studies must observe shim and VMM cgroups separately.

Compatibility Findings¶

The first two sandbox attempts were intentionally kept in the report because they identified reproducible version/configuration issues:

1. default_maxvcpus = 0 expanded to all 56 host CPUs, causing Firecracker to reject the VM. The experimental configuration copy restricted default_maxvcpus to 2.
2. The release Firecracker runtime-rs configuration set dial_timeout_ms = 45000 without a compatible reconnect_timeout_ms, so the runtime rejected the configuration. After applying the workaround from upstream Issue #13484, using dial_timeout_ms = 2000 and reconnect_timeout_ms = 60000, the sandbox started successfully.

Neither failed attempt left orphaned Firecracker processes. These override configurations must be re-evaluated when upgrading Kata; they must not be carried over blindly.

Reserved Environment Verification¶

After the rollback path was verified, the node was intentionally re-enabled as a dedicated Firecracker test worker. A systemd oneshot service now mounts the sparse data and metadata files, creates fc-devpool, and acts as a required prerequisite service for containerd.

The dependency was tested without Kata Pods running:

1. Stopped containerd and the pool service;
2. dmsetup and both loopback queries returned empty;
3. After starting containerd, the pool service started automatically;
4. fc-devpool was restored, and the devmapper plugin reported ok.

After destroying the previous pool but keeping the containerd content store, two containerd.io/gc.ref.snapshot.devmapper labels remained, so the first reserved Workbench creation produced snapshot does not exist. After confirming there were no active devmapper snapshots or Pods, these labels were cleared and containerd was restarted. On the next Workbench creation, both the sandbox and workload images unpacked successfully into the new pool, and the Pod then became Ready.

containerd was then restarted online while the Workbench remained active. The Pod UID and Firecracker PID did not change, the container restart count stayed at 0, guest uptime continued to increase, and files under Pod emptyDir remained readable.

Subsequent Key Experiment Results¶

1. Full Snapshot and Restore¶

The single restore-to-ICMP-ready time was approximately 49 times faster than the single cold-start baseline. This is not a benchmark conclusion: this round did not execute repeated immutable disk clones or collect multiple controlled samples.

2. Kata Concurrency¶

At peak, including the reserved Workbench, there were 11 Firecracker processes with aggregate RSS of approximately 1.45 GiB. After deleting the batches, no extra VMMs or devmapper snapshots remained.

3. Agent Applications¶

OpenClaw 2026.8.2, DSH 0.1.2-alpha.4, Hermes 0.21.0, and Codex 0.150.1 all ran as Kata/Firecracker Pods, all Ready with restart count 0.

The test worker could not route directly to the model endpoint, and the Kata TAP path could not use the regular Service VIP. Admission testing temporarily used a relay and direct Pod connectivity; the relay was deleted after completion. Credentials were injected via Kubernetes Secrets and do not appear in manifests, screenshots, or the report.

5. Observability¶

At one sampling instant, the four payload containers used 1,094 MiB combined, while the five Firecracker processes on the host, including the reserved Workbench, had aggregate RSS of approximately 2.68 GiB. This shows that Kubernetes metrics counting only payloads underestimate full runtime overhead. After application tasks finished, the adapter reported zero active leases.

7. Native Jailer Baseline¶

A standalone 2 vCPU, 256 MiB microVM was started with jailer 1.16.1 and retained for subsequent testing. It used UID/GID 65534, a per-instance chroot, new PID and mount namespaces, 64 CPU shares, and an nofile limit of 128. After VM startup, every observed VMM thread was in seccomp mode 2; the main process was NoNewPrivs=1 with effective capabilities of 0.

Node Final State¶

• Worker is online and schedulable; the dedicated sandbox.aik8s.run/kata-fc=true:NoSchedule Taint prevents ordinary workloads from using it;
• CNI is Ready on the Worker;
• RuntimeClass/kata-fc-lab remains installed and only selects test nodes with the specified Label;
• The firecracker-lab Namespace and the long-running kata-fc-workbench Pods within it remain active and available for interactive testing;
• The four Agent application Pods remain Ready and stay on the dedicated Worker for inspection, but the temporary model network relay has been removed;
• The security-smoke microVM started by the native jailer remains active for subsequent security testing;
• containerd, CRI, devmapper, kubelet, and the persistent Pool service are all in an active state; the Pool service has been configured to start at boot and is ordered before containerd;
• The original containerd configuration still has a backup with its SHA-256 recorded; the current configuration includes the kata-fc Handler;
• The base Firecracker experiments, Kata files, and offline compressed archives remain on disk for repeatable testing;
• The temporary artifact transfer resources have been removed;
• This document contains no credentials or private endpoints.

Follow-up Work¶

1. Replace the Loopback Thin Pool with production-grade dedicated storage, and configure persistent activation, monitoring, and recovery procedures.
2. Track Kata Issue #13484 and remove the timeout workaround after the upstream fix has been verified.
3. Validate persistent volumes, NetworkPolicy, resource pressure, kubelet/containerd abnormal failure, and concurrent deletion.
4. Synchronize pinned versions of Kata, Firecracker, Guest, and workload artifacts through an organization-approved supply chain validation path.
5. Replace the one-off Ubuntu 18.04 quickstart Guest with a continuously maintained, reproducibly built image.
6. Extend the bounded ⅕/10 Pod smoke tests to repeated 50/100 Pod concurrency tests, and compare against CubeSandbox using equivalent Ready criteria.
7. Fix Kata TAP network reachability to the Service VIP, and replace the temporary model relay with a production outbound link.