CubeSandbox failure drills — control-plane outages, VMM termination, and snapshot recovery¶

Sandbox reliability involves more than creation speed. Does a task continue when the API fails? Does a proxy outage merely make a live task unreachable? Can execution resume after the compute Pod is recreated? Will a recovered task repeat operations it already performed against an external system?

We injected component failures into an initially idle Kubernetes test installation of CubeSandbox. The formal results use a v0.7.3 Rust Shim verified against the actual executable. Of 12 planned profiles, 11 reached fault injection; the HTTP path-rule baseline failed, so its fault was not injected. Compute Pod recreation received an additional attempt, with the failure retained.

Three findings matter most. Existing tasks generally continued during control-plane outages. After one virtual-machine runtime was killed, an explicit full-snapshot restore recovered its saved execution state. However, recreating the compute Pod in this installation's Pod-network mode stopped both original tasks, and subsequent recovery encountered port-map conflicts and NoResource. This configuration has not passed compute-Pod recovery acceptance and should not be deployed on the assumption of automatic recovery.

The scripts, sanitized results, and per-request receiver records are public. This is a component-failure and recovery experiment on one compute node, rather than validation of recovery across nodes, sites, or host power loss.

1. Verify which runtime was actually upgraded¶

Initially, component images were already tagged v0.7.3, but sandboxes created from an older template still used the node's retained v0.7.1 Rust Shim. CubeSandbox supports multiple component versions: templates and snapshots bind versions, so updating the current toolbox does not automatically move historical templates to the new runtime. Component multi-version mechanism

We created a separate template through the official OCI (Open Container Initiative) image-build workflow, retaining the same source image, startup behavior, and envd client service, and repeated the entire suite. We did not edit the historical template's component bindings in place.

VMM means Virtual Machine Monitor. Before injection, the script required the process to identify this sandbox, execute from the component-version directory, and hold an anon_inode:kvm-vm handle. This installation had no separate cube-hypervisor process; selecting a process by an assumed name would have targeted the wrong boundary.

Both binaries returned development placeholders for --version. Version attribution therefore combines the component manifest, actual executable path, and file hash. An image tag or CLI version string alone was not treated as runtime proof.

2. Identify shared failure boundaries¶

flowchart TB
  A["Client and observers"] --> B["CubeAPI / CubeMaster / CubeOps"]
  B --> C["cube-node compute Pod<br/>Cubelet and runtime management"]
  A --> D["CubeProxy<br/>Command and file access"]
  D --> E["Rust Shim / VMM<br/>Guest task and memory"]
  C --> E
  E --> F["CubeVS kernel forwarding and IP restrictions<br/>CubeEgress HTTP proxy"]
  F --> G["Independent receivers A / B"]
  B --> H["MySQL / Redis / MinIO<br/>Metadata and artifact dependencies"]

Arrows represent functional relationships, not a requirement that every packet traverse every component in sequence. In particular, outbound business requests from the guest and client commands through CubeProxy take different paths.

The cube-node installation used Pod networking and hostPID: true. PID 1 visible inside the container was a host process: killing it would not be a legitimate container-exit simulation. Target selection used the PID, Linux process start time, command hash, container cgroup, and KVM handles. An independent watchdog backed up recovery from Cubelet and Nginx process suspension.

2.1 Conditions held constant across cases¶

Each case ran one already-created probe sandbox, except the VMM and compute-Pod cases, which ran two. During faults, the workload did not continuously create or delete sandboxes, take snapshots, or perform model inference: it advanced an in-memory counter and sent short HTTP requests. Sandboxes were created with a 900-second timeout. These were short interruptions, not expiration-and-reaping tests.

Only the specified component was stopped in each case. CubeAPI, CubeMaster, CubeOps, and CubeProxy each had one replica when targeted; the lifecycle-management components retained their original deployment. Dependency outages preserved the original persistent volumes. The compute host, cluster network plugins, and receivers were outside the injection scope.

Every formal case used the new template and verified v0.7.3 executable, with the temporarily aligned egress CIDR configuration described earlier. Historical v0.7.1 records and configuration-debugging attempts remain separate. Template bindings and configuration changed, so these records cannot establish a performance gain between versions.

2.2 Network baselines: IP targets and HTTP paths are different cases¶

In IP-allowlist cases, A and B were the IPv4 addresses of two receiver Pods, both on port 8080. The sandbox used allow_internet_access=False, with only A in network.allow_out. Each guest iteration visited A and B, with a 0.8-second request timeout and a 0.35-second interval. Connection reuse and environment proxies were disabled.

The HTTP path-rule case instead visited port 80 on the same receiver Pod A: /probe/allow* was allowed, other paths on the same host were denied, and /probe/deny was the negative target. Here, “allowed / denied” means different paths, not different Pods. HTTPS decryption, domain rules, and TLS (Transport Layer Security) handshakes were not tested.

A separate environment preflight obtained HTTP 200 from both A and B when both were allowed, then checked the A-only restriction. That preflight did not record runtime hashes equivalent to the formal matrix and cannot stand in for an unrestricted control repeated in every v0.7.3 case. The formal runner actually performs a restricted baseline per case: observe for at least 10 seconds and require at least 3 A receipts and 0 B receipts before proceeding. Zero B receipts therefore means no bypass was observed in this traffic, not a security proof covering all bypass paths.

2.3 What each probe establishes¶

API and command probes run concurrently within each sample, followed by a 0.6-second delay between samples. Timeouts reduce sample counts, so the denominators do not represent an identical sampling rate. Recovery polling requires 3 consecutive successes for API, commands, and identity, with a maximum 75-second wait. Counter advancement and receiver traffic need separate inspection; that boolean predicate does not cover all business semantics.

The result field remaining_sandboxes is read after old-task cleanup and before the new smoke-test task is created; emptiness is checked separately at the end of the entire run. MEASURED means the measurement completed, not that every recovery acceptance passed. Monitoring stage 3 means the test workflow ended and can also occur for failed cases.

2.4 Where measurement windows begin¶

The pre-fault receipt window runs from baseline to fault_issued. The interval from fault_issued to fault_confirmed is an action transition and is excluded from fault receipt counts. Only fault_confirmed to restore_issued is the reported fault window. Witness and snapshot preparation extend the pre-fault window in VMM and Pod cases. The fault window also includes the final probe and witness read, so it can exceed the 14-second observation budget and is not the full business-outage duration.

Guest requests and receiver recording continue through action transitions, but the controller does not continuously sample API and commands there, so transient availability is not established. Monitoring stage 1 starts at fault_issued and includes that transition, whereas tabulated fault counts start at fault_confirmed; the plotted stage length cannot be used directly as their denominator. Stage 2 only means the program entered recovery steps, not that the task has recovered.

All values below are seconds, derived from adjacent timestamps in the public results and rounded to two decimal places. For process signals, “confirmed” means exact identity verification and a successful signal call. Scaling cases wait until the matching Pod list is empty. Pod recreation waits for a different UID to reach Running. These are different confirmation strengths.

Receipt counts help establish task continuity, not component throughput. A 2-second scrape and 8-second rate window delay or smooth curve edges; they are not exact exit timestamps. There is one formal attempt per profile and two for the Pod. These records do not provide repeated-run distributions, long-term availability, or recovery-duration P95 / P99 values.

3. Control-plane outages separate execution from management¶

Cases ran sequentially, with recovery and acceptance checks before the next fault. The Cubelet case suspended a process; it did not demonstrate automatic recovery from a crash and restart.

3.1 CubeAPI: stop the sole API entry point, retain existing task access¶

3.2 CubeMaster: stop management while observing an existing task¶

3.3 CubeOps: briefly stop operational metadata management¶

3.4 CubeProxy: interrupt command access while observing guest egress¶

3.5 Cubelet: suspend and resume the same process¶

Actual monitoring for cube-proxy-r073a01. The allowed receiver continued receiving requests while the command channel fell to 0, then returned to 1. The counter shows only successful command reads; a gap means the counter could not be read, rather than that execution stopped.

For incident response, “the sandbox is unreachable” needs a more precise diagnosis: did the guest stop, did its outbound path break, or did the access proxy fail? API responses alone can misclassify a live task as dead. Receiver traffic alone can conceal the fact that users can no longer execute commands.

4. Kill one VMM: recovery returns to the snapshot point¶

This case added a witness sandbox that received no fault. The main task held its counter steady, completed a full snapshot, resumed execution, and then had its actual Rust Shim killed.

The restored guest execution state belongs to a newly created sandbox object. A successful public task_recovered value does not mean that the original runtime process or sandbox ID came back automatically. The witness checks whether one VMM failure spreads to another task on the node; it is not in an independent failure domain.

The main task's receive rate fell to 0, commands failed, and successful reads reappeared after explicit recovery. The exact 39 → 46 → 39 sequence comes from raw guest-state records. Metrics sampling is not a millisecond recovery timer.

The full snapshot preserved guest memory and process state, rather than simply restarting a script that reread a file. However, restoration returned to 39 instead of the last pre-kill value of 46. Progress after the snapshot was not automatically retained.

The independent receiver also recorded 6 previously received allowed requests, sequence numbers 40–45, replayed after recovery. It was outside the snapshot and did not roll back with the guest. An agent that had submitted an order, sent a message, or written an external database could therefore repeat an already completed side effect.

Task state and business idempotency still need their own design: persist idempotency keys for external writes, reconcile completed steps after restoration, and let the task system decide which operations to retry. Restoring memory does not restore the external world.

5. Compute Pod recreation stopped tasks and defeated recovery¶

This case ran both a subject and a witness, then deleted their cube-node Pod. The replacement had a different UID, both original Rust Shim processes were gone, and neither guest command channel worked. The main task had 0 / 0 A / B receipts in the fault window. Subsequent automatic-recovery polling did not observe the main task recover.

The replacement reached Running after 41.51 seconds in r073a01 and 44.14 seconds in r073a02, counted as action transitions. A working API list afterward did not establish an executable original guest: each fault window had only 4 command probes, all failed. Treat replacement-Pod startup, original-task recovery, snapshot-based creation, and fresh-template creation as four separate acceptance points.

The first explicit snapshot restoration failed with this sanitized error structure:

network runtime EnsureNetwork failed:
ifindex/listen_port 4/49983 already maps to host port 20078,
cannot map to 20000

ifindex is a Linux network-interface index. The new Pod had a changed network namespace and interface lifecycle, while host-pinned CubeVS port maps retained old entries. A new allocator started at lower ports, and the implementation refused to overwrite a conflicting mapping. v0.7.3 port-map conflict checks

The first attempt's snapshot was subsequently removed by test cleanup. Attempt r073a02 therefore explicitly retained its failed snapshot. It again failed to recover automatically, and its new-task check hit the same class of mapping conflict. The first failure cannot be dismissed as an isolated event.

New-task creation recovered; the original task did not¶

After verifying an empty API inventory and no live CubeSandbox VMMs, we backed up and checked two dedicated maps, then briefly suspended Cubelet writes. Each repair removed 4 orphan entries from each map:

/sys/fs/bpf/local_port_mapping
/sys/fs/bpf/remote_port_mapping

This was an idle-installation repair with no active CubeSandbox virtual machines, not a generic online repair procedure. No eBPF programs were detached, /sys/fs/bpf was not cleared, and the cluster CNI (Container Network Interface), OVS (Open vSwitch), and unrelated virtual-machine networks were unchanged.

Creation, execution, and deletion worked after cleanup, but restoring the second retained snapshot still returned NoResource. Snapshot and replica metadata reported READY, while the replica address still referred to the pre-recreation Pod. This is a lead for investigating placement and replica registration, not a root cause proven by a successful corrective experiment. NoResource should not automatically be read as insufficient machine memory.

The result is therefore: the original task was not recovered; the environment's new-task path was restored. We did not edit database records, bind an obsolete address, or force reuse of a vanished network namespace to produce a success screenshot.

The retained snapshot's cleanup request also dialed the old Pod address and timed out. Its cleanup remains outstanding pending investigation of replica registration; it does not consume a running virtual machine. This outstanding item is included in the public results.

The same-version official upgrade guide already warns that Big Pod recreation in Pod-network mode interrupts sandbox networking without self-healing and requires draining first. hostNetwork changes the network namespace lifecycle, but the supported isolation, drain, and validation process still applies. Switching that setting alone is not evidence of high availability. Kubernetes upgrades and networking mode

6. Dependency recovery requires more than a working list page¶

The two experiments ran separately in sequence. API-list and command probes represent those specific operations, not the recovery of every create, delete, or snapshot operation.

6.1 MySQL: single-instance service outage with its volume preserved¶

6.2 Redis: preserve the volume and distinguish timeouts from resource state¶

6.3 MinIO: stop object storage and observe cached local execution¶

The MySQL result shows that Pod readiness and a lightweight list probe do not establish that name resolution, connections, and all state changes have recovered. The Redis result illustrates that a timeout does not prove the server performed no action. Repeated create or delete calls need resource-state checks.

The MinIO case covered a cached running task and basic operations after recovery. It did not attempt a cold restoration requiring remote object reads while storage was unavailable. It therefore provides no evidence for object-storage high availability or S3-compatible snapshot disaster recovery. S3 here means the Simple Storage Service-compatible object-storage interface.

7. Validate IP restrictions separately from HTTP path rules¶

7.1 Suspend CubeEgress Nginx and observe the IP-restricted path¶

Layer-3 IP allowlists and layer-7 HTTP path rules follow different enforcement paths. Suspending the CubeEgress Nginx master and workers produced 39 A receipts and 0 B receipts in the IP-allowlist fault window. This establishes continuing allowed traffic and no observed receipts at the denied target under this configuration. Rules were not removed during the fault to make B reachable as a reverse control, so zero receipts alone do not attribute all blocking exclusively to CubeVS or establish HTTP path enforcement during proxy failure.

7.2 HTTP path rules: baseline failure, no fault injected¶

The allowed request under HTTP path rules timed out before injection, so the profile is BLOCKED_BASELINE and no fault was injected. Investigation found a mismatch between the guest network range and egress listener configuration. Two environment variables were temporarily aligned, but the allowed path still failed with the new runtime. Formal records identify that temporary aligned configuration; original values were restored at the end.

The boundary matters: the IP-rule case was verified, HTTP proxy rules were not shown to work, and universal fail-closed behavior was not established. DNS, IPv6, established long-lived connections, and rule-loading intervals require separate cases.

8. Official UI and actual execution evidence¶

Actual official Console with Mock mode off. Visible masks redact internal node, template, and environment identifiers. This sandbox was created for final command and creation acceptance and removed after capture. It is not evidence that the original task survived the Pod fault.

This is the accompanying custom evidence viewer, not the official Console. It separately presents successful explicit VMM snapshot restoration, failed compute-Pod recovery, and the failed HTTP path baseline. Failed profiles remain failed.

9. Make the drill a repeatable recovery acceptance process¶

The program records every case as a separate attempt, including code and executable hashes, action times, original errors, and receiver records. Private configuration, full manifests, snapshot identifiers, and access tokens stay in a local ignored directory. Public results are exported through an explicit field allowlist.

# 配置中使用自己的专用测试安装和两个接收端，先完成目录 README 的预检。
python3 examples/cubesandbox-fault-drill/watchdog.py \
  --config .local/cube-drill/private/settings.json

# 在另一终端逐项执行；不要并发运行故障场景。
python3 examples/cubesandbox-fault-drill/run_drill.py \
  --config .local/cube-drill/private/settings.json \
  --case vmm --attempt a01

The comments above preserve the original test commands. Configure a dedicated installation and both receivers, complete the directory README preflight, start the watchdog, then run cases sequentially in another terminal.

A useful recovery acceptance process has four checks:

1. Processes and interfaces: component readiness, the actual runtime, survival of original processes, and interface or namespace changes.
2. Tasks and traffic: identity and counter continuity, received allowed traffic, and no receipts at the denied target.
3. State changes: verify creation, execution, file operations, deletion, and snapshots separately instead of treating a list endpoint as all management functionality.
4. Business consistency: measure progress lost since the snapshot, request replay, external idempotency, and reconciliation.

This experiment exercised listing, commands, creation, deletion, snapshots, and state verification. Guest file APIs installed the probes; this is not a complete matrix of filesystem failures and volume recovery.

Production planning should establish compute-node isolation and draining, a stable network lifecycle, and independent metadata and snapshot backups before promising cross-node recovery targets. Local XFS snapshots require the origin node. Shared S3 snapshots additionally depend on completed synchronization, compatible compute nodes, and real read availability, all of which need their own drill. Cross-node snapshot conditions

No host reboot, power loss, cross-node, or cross-availability-zone recovery was performed. Original component replica counts and temporary environment variables were restored, test sandboxes were removed, and all five pre-existing virtual machines retained their UIDs and remained Running / Ready. The new template was retained for future validation.

References and companion material¶

• CubeSandbox v0.7.3 source
• Reproduction code and sanitized JSON / CSV
• CubeSandbox process-memory snapshot experiment
• Microsandbox, BoxLite, and OpenSandbox failure experiments