Sandboxes in LLM Training: From Agent Reinforcement Learning to Kubernetes Environment Engineering¶

How do we know whether code generated by a model actually runs? How can we tell whether a coding agent fixed a repository issue? Training systems increasingly let models act in executable environments, run commands, observe results and adjust behavior based on feedback. A sandbox supplies a controlled execution space; the task environment and verifier define available actions and determine whether the task is complete.

A training sandbox is environment infrastructure, not a model-training algorithm that replaces the GPU trainer. It is particularly useful for code verification, tool use, software-engineering agents and browser-use training. Ordinary text pretraining generally does not require a sandbox for every sample; a sandbox enters the picture when a training workflow needs to execute code, tools or interactive environments.

This survey draws on public material available as of 2026-11-09. Its diagrams and numbers are illustrative; it reports no local training, benchmark or security certification. For background, see Agent RL: veRL and Miles and Agent sandbox selection.

1. A code-repair task as an example¶

Suppose a model must fix a billing function in a Python project. The environment prepares the repository, dependencies and tests. The model reads code, edits files, runs tests and uses failures to guide another attempt. An independent verifier ultimately checks whether the result satisfies the task.

flowchart LR
  T[Task and environment template] --> P[Sandbox control plane]
  P --> E[Isolated execution environment]
  L[Model / agent loop] -->|Code and tool calls| E
  E -->|Output and state| L
  L -->|Trajectory| V[Independent verifier / reward]
  E -->|Final result| V
  V -->|Reward and trajectory| TR[Trainer updates parameters]
  TR -->|Updated model| L

This conceptual flow does not require synchronous execution or prescribe where components run. A rollout is a sampled interaction trajectory and may contain multiple model outputs, tool calls and environment observations. A trainer can update parameters using trajectories and rewards; the same infrastructure can instead generate data or run evaluations without updating weights. Code execution can be a tool in a multi-turn interaction or a final-answer verifier, depending on the configuration. verl Sandbox Fusion integration

An environment need not always be an independent VM: a simulator or controlled API can provide state and feedback. Conversely, a container that executes commands does not automatically supply tasks, verification or rewards.

2. Three common uses¶

2.1 Verifying generated code¶

After the model outputs a function, a verifier runs tests in an isolated environment and collects exit codes, standard output, timeouts or exceptions for rule-based scoring. This can be a single-turn task: the sandbox executes and verifies code without maintaining a multi-turn session. verl provides a Sandbox Fusion example; confirm interfaces, versions and deployment against the target release. verl example

2.2 Training software-engineering agents¶

Given a repository and issue, an agent can search, edit and test over multiple turns. The environment must retain file changes within the same trajectory, and independent tests or other rules evaluate the final outcome. SWE-Gym supplies tasks and environments for software-engineering agents; it occupies a different layer from the underlying isolation runtime.

Verifiers need protection against reward hacking: if the agent can edit final tests, read reference answers or change the score directly, a passing result can be misleading. Protect hidden tests and scoring credentials, and where needed run final verification in a separate boundary the agent cannot write to. Also inspect patch quality, resource use and timeouts rather than relying on a single success flag.

2.3 Browser and computer use¶

An environment can provide a browser, web pages, mouse and keyboard or desktop to operate a simulated system. A verifier checks business state, such as completion of a simulated form. Separate read-only pages, reversible test data and real external systems so that training cannot send real messages or execute real transactions. Reset must cover accounts, cookies, caches and server-side data; restarting a browser alone does not guarantee a clean state.

3. How data enters training¶

In agent reinforcement learning (Agent RL), the model produces actions under its current policy, the environment executes them and returns observations, and the sampler records trajectories including tool input/output and tokens. Rewards can come from tests, rules, models or combinations; the trainer uses the selected algorithm to compute advantages and losses and update weights. Not every approach uses the same algorithm, and not every tool call needs an immediate reward. verl agent-loop and reward extensions

Keep the task and environment version, model and sampling configuration, tool calls, final artifacts, verifier version, reward and failure reason. Separate training from evaluation data; keep hidden tests outside model-accessible directories so leakage is not mistaken for generalization. Report success, incorrect answers, timeouts, environment failures and unexecuted cases separately. Preserve initial attempts when retrying infrastructure failures instead of reporting only eventual successes.

3.1 Public industry examples: which stages actually used environments?¶

Public reports disclose different levels of detail. Some describe sandbox implementation explicitly; others mention only executable environments or compiler-based tests. The evidence below distinguishes these cases: code verification alone does not establish a particular sandbox implementation, and post-training is not presented as base pretraining.

Kimi K2: one environment infrastructure supports data synthesis and RL¶

The Moonshot AI Kimi K2 report describes combining stateful simulated tool environments with real execution sandboxes when synthesizing SFT data, especially for coding and software engineering where real feedback matters. Trajectories can be filtered using objective results such as test pass rates. Its RL section describes constructing software-engineering environments from issues, pull requests and executable tests; the report says its Kubernetes sandbox infrastructure supports over 10,000 concurrent instances.

Sandboxes can therefore support both generating or filtering training data and RL interaction and verification, but those stages use different training signals. The public number is the team's reported instance concurrency, not tasks per second, GPU utilization, or a capacity another cluster will achieve. It does not prove that every sample in base pretraining ran in a sandbox.

Qwen3-Coder-Next: executable tasks for mid-training and post-training¶

The Qwen3-Coder-Next report describes constructing software-engineering tasks from executable repositories and tests, including controlled bug injection, resulting in about 800,000 verifiable task instances. Its internal MegaFlow organizes each coding task into agent rollout, evaluation and post-processing. Rollout usually co-locates the agent and execution-environment containers in one Pod; evaluation uses a dedicated container.

The report calls the coding- and agent-focused stage starting from an existing base model mid-training; it includes multi-turn trajectories generated and filtered from executable tasks, followed by SFT and RL. This is an example of execution environments helping produce mid-training data and supporting post-training, not proof that every pretraining token triggers a sandbox. The report also observes limited transfer between some agent scaffolds, making diversity of environments and tool formats an important evaluation dimension. The 800,000 figure is the reported number of task instances, not concurrent sandboxes.

OpenAI: sandboxing as one layer of training security¶

OpenAI's October 2026 frontier-model training environment security report describes restricted Linux containers inside microVMs for code execution and computer-use tasks, combined with network controls, cloud identity and monitoring. Tasks requiring accelerators use GPU-backed containers and user-space sandboxing among the described approaches. It discusses boundary-crossing risks in training and evaluation and says frontier-model RL training resumed after hardening.

The lesson is defense in depth: outer network and identity controls should still constrain access if one layer fails. The report does not disclose complete resource configurations for every training task or establish the same sandbox for all pretraining and post-training.

DeepSeek and Open R1: do not mistake a reproduction for internal architecture¶

The DeepSeek-R1 report uses compilers and predefined test cases to illustrate correctness feedback for coding problems in RL reward design. That explanation does not identify a specific isolation product or architecture. The independent Hugging Face Open R1 reproduction explicitly supports executing generated code through sandbox providers and scoring test results.

Together these examples explain how verifiable rewards can be implemented, but Open R1's choices cannot establish DeepSeek's internal sandbox design.

Lessons for platform teams¶

1. Identify the stage and signal: trajectory synthesis, mid-training, SFT and online RL place environment calls differently; “training used a sandbox” is too vague.
2. Build environments, verifiers and trainers as distinct components: version images, tests, permissions and rewards; separate final-verification write access from the agent environment.
3. Measure both safety and throughput: instance concurrency, task completion, P95 creation latency and useful trajectory production are different metrics; layered isolation and network controls also require testing.

4. Combining open-source components¶

These components can be combined and may overlap. Interfaces and compatibility change across releases; do not assume they can be swapped without integration work. Agent Sandbox supports different underlying runtimes, and its quickstart explicitly presents gVisor and Kata as optional stronger isolation configurations: installing the controller does not automatically establish a VM boundary. Official quickstart

5. Kubernetes architecture and choices¶

A design can place trainers and model-serving workloads in GPU pools, while running the environment control plane and many sandboxes mainly on CPU nodes. Allocate GPUs separately when a sandbox must execute GPU or CUDA tasks. Availability depends on runtime, device passthrough and driver compatibility; do not assume every microVM or container isolation mechanism supports it.

These are typical characteristics, not an absolute security ranking or guarantee. Test escape risks, versions and configuration, patches, device access and multi-tenant boundaries for the actual solution.

A warm pool reduces cold-start waiting; it is not free capacity. Precreated containers or VMs can reduce image pulls and initialization but consume CPU, memory and disk. Before reuse, reset files, processes, credentials, network connections and application data. If clean reset cannot be demonstrated reliably, destroy and recreate the environment. Agent Sandbox offers templates, claims and warm-pool resources; do not treat example startup times as measurements for your own cluster. Agent Sandbox documentation

At high concurrency, the control plane can become a bottleneck. Frequent Pod creation, storage mounts, IP allocation, image pulls and status updates pressure the API server, scheduler, etcd, registry and node disks. Limit concurrency per node and tenant, smooth pool replenishment, set timeouts and garbage collection, and protect shared clusters with quotas and rate limits. GPU count alone is not a training-throughput estimate.

6. Operational metrics and capacity estimation¶

Start with end-to-end metrics: effective tasks completed, verified successes, cost per usable trajectory, and latency and failure rates for queueing, creation, execution and verification. Then inspect warm-pool hit rate, environment-creation P50/P95/P99, image pulls, CPU/memory/disk, leaked sandboxes, verifier waits and idle inference GPUs. Group latency by comparable task and environment version, separating cold starts from warm hits.

A rough example: for a target of 120 tasks per minute, with each occupying a sandbox for 30 seconds on average, the idealized average number of active sandboxes is 120 × 30 / 60 = 60, assuming stable arrivals, representative averages and no other bottleneck. Real capacity must cover bursts, tail latency, retries, warm-pool reserve and node failure; this is not a benchmark. If a task calls tools several times, use its total actual environment occupancy, not the number of tool calls as the task count.

Trace the whole pipeline. If inference GPUs are idle and many tasks wait for environments, inspect warm pools and scheduling. If environments are ready but wait for models, inspect inference throughput and concurrency. If tests are slow, inspect verifiers, disks and dependencies. Validate correlations over the same time window rather than attributing the issue to a single GPU utilization number.

7. A practical adoption sequence¶

1. Use a small set of non-sensitive coding tasks to validate “create environment → model actions → independent verification → cleanup” and retain raw trajectories.
2. Design isolation, secrets, outbound networking, verifier permissions and retention; conduct negative tests to confirm unauthorized behavior is blocked.
3. Measure cold starts, warm pools, concurrency, failure recovery and contamination cleanup; increase task volume gradually while observing Kubernetes control-plane and storage behavior.
4. After integrating training, report model success rates together with infrastructure failure denominators. Compare against a fixed-version evaluation set so throughput improvements do not come at the expense of valid sample quality.

This is a design and selection survey. Component feature lists cannot establish production readiness or training improvement. A later test-cluster evaluation should record versions, actual isolation, task sets, failures and resource consumption. The existing OpenSandbox practice and sandbox failure and network-policy practice are useful runtime background, but they do not replace a training-loop evaluation.