From Contiv to Kube-OVN: A Single-Node Migration Keeping the Physical VLAN¶

The difficult part of replacing a container network is often keeping the rules of the existing data center: the same business subnet, VLAN uplink, and physical gateway, while preserving host management connectivity.

This experiment selected one node from a Kubernetes test cluster running a customized Contiv implementation. The node first left the original cluster for an isolated Kube-OVN Underlay test, then rejoined it. After rejoining, the node became Ready. Two ordinary Pods used physical-network addresses, and bidirectional HTTP between the new Pods and existing Contiv Pods passed. Other nodes retained their original networking. The canary ran neither Contiv netplugin, Cilium, nor kube-proxy.

The migration account reconstructs archived observations. Monitoring was added to the still-running single-node environment on October 19, 2026, as described in Section 8.1, without switching the network again. Cluster names, node names, internal domains, IPs, MACs, VLANs, and internal paths have been replaced. Example addresses are not original addresses. Test counts, packet lengths, and success/failure records are preserved. Below, worker-canary is the test node and control-helper hosts the new network's control components.

1. What the Existing Network Had to Preserve¶

The original Contiv deployment was maintained internally. Its behavior cannot be inferred from the maintenance status of the community project. In this environment, Netmaster managed the shared address pool, host-level netplugin created Pod interfaces, and Open vSwitch (OVS) attached workloads to physical VLANs. Cilium chained onto the existing network to supply Service handling and other capabilities on the original nodes. See Contiv, OVS, and Cilium Chained Networking in Practice for that original path.

CNI stands for Container Network Interface; IPAM is IP Address Management; VLAN is Virtual Local Area Network. Here, Underlay means that Pods use the physical network's addressing and VLAN forwarding rather than a separate tunneled business-address space. Kube-OVN maintains logical networks through Open Virtual Network (OVN), with OVS executing forwarding.

The migration preserves the physical-network contract. It does not require identical bridges, flow tables, or Pod MACs. Contiv's original NORMAL switching path differs from the flow tables compiled by OVN. Sharing a VLAN does not imply identical control planes or recovery procedures.

flowchart TB
  subgraph OLD["Before: existing nodes"]
    direction LR
    O1["Pod eth0"] --> O2["Contiv veth<br/>OVS NORMAL"] --> O3["bond0<br/>business VLAN 200"]
  end
  subgraph NEW["After: canary node"]
    direction LR
    N1["Pod eth0"] --> N2["OVN logical network<br/>OVS br-int"] --> N3["localnet / br-biz<br/>bond0 · VLAN 200"]
  end
  OLD ~~~ NEW
  OLD --> PHY["Existing physical switches and gateway<br/>Pod source IP / MAC preserved"]
  NEW --> PHY

The diagram shows how both types of node reach the same physical network. Management interface bond0.210 retains its configuration but still shares the underlying Bond with business traffic; their failure domains are not fully separated.

2. Separate the Two Acceptance Stages¶

Installing the new CNI directly into the original cluster would combine controller scope, old sandbox cleanup, physical wiring, and address ownership into one problem. Instead, the drained node first became an isolated experimental cluster. Only after validating the network itself did it rejoin the original cluster.

The isolated stage initially enabled Underlay–Overlay Interconnection (U2O). This consumes an additional Underlay address and can affect Pod routing choices. To meet the final physical-gateway requirement, the default business Overlay subnet was removed, U2O was disabled, and the business Underlay became the default network. Ordinary Pods were checked to use the physical gateway again.

The internal join network remained for node-to-OVN connectivity. It was not an Overlay address pool for ordinary workloads. To determine whether business traffic is pure Underlay, inspect Pod addresses, routes, logical-switch localnet ports, and actual packets. A daemon argument containing geneve is insufficient on its own.

The isolated-stage “pure Underlay adjustment and acceptance” took 43.8 seconds. That excludes earlier node cleanup, image preparation, and Bond rewiring, so it is not a complete migration duration.

The handoff retained the original kubelet identity, runtime and CNI configuration, OVSDB state, interfaces, routes, and address-allocation records. Old Pods should exit while their old CNI service is still available so CNI DEL can reclaim their networks; the residual-sandbox problem encountered here is discussed below. Old network agents on the target were then stopped, and competing OVS management of the same physical interface had to be excluded before Kube-OVN took over. Replacing one CNI configuration file while two agents still manage the uplink is insufficient, and deleting the entire OVS database is no substitute for a scoped handoff.

3. Establish IPAM Ownership Before Starting New Pods¶

A tempting mistake is to select addresses that do not answer ping and hand them to the new CNI. A missing response only describes that observation; it does not prevent Netmaster from assigning the address elsewhere a moment later.

For this experiment, the original Netmaster's allocation API first created reservation endpoints for six addresses, marking them occupied in the shared pool. Only then could Kube-OVN allocate those six addresses. These reservations were control-plane records, not duplicate interfaces using the same IPs on the physical network. Reservation owners and the original allocation bitmap were checked individually.

The six positions are represented below with anonymized addresses:

10.244.8.18  10.244.8.29  10.244.8.33
10.244.8.38  10.244.8.43  10.244.8.57

The Kube-OVN Subnet retained the original /22 mask and physical gateway, using excludeIps to exclude everything else. Shrinking the Pod mask simply because only a few addresses are needed would change which destinations the Pod considers on-link.

Available capacity here means the deliberately restricted experimental pool, not free space across the physical /22. Extra replicas during rolling updates need addresses too. When all six slots are occupied, more ordinary Pods cannot simply be added.

Reservations work for a bounded experiment. Production should instead delegate non-overlapping ranges explicitly and define their reclamation lifecycle. Creating a Kube-OVN IPPool does not automatically exclude its addresses from Netmaster. Per-Pod integration with the old IPAM would also require idempotent allocation, concurrency handling, failure compensation, deletion/release, and reconciliation after controller restarts.

4. Wiring Choice: Why the Final Setup Used bond0 Directly¶

The same business VLAN can be tagged at different layers. The difference is who owns tagging and whether management interfaces need relocation.

Both can connect to the same physical VLAN. However, processing the tag in a Linux subinterface and then configuring another business tag in OVN cannot be combined blindly. A mismatch can cause double tagging or incorrect classification; identical VLAN numbers do not guarantee connectivity.

The original wiring already used OVS and bond0 for business traffic while retaining the management IP on a Linux management-VLAN subinterface. The second attachment kept that structure closest to the original. No additional bond0.200 was created, and switch configuration was not changed for this experiment. Kube-OVN Underlay interface and VLAN definitions

4.1 The Three Resources Have Different Responsibilities¶

ProviderNetwork selects the node's physical interface, Vlan binds the business tag to that provider, and Subnet defines Pod addressing, gateway, and allocation limits. The following is the anonymized structure of the applied configuration. It omits experimental authorization isolation and installation settings and is not a standalone installation bundle for an existing cluster.

apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
  name: biz
spec:
  defaultInterface: bond0
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: worker-canary
      example.net/cni-canary: "true"
  exchangeLinkName: false
  autoCreateVlanSubinterfaces: false
  preserveVlanInterfaces: false
  vlanInterfaces: []
---
apiVersion: kubeovn.io/v1
kind: Vlan
metadata:
  name: business-vlan
spec:
  id: 200
  provider: biz
---
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: business-underlay
spec:
  protocol: IPv4
  cidrBlock: 10.244.8.0/22
  gateway: 10.244.8.1
  vlan: business-vlan
  namespaces: [cni-canary]
  default: true
  logicalGateway: false
  u2oInterconnection: false
  natOutgoing: false
  enableLb: false
  disableGatewayCheck: false
  mtu: 1500
  excludeIps:
    - 10.244.8.1..10.244.8.17
    - 10.244.8.19..10.244.8.28
    - 10.244.8.30..10.244.8.32
    - 10.244.8.34..10.244.8.37
    - 10.244.8.39..10.244.8.42
    - 10.244.8.44..10.244.8.56
    - 10.244.8.58..10.244.11.254

logicalGateway: false preserves the physical gateway; u2oInterconnection: false disables business Underlay/Overlay interconnection; natOutgoing: false matches the goal of retaining Pod source IPs. Because Services were outside rejoin-stage acceptance, enableLb was false. Nor is default: true a per-node switch: its effective scope depended on the controller isolation described below.

4.2 preserveVlanInterfaces Does Not Mean “Leave Interfaces Untouched”¶

The name can be misleading. In the pinned version, the relevant implementation creates OVS internal ports and transfers addresses and routes from the original VLAN interfaces. Preserving network functionality does not mean preserving the original Linux interface's identity, address ownership, and wiring unchanged. VLAN interface implementation in 1.16.12

This experiment left that workflow disabled to retain the management VLAN subinterface. Before/after checks covered its ifindex, MAC, addresses, MTU, return route, and Bond members. This configuration worked on the tested node; it is not a universal guarantee for every Linux distribution, network-management service, or NIC combination.

Management and business traffic still shared the physical uplink, so OVS failures could affect remote management. A dedicated physical management interface and out-of-band console would provide a stronger recovery path than VLAN separation on the same Bond alone.

4.3 Follow the OVS Structure Through to the Wire¶

The wiring summary below replaces bridge names, port names, and VLAN identifiers:

Bridge br-biz
  Port bond0                     trunks: [0, 200]
  Port patch-localnet-to-br-int   type: patch

Bridge br-int
  Pod veth ports
  patch port to br-biz

Management: bond0.210             existing IP / route retained
Absent:     bond0.200             no new business VLAN subinterface

br-int executes OVN logical-network flow tables. A localnet connects the logical switch to the physical provider network, and patch ports connect the OVS bridges. Business packets then leave through VLAN 200 on bond0. For off-subnet destinations, the Pod's default route points to the original physical gateway.

The isolated pure Underlay stage captured the following ICMP headers. Only relevant fields are retained; IPs and MACs are replaced:

02:00:00:00:20:29 > 02:00:00:00:20:01, ethertype 802.1Q,
length 1518: vlan 200, ethertype IPv4,
10.244.8.29 > 10.244.8.1: ICMP echo request, length 1480

02:00:00:00:20:01 > 02:00:00:00:20:29, ethertype 802.1Q,
length 1518: vlan 200, ethertype IPv4,
10.244.8.1 > 10.244.8.29: ICMP echo reply, length 1480

Together with a 1472-byte ICMP payload, DF (Don't Fragment), and 3 packets sent / 3 received, this confirms that the tested path carried a 1500-byte IP packet. The 1518-byte Ethernet frame includes one 802.1Q tag and excludes the FCS (Frame Check Sequence). This does not establish all paths, all packet sizes, or sustained throughput.

5. Rejoining the Original Cluster: DaemonSet Placement Is Insufficient¶

DaemonSet selectors can limit node agents to one node, while the controller may still observe and reconcile cluster-wide Nodes, Namespaces, and Pods. Restricting where CNI is installed does not restrict which objects a controller processes. The pinned version's default-subnet selection also reads Pod and Namespace network annotations. Pod network handling in 1.16.12

Therefore, running the default full-cluster installer and then pointing to a one-node selector would not establish isolation. Before rejoining, this experiment built and tested a bounded experimental control plane.

flowchart TB
  subgraph CONTROL["control-helper: control plane only"]
    direction LR
    DB["Dedicated OVN database"] --- CTL["Kube-OVN controller"] --- CP["Scoped API gateway"]
  end
  subgraph API["Original Kubernetes API Server"]
    direction LR
    A["Real CA / service-account authentication"] --> R["RBAC + admission constraints"]
  end
  subgraph WORKER["worker-canary: only this node changes primary CNI"]
    direction LR
    NP["Scoped API gateway"] --- CNI["OVS / CNI node agents"] --- POD["Two Underlay Pods"]
  end
  CONTROL ~~~ WORKER
  CONTROL --> API
  WORKER --> API

The scoped API gateway was custom-built for this experiment. It is not an official Kube-OVN “single-node primary CNI mode.” Connections in the diagram show component relationships and API boundaries, not business-packet forwarding hops.

The gateway used the real API Server CA and projected ServiceAccount tokens. Components were not made to appear successful through fabricated API data. Nevertheless, the gateway depends on version-specific client behavior and remains a custom experiment mechanism. Admission rules also cover the defined resources and placement paths, not every privileged actor or arbitrary network plugin.

A taint alone is insufficient too: a DaemonSet may carry tolerations, and specifying nodeName bypasses normal scheduler placement. Actual placement was therefore inspected, and admission cases verified rejection of the two known Cilium placement paths.

The helper hosted the OVN database and controller using a dedicated data directory without taking over its NICs or OVS. Remote control-plane placement and one-node data-plane installation are separate properties that must both be checked.

6. Diagnosing NotReady: Permissions, Caches, and Protection Logic¶

The node remained NotReady during part of the rejoin. This involved several interacting causes: old runtime state, scoped API rules, and protective rollback behavior.

TLS means Transport Layer Security. Fixing cache startup required neither global Secret reads nor arbitrary Secret writes. After narrowing access by resource, name, and verb, both component startup and continued denial of out-of-scope requests were verified.

In the final recovery window, CNI became ready about 9 seconds after recovery began, the two ordinary Pods were ready after about 13 seconds, and the 12 node-local checks completed after about 15 seconds. These times exclude preceding diagnosis.

Node Ready indicates node health conditions, not Pod data-plane correctness. NotReady does not necessarily mean SSH has failed either. One NotReady period here followed the protection logic deliberately stopping kubelet and therefore its heartbeats.

A known issue remained: the controller periodically attempted to read the CertificateSigningRequest (CSR) cache and the scoped gateway denied it. OVN IPsec was disabled; global certificate reading and approval were not granted. This did not block the tested data paths, but it is another reason the custom gateway is not yet a production support model.

7. SSH Protection: Restore Access Before Resuming Anything¶

Business and management traffic share a Bond. A long-lived SSH session is insufficient protection: its survival does not prove new connections can authenticate.

The experiment used three layers:

1. Independent node guard: a system service checked the management IP, return interface and gateway, Bond mode and members, interface identity, MTU, and SSH listener. It did not depend on Kubernetes being healthy.
2. Fresh external authentication probes: the office client periodically established a new SSH authentication chain and wrote a heartbeat tagged with the current run identifier. An old open session was not accepted as proof of health. A heartbeat older than 45 seconds aborted the stage.
3. Independent stage deadline: a rollback deadline was armed before switching. A stuck installer or disconnected session could not wait indefinitely. Rollback was disarmed only after acceptance results were persisted and external probes were still fresh.

Rollback first marked the experiment aborted, stopped kubelet and the CNI agent that could reattach the uplink, detached the Bond from the experimental business bridge, and restored management addresses and routes. It did not resume the test or automatically reconstruct the complete original Contiv service.

A full return to the old network additionally requires restoring runtime identity, the sole primary CNI configuration, host network services, and the old plugin from backups, followed by fresh-Pod validation. Original IPAM reservations must remain until all Kube-OVN users of those addresses have exited.

The external SSH observation window during rejoin contained 90 samples: 89 successes and 1 failure. The failed authentication chain exited in roughly 0.232 seconds and later probes recovered. The evidence cannot distinguish jump-host authentication failure from target-node connectivity failure, so this cannot be described as uninterrupted SSH throughout.

8. What Final Acceptance Established¶

The figure below is generated from anonymized archived JSON, covering node-local acceptance and isolation checks. It is an experiment-record visualization, not a live dashboard or Grafana screenshot. Original internal files are not public attachments.

The anonymized acceptance data preserves both stages' counts, individual checks, and the SSH failure record.

Both rejoined-stage Pods used the physical default gateway and MTU 1500. Kube-OVN generated MACs different from the original reservation records. Bidirectional HTTP and gateway communication demonstrate that this physical network accepted those MACs, but switch port-security configuration was not inspected. Environments with MAC-count limits, static IP–MAC bindings, or DHCP snooping / ARP inspection must check those policies first.

Service / DNS success belongs to the isolated stage. The rejoined stage installed neither kube-proxy nor Cilium and did not enable Subnet load balancing; earlier results cannot be carried forward. Kube-OVN Underlay supports Services, but that does not establish correct discovery, forwarding, and return paths in every mixed-CNI cluster.

8.1 Monitoring Follow-up: Address Pool, Control Plane, and CNI / OVS¶

On October 19, the controller's 10660/metrics and CNI's 10665/metrics endpoints were already enabled, but neither was a target in the existing monitoring stack. We appended one VMServiceScrape to the existing VictoriaMetrics K8s Stack, discovering the two components through narrowly selected Headless Services at a 15-second interval. These services discover actual endpoints without ClusterIP forwarding; they do not constitute additional business-Service validation. Official monitoring guide

All 75 original targets were preserved, bringing the total to 77, with both new controller/CNI endpoints UP. vmagent did not restart, and Kube-OVN Deployment/DaemonSet specifications and Pod identities remained unchanged. Grafana reuses the existing VictoriaMetrics datasource, with separate address-pool, control-plane, and CNI/OVS views.

The following are real light-theme dashboard screenshots. Titles and legends use component roles, hiding internal nodes, addresses, and network names while retaining the original values and curves. They show a short follow-up observation on October 19, rather than migration-day history or a before/after performance comparison.

The business pool still contains 2 used and 4 available addresses, totaling 6 reserved addresses, matching the rejoin records. The internal node join network is excluded; these figures do not represent the entire physical /22 pool. CPU curves show cores used by the controller/CNI processes. Resident Set Size (RSS) measures resident process memory. Neither measures the entire host's resource usage.

The control-plane view retains observed 403 and 404 responses instead of replacing errors with zero. API-client traffic first reaches the experimental scoped gateway. Logs confirm denied CSR-cache reads, but status-code aggregates cannot attribute every 4xx response to the same resource. Identifying the 404 requests requires request paths or audit records.

No workqueue backlog was observed in the window, and retry rates were zero. A read-only follow-up request to the existing Underlay HTTP endpoint returned HTTP 200 and 612 bytes. This establishes that the checked business path remained reachable at that moment; it does not resolve control-plane permission issues. Other previously offline monitoring targets were not restored by this addition either, so new targets being UP must not be presented as full-stack health.

Three definitions matter in the third view:

This integration collects built-in controller/CNI metrics and reuses existing container metrics. The bounded deployment has no separate ovn-monitor or kube-ovn-pinger, so this follow-up includes neither database-internal health and logical-port topology nor dedicated Pod-probe RTT/loss charts. Healthy processes and empty queues do not establish those dimensions. Official metric definitions

The anonymized monitoring snapshot records the observation window, collection-change checks, and chart data.

9. Collect Evidence Layer by Layer¶

The following commands read state only. Pod names, namespace, and addresses are examples; use an authorized method to access the host.

kubectl get node worker-canary -o wide
kubectl -n cni-canary get pods -o wide
kubectl get provider-network biz -o yaml
kubectl get vlan business-vlan -o yaml
kubectl get subnet business-underlay -o yaml

# On the canary host
ip -j -d link show bond0
ip -j addr show bond0.210
ip route get 192.0.2.10
ovs-vsctl show
ovs-vsctl port-to-br bond0
ovs-vsctl list-ports br-int

# Inside an ordinary Pod
kubectl -n cni-canary exec underlay-client -- ip addr show eth0
kubectl -n cni-canary exec underlay-client -- ip route
kubectl -n cni-canary exec underlay-client -- curl -fsS http://10.244.8.33

A useful order is: the runtime's selected CNI configuration → controller address allocation and annotations → local CNI cache and socket readiness → veth attachment to OVS → localnet / ProviderNetwork → VLAN, ARP, and return routing → bidirectional old/new Pod requests → separate Service and DNS acceptance.

OVN flow tables cannot be diagnosed by looking only for NORMAL. Continue with logical switches, ports, counters, and traces. An OVS internal interface reporting DOWN also does not establish that the entire data plane is down.

10. What Remains Before a Production Replacement¶

This experiment completed one node's primary-CNI replacement while preserving the physical VLAN contract. It supports further evaluation of Kube-OVN Underlay, not an immediate whole-cluster replacement.

This run did not test sustained throughput, long-lived connections, cross-node Kube-OVN migration, or production scale, and it did not establish a speed advantage over Contiv. Compatibility with the existing physical network addresses one part of migration feasibility. Production adoption also requires coherent address management, Services, control-plane operations, and recovery.

For a new cluster, delegating a separate address range in advance, agreeing on business VLANs and physical gateways, and deploying a complete Kube-OVN stack is generally easier to operate than maintaining two primary CNIs indefinitely in one existing cluster. Where the old environment must remain, validate the single-node boundary, bidirectional traffic, and rollback before deciding how to expand.

References¶

• Kube-OVN 1.16 Underlay installation and physical-network requirements
• Kube-OVN 1.16.12 ProviderNetwork fields
• Kube-OVN 1.16.12 node VLAN and OVS wiring implementation
• Kube-OVN 1.16.12 Pod network selection and handling
• Anonymized acceptance data for this case