Pre-Provision Kubernetes Node Capacity

Reserve Kubernetes node capacity by running low-priority pause pods with K8s.CapacityPreProvisioner. When real application pods arrive, they preempt the pause pods and start immediately without waiting for node scale-up or container image pulls. Optionally, cache container images on pre-provisioned nodes via a DaemonSet.

Use capacity pre-provisioning when your cluster needs fast startup times for bursty workloads or pre-cached container images to reduce image pull latency.

Prerequisites

Role: You must have the C3.ClusterAdmin role. All K8s.CapacityPreProvisioner APIs are scoped to the cluster-admin action group.
Kubernetes cluster: The cluster must be running in Kubernetes mode (not local or Jenkins).
Hardware profile: You must know the CPU, memory, and optionally computeKind for the target node pool.

How it works

K8s.CapacityPreProvisioner manages a Kubernetes Deployment of low-priority pause pods:

Pause pods run with a low priority class (e.g. -9). They reserve node capacity but consume minimal resources.
Real workloads (priority 0) preempt the pause pods when they arrive, starting immediately on already-scaled nodes.
Image caching (optional) uses a DaemonSet to pull and cache container images on nodes, so workload pods avoid image pull latency.
Topology spread distributes pause pods evenly across availability zones by default, or pins to specific zones when configured.

Configuration is stored at the cluster level. The Kubernetes Deployment name is always derived as c3pp-<name>.

Create a capacity pre-provisioner

Use ensure to create a new pre-provisioner. The name must match ^[a-z0-9]+$ (lowercase alphanumeric only, no hyphens):

JavaScript

K8s.CapacityPreProvisioner.ensure(
    K8s.CapacityPreProvisioner.Spec.make()
        .withName('mypool')
        .withHardwareProfile(HardwareProfile.builder().cpu(4).memoryMb(8192).computeKind('gpu').build())
        .withImages([ContainerImage.fromString('my.registry.io/ml-model:latest')])
        .withZones(['us-west-2a', 'us-west-2b'])
        .withNodeCount(5)
);

After ensure, call reconcileInfra to create the Kubernetes Deployment:

JavaScript

K8s.CapacityPreProvisioner.forName('mypool').reconcileInfra();

Update configuration

Use ensure with a partial spec to update only specific fields. Fields not included in the spec are preserved:

JavaScript

// Update node count only — hardware profile, images, zones are preserved
K8s.CapacityPreProvisioner.ensure(
    K8s.CapacityPreProvisioner.Spec.make()
        .withName('mypool')
        .withNodeCount(10)
);
K8s.CapacityPreProvisioner.forName('mypool').reconcileInfra();

You can also modify config directly and reconcile:

JavaScript

var p = K8s.CapacityPreProvisioner.forName('mypool');
p.config(false).withNodeCount(3).setConfig(ConfigOverride.CLUSTER);
p.reconcileInfra();

List all pre-provisioners

JavaScript

K8s.CapacityPreProvisioner.list();

Check status

Returns desiredPods (from config) and live podStates from Kubernetes, including DaemonSet image cache status:

JavaScript

K8s.CapacityPreProvisioner.forName('mypool').status();

The status includes:

desiredPods: Configured pause pod count
podStates: Live pod states from Kubernetes
daemonSet: Image cache status with totalPods, readyPods (images cached), pendingPods (starting up), and degradedPods (>10 min without reaching Running state) with error details

Terminate a pre-provisioner

Terminates the Kubernetes Deployment and clears the config:

JavaScript

K8s.CapacityPreProvisioner.forName('mypool').terminate();

Image caching

When imageCacheOnDaemonSet is enabled (default), a DaemonSet automatically pulls and caches container images on nodes. Workloads can use pod affinity to target nodes where images are cached:

YAML

affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: c3__warm_images_ready-0
          operator: In
          values: ["0true0"]
      topologyKey: kubernetes.io/hostname

The DaemonSet uses the label c3__warm_images_ready-0=0true0 to signal when images have been successfully cached on a node.

Pod affinity helpers

Rather than hand-writing the affinity YAML above, call podHardAffinitySpec or podSoftAffinitySpec to generate the pod affinity spec that targets this pre-provisioner's warm nodes. Both are callable by the C3.Developer action group:

JavaScript

// Hard requirement — only schedule on nodes with this pool's images cached
K8s.CapacityPreProvisioner.forName('mypool').podHardAffinitySpec();

// Soft preference — prefer warm nodes but still schedule if none available
K8s.CapacityPreProvisioner.forName('mypool').podSoftAffinitySpec();

The returned JSON selects image pre-puller pods for the named pool that report c3__warm_images_ready-0=0true0, co-scheduling by kubernetes.io/hostname. Splice the result into your workload's pod spec affinity.

Circuit breaker

Image caching can fail — for example a DaemonSet start failure or a catastrophic image pull failure. When that happens, DaemonSet pods never flip c3__warm_images_ready-0 to 0true0, and hard-affinity workloads that select on it can never schedule.

Enable the circuit breaker to stamp c3__warm_images_ready-0=0true0 by default (instead of 0false0) on DaemonSet pods, so those workloads still land on pre-provisioned nodes:

JavaScript

var p = K8s.CapacityPreProvisioner.forName('mypool');
p.setCircuitBreaker(true);
p.reconcileInfra();

Disable it (restore normal cache-gated behavior) with p.setCircuitBreaker(false) followed by reconcileInfra.

Configuration reference

Field	Type	Default	Description
`name`	`string` (required)	—	Unique name matching `^[a-z0-9]+$`. K8s Deployment is `c3pp-<name>`.
`nodeCount`	`int`	`0`	Desired pause pod replicas.
`hardwareProfile`	`HardwareProfile` (required)	—	CPU, memory, and optional `computeKind` for node targeting.
`images`	`[ContainerImage]`	`[]`	Container images to pre-cache via DaemonSet.
`zones`	`[string]`	`[]`	AZ targeting. Empty spreads across zones automatically.
`imageCacheOnPausePods`	`boolean`	`false`	When `true`, pull images on pause pods (Deployment).
`disableDaemonSetImageCache`	`boolean`	`false`	When `true`, disable DaemonSet image caching.
`disableCapacityPreprovisionerDeployment`	`boolean`	`false`	When `true`, disable pause pod Deployment (DaemonSet only).
`pauseImageUrl`	`string`	(derived from default registry)	Fully qualified pause container image URL.
`imagePullerImageUrl`	`string`	`c3cidev.azurecr.io/c3server/image-puller:1.0.3`	Container image URL for the DaemonSet labeler (signals when images are cached).
`circuitBreakerEnabled`	`boolean`	`false`	When `true`, DaemonSet pods are stamped `c3__warm_images_ready-0=0true0` by default so hard-affinity workloads still schedule when image caching fails. Set via `setCircuitBreaker`.

Behavior matrix

The following table shows how different flag combinations affect which Kubernetes resources are created:

`disableCapacityPreprovisionerDeployment`	`imageCacheOnPausePods`	`disableDaemonSetImageCache`	Result
`false`	`false`	`false`	Default: Deployment (no image sidecars) + DaemonSet
`false`	`true`	`false`	Deployment (with image sidecars) + DaemonSet
`false`	`true`	`true`	Deployment (with image sidecars), no DaemonSet
`false`	`false`	`true`	Deployment (no image sidecars), no DaemonSet
`true`	`*`	`false`	No Deployment, DaemonSet only
`true`	`*`	`true`	No Deployment, no DaemonSet

Legend: * = flag value is ignored when disableCapacityPreprovisionerDeployment=true

Status reporting

The status() method returns detailed information about the pre-provisioner and its Kubernetes resources:

JSON

{
  "desiredPods": <number>,
  "podStates": {
    "running": <number>,
    "pending": <number>,
    "failed": <number>
  },
  "daemonSet": {
    "totalPods": <number>,
    "readyPods": <number>,
    "pendingPods": <number>,
    "degradedPods": <number>,
    "errors": [<error messages>]
  }
}

Key metrics:

desiredPods: Configured pause pod count from the Deployment
podStates: Live pod states (running, pending, failed) from Kubernetes
daemonSet.totalPods: Total DaemonSet pods across all nodes
daemonSet.readyPods: Pods with images successfully cached (labeled c3__warm_images_ready-0=0true0)
daemonSet.pendingPods: Pods not yet in Running state but within the 10-minute threshold (normal startup — pulling images, creating containers)
daemonSet.degradedPods: Pods that have not reached Running state within 10 minutes (stuck in ContainerCreating, ImagePulling, Pending, CrashLoopBackOff, ImagePullBackOff, etc.)
daemonSet.errors: Detailed error messages from degraded pod container statuses

Copy link to this sectionPrerequisites

Copy link to this sectionHow it works

Copy link to this sectionCreate a capacity pre-provisioner

Copy link to this sectionUpdate configuration

Copy link to this sectionList all pre-provisioners

Copy link to this sectionCheck status

Copy link to this sectionTerminate a pre-provisioner

Copy link to this sectionImage caching

Copy link to this sectionPod affinity helpers

Copy link to this sectionCircuit breaker

Copy link to this sectionConfiguration reference

Copy link to this sectionBehavior matrix

Copy link to this sectionStatus reporting

Copy link to this sectionSee also

Prerequisites

How it works

Create a capacity pre-provisioner

Update configuration

List all pre-provisioners

Check status

Terminate a pre-provisioner

Image caching

Pod affinity helpers

Circuit breaker

Configuration reference

Behavior matrix

Status reporting

See also