Servers & compute

AI training cluster

A multi-GPU cluster you own and run in your own hall. Dense GPU nodes on an NVLink-class mesh, joined by a 400 Gb/s RDMA fabric, fed by all-flash storage and one scheduler. Direct-to-chip cooling lets a rack pull past 60 kW without turning the room into a wind tunnel.

Talk to engineering See all solutions

The problem

Waiting

A training run moves at the speed of its slowest link, and on rented infrastructure that link is almost never the GPU. Instances land on whatever rack the provider has free, so eight machines that should share one switch end up three hops apart, and an all-reduce that should take milliseconds stalls on the network. The meter runs whether the GPUs are saturated or waiting on a straggler. Pause to inspect a checkpoint and it keeps running. Your data has to leave the building and come back, which dataset owners and compliance teams rarely like. Past a few nodes, utilisation quietly falls apart, because nothing was placed to keep the GPUs fed. A cluster you own, wired as one machine and scheduled as one machine, fixes the part the hourly price hides. It keeps every GPU busy.

Design targets

400 Gb/s RDMA per GPU, rail to the spine

> 90% Scaling efficiency to 64 GPUs

< 45 C Coolant to the cold plate

1.06 Design PUE at the rack

The system

One machine

Imagery in production

GPU nodes

Eight accelerators per node, wired all-to-all inside the box.

Each node carries eight accelerators on an NVLink-class switch mesh, so every GPU reaches every other GPU inside the chassis at full bandwidth without ever touching the network. The host side is dual server CPUs on PCIe Gen5 with GPUDirect, so data moves from NVMe and the fabric straight into GPU memory. It never bounces through system RAM. One node trains a mid-size model on its own. The fabric is what turns thirty-two of them into a single machine.

Imagery in production

The fabric

Where scaling is won or lost.

Between nodes, every GPU gets its own 400 Gb/s RDMA rail into a non-blocking fat-tree, so a gradient all-reduce across the whole cluster runs GPU to GPU with no CPU in the path. Rail-optimised means GPU 0 on every node shares a leaf, GPU 1 shares the next, and so on. Collective traffic stays off the spine, and bandwidth holds flat as you add nodes. This is the layer rented infrastructure gets wrong. Placed right, scaling efficiency stays above ninety percent to sixty-four GPUs, instead of falling off at eight.

Imagery in production

Storage and scheduler

Keeping the GPUs fed, and shared fairly.

Datasets and checkpoints live on an all-flash parallel filesystem that reads at hundreds of gigabytes per second. That is enough that loading never starves a full cluster of GPUs in the middle of an epoch. A Slurm-class scheduler gang-schedules a job so all of its GPUs start together, keeps teams to a fair share, and checkpoints to fast storage. A long run survives a node fault and resumes, rather than starting over. One cluster, many teams, and no single team able to strand the GPUs.

Closer look

Cooled at the chip.

Heat comes off the accelerators through cold plates, not fans. A liquid-to-liquid CDU feeds coolant to the plates at under 45 C, with redundant pumps and leak detection wired to the rack controller. That is how a single rack pulls past 60 kW and still runs quiet, at a design PUE of 1.06.

Imagery in production

Direct-to-chip cooling, N+1 CDU per rack.

What is in the rack

The build

Every line item is chosen and integrated so the cluster behaves as one machine, not a shelf of servers.

Imagery in production

Compute

GPU training node

8 GPU, liquid

Eight accelerators on an NVLink-class mesh, dual Gen5 host CPUs, HBM3 memory and direct-to-chip cold plates. This is the unit you add to grow the cluster.

Request datasheet

Imagery in production

Fabric

400 Gb/s RDMA switch

Non-blocking

Rail-optimised leaf and spine that gives every GPU its own 400 Gb/s RDMA path. GPUDirect and NCCL are tuned at commissioning.

Request datasheet

Imagery in production

Storage

All-flash parallel store

Hundreds of GB/s

An NVMe parallel filesystem for datasets and checkpoints, sized so loading never starves the GPUs during an epoch.

Request datasheet

Imagery in production

Facility

Coolant distribution unit

Per rack, N+1

A liquid-to-liquid CDU that feeds the cold plates at under 45 C, with redundant pumps and leak detection reporting to the rack controller.

Request datasheet

Imagery in production

Software

CalyOS Server + scheduler

On cluster

Provisioning, the Slurm-class scheduler, fair-share and gang scheduling, telemetry and checkpoint management. All of it on hardware you own.

Request datasheet

Why own it

Yours

On-prem

Owned

The cluster sits in your hall, on your network, under your scheduler. Datasets never leave the building. The GPUs are busy for you, not metered by the hour, and when a run pauses, so does the only cost that matters. You buy the machine once and run it flat out.

Imagery in production

GPUs busy, not metered

You size the cluster to your own workload and run it around the clock. There is no hourly meter to race and no idle-time bill.

Data stays in the building

Training data and checkpoints sit on your storage, inside your network. That keeps dataset owners and compliance teams on side.

Placed to scale

Nodes, rails and storage are wired as one machine, so scaling efficiency holds instead of collapsing past a few nodes.

One capital cost

You buy the configured cluster once. No per-GPU-hour licence, no forced subscription behind it.

Good to know

Straight answers.

Does our training data ever leave the building?

No. Datasets and checkpoints live on your own storage, inside your own network. Nothing is copied out to a provider and back, which is what keeps dataset owners and compliance teams comfortable.

Why does scaling hold past a few nodes?

Every GPU gets its own 400 Gb/s RDMA rail into a non-blocking fat-tree, and the rails are placed so collective traffic stays off the spine. That holds scaling efficiency above ninety percent to sixty-four GPUs, where rented infrastructure falls off at eight.

How do you cool a rack that pulls past 60 kW?

Direct-to-chip liquid cooling. A liquid-to-liquid CDU feeds the cold plates at under 45 C, with redundant pumps and leak detection reporting to the rack controller. Design PUE at the rack is 1.06, so almost all the power reaches the accelerators.

Who runs the scheduler?

You do, from CalyOS Server. A Slurm-class scheduler gang-schedules each job so all its GPUs start together, keeps teams to a fair share, and checkpoints to fast flash so a node fault resumes a run instead of restarting it.

“A GPU you rent is billed whether it trains or waits. A GPU you own is bought once, then earns its keep every hour a run is live. That is the whole case for owning the cluster.”

The Calyraen compute team

Ready when you are

Build it in your own hall.

Tell us the model size, the node count you have in mind and the power you can give a rack. We will size the cluster and send back a build.

Talk to engineering See all solutions

Under the hood

Three layers

Imagery in production

Compute

Eight GPUs that behave as one

Inside a node, eight accelerators share an NVLink-class switch mesh, so any GPU reaches any other at full bandwidth without a network hop. HBM3 keeps model shards and activations close to the cores, and a dual Gen5 host with GPUDirect lets data land in GPU memory straight from storage and the fabric. A single node is enough to train a mid-size model; the point of the mesh is that model-parallel shards inside the box pay almost no communication tax.

Imagery in production

Fabric

A rail for every GPU

Between nodes, each GPU owns a 400 Gb/s RDMA rail into a non-blocking rail-optimised fat-tree. Collective operations like all-reduce run GPU to GPU over GPUDirect RDMA with no CPU in the path, and rail optimisation keeps that traffic off the spine so bandwidth stays flat as the cluster grows. NCCL is tuned to the topology at commissioning, which is the difference between ninety percent scaling and thirty.

Imagery in production

Storage and schedule

Fed, shared and restartable

An all-flash parallel filesystem serves datasets and checkpoints at hundreds of gigabytes per second so data loading never starves the GPUs. A Slurm-class scheduler gang-schedules each job, enforces fair-share between teams, and writes periodic checkpoints to fast storage, so a fault costs minutes of rollback rather than a whole run.

Sized to fit

Scale

The same building block from a single rack to a hall.

Starter

One rack, up to 32 GPUs

Four training nodes, one leaf switch, a flash store and a CDU. Trains real models today and seeds the fabric to grow.

Pod

A row, 128 to 256 GPUs

Leaf and spine across several racks, one parallel filesystem, one scheduler domain. The point at which rail optimisation earns its keep.

Hall

Multiple pods, one schedule

Pods federated under one fair-share scheduler and one namespace, so teams share capacity without stranding GPUs.

The topology at a glance

8 GPUs per node

400 Gb/s RDMA rail per GPU

Non-blocking Rail-optimised fat-tree

1 Scheduler namespace

Engineering questions

Fabric

InfiniBand or Ethernet?

Either. The default is a 400 Gb/s InfiniBand-class rail-optimised fabric; where you standardise on Ethernet we build the same non-blocking topology on 400 GbE with RoCEv2 and lossless configuration. Both give GPUDirect RDMA and are tuned to NCCL at commissioning.

Can we mix in nodes we already own?

Usually yes. If your existing GPU nodes expose the right RDMA NICs and PCIe generation we can bring them into the fabric and the scheduler as a separate partition, though a scaling number is only guaranteed on the nodes we integrate.

How does model-parallel traffic stay off the network?

Tensor and pipeline parallel groups are placed inside a node so their heavy traffic rides the NVLink-class mesh, while data-parallel all-reduce crosses the RDMA fabric. The scheduler and NCCL topology file enforce that placement rather than leaving it to chance.

Performance

Utilisation

The number that decides the bill is not peak FLOPS on a spec sheet, it is how much of it you actually use during a run. Everything here is built to hold model FLOPs utilisation high as you scale.

Measured on the floor

> 90% Scaling efficiency, 8 to 64 GPUs

40 to 60% Model FLOPs utilisation, typical

< 5% Time lost to comms at 32 GPUs

Minutes Rollback after a node fault

The difference

Coupled

Owned, coupled clusterRented instances

GPU placement Nodes on one fabric, rails planned Wherever the provider has capacity

All-reduce path GPU to GPU over RDMA, no CPU Through host and shared network

Scaling efficiency Over 90% to 64 GPUs Falls off past a few nodes

Cost while paused Nothing, you own it Metered by the hour, idle or not

Where the data lives Your storage, your network Copied out and back

“The all-reduce time barely moves between eight GPUs and sixty-four. That is the whole game for us, and it is why a run that took nine days now takes under four.”

Principal research engineer, an autonomous systems lab

In the hall

Thermals

Imagery in production

Cooling

Heat leaves at the chip

Each accelerator sits under a direct-to-chip cold plate fed with coolant at under 45 C from a per-rack coolant distribution unit. Taking heat away as liquid at the source is why a rack can dissipate past 60 kW without the deafening airflow an air-cooled hall needs, and why the design PUE sits near 1.06 rather than 1.5. Leak detection reports to the rack controller and the CDU carries N+1 pumps.

Imagery in production

Power

Sized for a full rack

A training rack draws tens of kilowatts continuously, so power is delivered over busbar with redundant feeds and per-node metering. We specify the PDU, breaker and feed for your actual node count and give facilities the real steady-state and peak draw, not a nameplate figure, so nothing trips when the whole cluster steps into a run at once.

Imagery in production

Telemetry

Every watt and degree logged

GPU temperature, coolant flow, power draw and fabric health stream into one dashboard, with alerts before a threshold rather than after. The same telemetry drives scheduler decisions, so a node running hot or a failing link is drained of jobs automatically instead of taking a run down with it.

What we run

Kept

Managed

Uptime

A cluster is only useful when it is up and the GPUs are busy. We commission it to a measured scaling number, monitor thermals and fabric health, and hold spares so a failed node is drained and swapped without standing the whole run down. You get the utilisation, not the pager.

Imagery in production

Cooled at the source

Direct-to-chip liquid at under 45 C, per-rack CDU with redundant pumps and leak detection.

Power specified for real load

Busbar delivery, redundant feeds and the true steady-state and peak draw handed to facilities.

Spares on the shelf

A failed node is drained by the scheduler and swapped from local spares, its jobs resumed from the last checkpoint.

Commissioned to a number

We hand over a measured scaling-efficiency figure on your models, not a spec-sheet claim.

Commissioning

Standing it up

Site and facility survey

Floor loading, power feed, coolant supply and return, and network hand-off checked against the build before anything ships.
Install and plumb

Racks landed, nodes seated, cold plates and CDU plumbed, busbar and fabric cabled to the rail plan and leak-tested.
Tune and prove

NCCL tuned to the topology, then a scaling test on your own model to a measured efficiency figure, not a synthetic benchmark.
Handover

Scheduler, fair-share policy and telemetry configured to your teams, with a run-book and the commissioning results for your file.

What the hall needs to provide

40 to 120 kW Per training rack

< 45 C Coolant supply to plates

1.06 Design PUE at rack

N+1 Pumps and power feeds

Specify it with us

Sizing

Send us the models you train and the hall you have. We will size the nodes, fabric and cooling to a scaling number and a power budget you can take to facilities.

Talk to engineering OEM & ODM

Resources

Papers

The briefs, the sizing method and worked deployments. Enough to scope the cluster before a single node ships. Open a card to read what is inside.

The library

Reading

Briefs for the people who sign, white papers for the people who integrate, and worked cases for the people who run it.

Brief

Imagery in production

Solution brief

The one-page case for owning it.

2 pages For decision makers

White paper

Imagery in production

Cluster sizing white paper

How we size nodes, fabric and storage.

18 pages For technical evaluators

White paper

Imagery in production

Fabric and NCCL tuning guide

For your own engineers.

22 pages For integrators

Use case

Imagery in production

Genomics model training

One rack, 32 GPUs.

ConfigStarter Rundata-parallel

Use case

Imagery in production

Autonomy perception stack

A pod, 256 GPUs.

ConfigPod Runtensor + data parallel

Use case

Imagery in production

Shared research cluster

Many teams, one schedule.

ConfigHall Runfair-share

Request the files

Files

Sent as PDF on request, tuned to the models and hall you send us.

AI training cluster datasheet Datasheet PDF, on request Cluster sizing white paper White paper PDF, on request Fabric and NCCL tuning guide Integration PDF, on request

Specify it with us

Build

Tell us the workload and we will come back with a configured cluster, a scaling number and a power and cooling budget. You own what we build.

Talk to engineering See all solutions

AI training cluster

Waiting

Design targets

GPU nodes

The fabric

Storage and scheduler

GPU training node

400 Gb/s RDMA switch

All-flash parallel store

Coolant distribution unit

CalyOS Server + scheduler

Owned

GPUs busy, not metered

Data stays in the building

Placed to scale

One capital cost

Build it in your own hall.

Eight GPUs that behave as one

A rail for every GPU

Fed, shared and restartable

One rack, up to 32 GPUs

A row, 128 to 256 GPUs

Multiple pods, one schedule

The topology at a glance

Utilisation

Heat leaves at the chip

Sized for a full rack

Every watt and degree logged

Uptime

Cooled at the source

Power specified for real load

Spares on the shelf

Commissioned to a number

Site and facility survey

Install and plumb

Tune and prove

Handover

What the hall needs to provide

Sizing

Papers

Solution brief

Cluster sizing white paper

Fabric and NCCL tuning guide

Genomics model training

Autonomy perception stack

Shared research cluster

Build