Servers and compute

The model runs here.

A language-model server that lives inside your building. It answers your apps over a standard API, packs many users onto a few GPUs, and never sends a prompt off site.

Talk to engineering See all solutions

The problem

Your prompts are data, and they are leaving.

People paste contracts, patient notes and source code into a public model because it helps them get the work done. Every one of those prompts, and every answer that comes back, crosses the internet to a vendor. You cannot prove where it was stored, whether it trained the next model, or who was allowed to read it. So security and legal say no, and the tool either gets banned or gets used in the shadows, which is worse. What you actually want is the chat, completion and embedding behaviour your team already likes, running behind your own firewall, on hardware you own, answering on an endpoint your apps already know how to call. You buy that server once. It does not meter your secrets by the token.

The numbers we build to

< 300 ms First token, 8B model

3,000 tok/s Throughput per GPU

64 Sessions at once, per node

0 bytes Prompt data leaving the building

The system

One rack, three layers.

Imagery in production

Serving layer

One endpoint feeds every app, and keeps the GPUs busy.

A few GPUs serve many people at once. Continuous batching adds and drops requests from a running batch on every decode step, so a short reply never waits behind a long one and the card is never idle mid-answer. The paged KV cache holds each session's attention state in fixed blocks, the way an operating system pages memory. Hundreds of conversations share one card without fragmenting it, and a repeated system prompt is cached once and reused across users. Apps reach it through an OpenAI-compatible /v1 endpoint. Anything already written against that API just points at your URL, and the first token streams back in under 300 ms warm.

Imagery in production

Model and quantisation

The right size for the work, not the biggest on the shelf.

You run a right-sized model, not the largest one made. A 7B to 14B model handles most business work: drafting, summarising, pulling fields out of documents, classification and code. It does that at a fraction of the memory and power a frontier model needs, which is what makes running it in-house affordable at all. Weights are quantised to INT8 or FP8. That roughly halves the memory and doubles throughput against FP16, with almost no quality lost on these tasks. Where density matters more than the last point of accuracy, a 4-bit AWQ build packs a 14B model onto a single 48 GB card. Models load from your own store and swap with no downtime. Pin a version, test the next one on a spare node, and cut over when you are ready.

Imagery in production

Hardware topology

Scales by the node, with a spare always standing by.

The unit of scale is a node, not a mystery region in someone else's cloud. Each node carries two to eight GPUs paired over NVLink, so a larger model can shard across the cards. It has redundant hot-swap power supplies and enterprise NVMe for the model store. You match the card to the job. The L40S 48 GB is the dense, cost-efficient choice for serving. The H100 80 GB is there when latency and the longest context windows matter. A tensor-parallel model spreads its layers across the paired cards over NVLink at full bandwidth. Need more concurrency? Add a node behind the load balancer and the numbers stack. A hot-spare node sits in N+1, so a failed card or power supply drains and reroutes without dropping the endpoint.

Turnkey

Live the same day.

The rack arrives cabled, the serving stack installed and a model of your choice loaded and benchmarked. No GPU drivers to wrangle, no Kubernetes to stand up first.

We build and benchmark it

Your rack is assembled, cabled and burned in, the serving stack is installed, and the model you chose is loaded and benchmarked before it ships.
Point your apps at one URL

Change one base URL in the apps that already speak the OpenAI API. Nothing else in your code has to move.
Start getting private tokens

The same day, your team is calling a model that answers from the rack in your own building. Every prompt stays on hardware you own.

Imagery in production

Private by design

It sits on your network.

The cluster is yours. It runs on your own VLAN, on hardware you bought once, with no route to the public internet required and no meter running on your prompts.

Talk to engineering

100 GbE

A low-latency 100 GbE fabric linking the nodes and the load balancer, isolated on your own VLAN with no route to the public internet required.

Request datasheet

What is included

Everything the endpoint needs.

The rack, the software and the people behind it come as one thing. Nothing here is an add-on you chase later.

The serving stack

The continuous-batching engine, paged KV cache and /v1 endpoint, tuned to the card it runs on.

Keys and audit

Per-team API keys, rate limits, and a logged request ID and token count on every call.

Dashboards

Prometheus metrics and Grafana boards for latency, throughput, queue depth and GPU load.

Support and spares

On-call engineering, signed updates, and a hot-spare node held in N+1.

Questions

For the team that will run it.

Does anything leave the building?

No. The cluster sits on your own VLAN with no route to the public internet required, so prompts, completions and the model all stay on hardware you own. That is the zero we design to: 0 bytes of prompt data leaving the building.

Will our existing apps work?

If they already speak the OpenAI API, yes. The serving stack exposes an OpenAI-compatible /v1 endpoint, so you change one base URL and keep the rest of your code.

Which model do we run?

A 7B to 14B model covers most business work: drafting, summarising, extraction, classification and code. You load it from your own store and swap versions with no downtime.

How fast does it answer?

The first token lands in under 300 ms on an 8B model when warm. From there a single GPU is built to push around 3,000 tokens a second, across up to 64 sessions at once per node.

What if a card or power supply fails?

A hot-spare node sits in N+1. When a card or a PSU goes, that node drains and reroutes without dropping the endpoint.

“A model that reads your contracts and your patient notes belongs where the rest of your secrets already live: on hardware you own, behind a door you control.”

The Calyraen engineering desk

Ready when you are

Bring the model in-house.

Tell us the apps you run and the models you have in mind. Our engineers will size a cluster and send back real numbers, not a brochure.

Talk to engineering See all solutions

Architecture

How a prompt becomes tokens.

A request crosses three stages inside the rack: an API surface your apps already speak, a scheduler that packs many users onto the cards, and a GPU runtime that runs the quantised model. Nothing about it is a black box you cannot see into.

Under the hood

The model runs in INT8 or FP8 kernels that roughly double throughput against FP16 at almost no quality cost on business tasks. Larger models shard tensor-parallel across NVLink-paired cards at full bandwidth. CUDA graphs capture the decode loop so per-token overhead stays flat as concurrency climbs, which is what holds first-token latency under 300 ms at load.

How the cluster is supplied

8B Default served model class

3 kW Draw per compute node

64 Concurrent sessions per node

< 300 ms First token, warm

Engineering questions

How it runs.

Which serving engine is it?

A continuous-batching, paged-attention engine of the vLLM class, hardened and packaged in the CalyOS serving stack. You get the throughput of modern batching without assembling and maintaining the stack yourself.

Can we run our own fine-tuned model?

Yes. Any Hugging Face-format model in the supported architectures loads from your model store, including your own fine-tunes and LoRA adapters. We help quantise and benchmark it on your node before it goes live.

What do our applications actually call?

A standard OpenAI-compatible base URL on your network. In most apps it is a one-line change from the public endpoint to yours, because the request and response shapes are identical, streaming included.

Specify it with us

Size it to your load.

Send us your peak concurrency and typical prompt and response length. We come back with a node count, a card and a first-token number you can hold us to.

Talk to engineering See all solutions

Performance

Latency you can feel.

Three numbers decide whether a private model is pleasant or painful: how fast the first token appears, how fast the rest stream, and how many people it serves at once before anyone waits. Here is where the cluster lands on each.

Measured on an 8B model

280 ms Time to first token, warm

45 tok/s Generation, single stream

3,000 tok/s Aggregate per GPU at batch

64 Concurrent streams before queue

The numbers

Three figures that matter.

What each one means for the person waiting on a reply.

Latency

Time to first token

How long from send to the first word appearing. On an 8B INT8 model over the LAN it is under 300 ms warm, so a chat feels instant rather than laggy.

Throughput

Tokens per second

A single stream generates around 45 tokens a second, faster than a person reads, while the node pushes roughly 3,000 tokens a second in total across a full batch.

Concurrency

Sessions per node

One node holds about 64 active streams before requests start to queue. Add a node behind the balancer and that headroom stacks close to linearly.

The difference

Owned versus rented.

Calyraen on-premPublic API

Where prompts go Stay inside your building Cross the internet to a vendor

Cost model Buy the node once Metered per token, forever

Latency floor LAN, under 300 ms Internet round trip plus queue

Used for training Never, by construction Per policy, and policies change

Runs offline Fully air-gapped if you wish Dead without a connection

Model version You pin it Deprecated on their schedule

“We sized it for forty concurrent users and it holds first token under a third of a second at peak. Legal stopped asking where the prompts go, because they stopped leaving.”

Head of platform engineering, a regional bank

Data handling

Nothing leaves the rack.

The cluster has no outbound dependency. Prompts, completions, embeddings and logs are written only to storage you control, there is no telemetry calling home and no vendor account behind it, and the whole thing runs with its internet cut. Every request is authenticated with a per-team API key and logged with a request ID and token count for audit, and prompt content can be kept out of those logs by policy. Models load from your own store, so you know exactly which weights are answering and nobody swaps them under you. For regulated work this is the line between a tool your security team bans and one they sign off, because the honest answer to where does our data go is nowhere.

Running it

Nodes run N+1 with a hot spare. Health checks drain a node showing GPU or memory faults and reroute new sessions to healthy ones, so a failed card degrades throughput rather than dropping the endpoint. Power and storage are redundant inside each node.

Rollout

Live in a fortnight.

Sizing

We measure your peak concurrency and typical prompt and response length, then pick node count, card and model class to a first-token target.
Install

The rack is mounted and cabled, the serving stack installed, and your chosen model loaded and benchmarked on site.
Integration

Your apps swap one base URL to the cluster, we issue per-team API keys and set rate limits, and traffic moves off the public endpoint.
Handover

Dashboards, a runbook and the on-call path are handed to your team, who own the cluster from day one.

Compliance questions

Where the data stays.

Does it phone home?

No. There is no telemetry, no licence check and no vendor account. The cluster serves entirely from your network and can be operated with its internet connection physically removed.

Can we run it fully air-gapped?

Yes. Models, engine and updates can be delivered on media and applied offline, so the cluster runs in an isolated enclave with no route to the internet at all.

How does this sit with GDPR and data residency?

Prompts and outputs never leave the premises, so the data stays in your jurisdiction under your existing controls. There is no processor to add to your records, because there is no third party in the request path.

Do your engineers see our prompts?

No. We support the hardware and the stack, not your traffic. Prompt and completion content stays in storage you own, and support access is to systems and metrics, never to request contents.

Specify it with us

Bring it in-house.

If prompts leaving the building is why a model is banned in your organisation, this closes it. Tell us the workload and we size the rack that keeps them home.

Talk to engineering OEM & ODM

The library

Source never leaves the network.

Model14B Rolecode completion

Request the files

Get the documents.

PDFs sent on request after a short qualifying call.

SLM inference cluster datasheet Datasheet PDF, on request Sizing and concurrency white paper White paper PDF, on request API and RAG integration guide Integration PDF, on request

“Our developers point their editor at one URL and get an assistant that has never seen the public internet. Same workflow, nothing leaving.”

Engineering director, a defence supplier

Specify it with us

Start with your numbers.

Send your peak users and prompt sizes and we return a sizing, a first-token figure and a fixed price. No meter behind it.

Talk to engineering See all solutions

The model runs here.

Your prompts are data, and they are leaving.

The numbers we build to

Serving layer

Model and quantisation

Hardware topology

We build and benchmark it

Point your apps at one URL

Start getting private tokens

It sits on your network.

GPU inference node

Serving accelerator

CalyOS serving stack

Model and embedding store

Top-of-rack switch

The serving stack

Keys and audit

Dashboards

Support and spares

Bring the model in-house.

How a prompt becomes tokens.

The endpoint your apps expect

Many users, a few cards

Quantised, parallel, fast

How the cluster is supplied

Size it to your load.

Latency you can feel.

Time to first token

Tokens per second

Sessions per node

Nothing leaves the rack.

You can see everything

On your schedule, reversible

One node can fail

Sizing

Install

Integration

Handover

Bring it in-house.

Solution brief

Sizing and concurrency white paper

API and RAG integration guide

Contract and document review

Clinical note drafting

Internal code assistant

Start with your numbers.