Your cart 0 items
Your cart is empty.
Searching the site...
No matches. Try another word.
Log in
Servers & compute

SLM inference cluster

A private language-model server that runs inside your own building. It serves tokens to your apps over a standard API, batches many users onto a few GPUs, and never sends a prompt off site.

Talk to engineering See all solutions
The problem

Prompts are your data, and they are leaving.

Staff paste contracts, patient notes and source code into a public model because it helps, and every one of those prompts and completions crosses the internet to a vendor. You cannot prove where it was stored, whether it was used to train the next model, or who was able to read it. So security and legal say no, and the tool either gets banned or gets used in the shadows where it is worse. What an organisation actually needs is the same chat, completion and embedding behaviour it already likes, running behind its own firewall, on hardware it owns, answering on an endpoint its apps already know how to call. That is a server you buy once, not a subscription that meters your secrets.

Design targets
< 300 ms Time to first token, 8B model
3,000 tok/s Aggregate throughput per GPU
64 Concurrent sessions per node
0 bytes Prompt data leaving the building
The system

One rack, three layers.

Serving, model and hardware. Each layer earns its place by fixing one thing a public API cannot give you. Open a layer for what it does and how far it goes.

Keeps GPUs busy
Imagery in production

Serving layer

Feeds every app on one endpoint.

Continuous batching, paged KV cache OpenAI-compatible /v1 endpoint First token under 300 ms
Keeps GPUs busy
Imagery in production

Serving layer

Feeds every app on one endpoint.

Continuous batching, paged KV cache OpenAI-compatible /v1 endpoint First token under 300 ms

The serving layer keeps a few GPUs busy with many users at once. Continuous batching adds and removes requests from a running batch every decode step, so a short reply never waits behind a long one and the card is never idle mid-generation. A paged KV cache stores each session's attention state in fixed blocks, the way an operating system pages memory, so hundreds of conversations share one card without fragmenting it. Repeated system prompts are cached once with prefix caching and reused across users. Apps talk to it through an OpenAI-compatible /v1 endpoint, so anything already written against that API points at your URL and works, streaming the first token in under 300 ms warm.
Half the memory
Imagery in production

Model and quantisation

Right size, not the biggest.

7B to 14B class, INT8 or FP8 weights 4-bit AWQ option for density Swap versions with no downtime
Half the memory
Imagery in production

Model and quantisation

Right size, not the biggest.

7B to 14B class, INT8 or FP8 weights 4-bit AWQ option for density Swap versions with no downtime

You serve a right-sized model, not the largest one. A 7B to 14B parameter model handles most business work, drafting, summarising, extraction, classification and code, at a fraction of the memory and power of a frontier model, which is what makes on-prem affordable in the first place. Weights are quantised to INT8 or FP8, roughly halving memory and doubling throughput against FP16 with almost no quality loss on these tasks. A 4-bit AWQ option packs a 14B model into a single 48 GB card when density matters more than the last point of accuracy. Models load from your own store and swap without downtime, so you pin a version, test a new one on a spare node, and cut over when you are ready.
N+1 spare
Imagery in production

Hardware topology

Scales by the node.

2 to 8 GPUs per node, NVLink paired L40S 48 GB or H100 80 GB Redundant PSU, hot-spare node
N+1 spare
Imagery in production

Hardware topology

Scales by the node.

2 to 8 GPUs per node, NVLink paired L40S 48 GB or H100 80 GB Redundant PSU, hot-spare node

The unit of scale is a node, not a mystery cloud region. A node carries two to eight GPUs paired over NVLink so a larger model can shard across cards, with redundant hot-swap power supplies and enterprise NVMe for the model store. You pick the card to the workload: L40S 48 GB for dense, cost-efficient serving, or H100 80 GB where latency and the longest context windows matter. A tensor-parallel model spreads its layers across the paired cards over NVLink at full bandwidth. Need more concurrency, add a node behind the load balancer and the numbers stack. A hot-spare node sits in N+1 so a card or PSU failure drains and reroutes without dropping the endpoint.
Solution optimized products

Configured for private serving.

Every line item selected and integrated to serve tokens on your own network.

Imagery in production
Compute

GPU inference node

2 to 8 GPU

Dual-socket server, two to eight NVLink-paired GPUs, redundant hot-swap PSUs and enterprise NVMe for the model store, rack-mounted and pre-integrated.

Request datasheet
Imagery in production
Compute

Serving accelerator

48 to 80 GB

L40S 48 GB for dense, cost-efficient serving or H100 80 GB for lowest latency and long context, sized to your concurrency and model class.

Request datasheet
Imagery in production
Software

CalyOS serving stack

OpenAI API

Continuous-batching engine, paged KV cache, OpenAI-compatible /v1 endpoint, per-team API keys, rate limits and audit logging, all on-prem.

Request datasheet
Imagery in production
Serving

Model and embedding store

On node

Local weight repository and vector cache, version-pinned models loaded on demand, hot-swap between versions without dropping the endpoint.

Request datasheet
Imagery in production
Networking

Top-of-rack switch

100 GbE

Low-latency 100 GbE fabric linking nodes and the load balancer, isolated on your own VLAN with no route to the public internet required.

Request datasheet
What is included

Everything the endpoint needs.

Turnkey

Integrated.

The rack arrives cabled, the serving stack installed and a model of your choice loaded and benchmarked. Your team changes one base URL in the apps that already speak the OpenAI API and starts getting private tokens the same day, with no GPU driver wrangling or Kubernetes to stand up first.

Imagery in production

The serving stack

Continuous-batching engine, paged KV cache and the /v1 endpoint, tuned to the card it runs on.

Keys and audit

Per-team API keys, rate limits, and a logged request ID and token count for every call.

Dashboards

Prometheus metrics and Grafana boards for latency, throughput, queue depth and GPU load.

Support and spares

On-call engineering, signed updates and a hot-spare node held in N+1.

“We stopped worrying about which staff had pasted what into which chatbot, because the model that answers them now sits in our own comms room.”
Chief information security officer, a healthcare provider
Architecture

How a prompt becomes tokens.

A request crosses three stages inside the rack: an API surface your apps already speak, a scheduler that packs many users onto the cards, and a GPU runtime that runs the quantised model. Nothing about it is a black box you cannot see into.

Under the hood

Three stages, one request.

Imagery in production
API surface

The endpoint your apps expect

The front door is an OpenAI-compatible API, with /v1/chat/completions, /v1/completions and /v1/embeddings, and server-sent-event streaming so tokens arrive as they generate. Each team gets its own API key and rate limit, so one runaway script cannot starve the rest. Anything already written against that API, an editor plugin, a chatbot, a nightly batch job, points at your URL and runs unchanged.

Imagery in production
Scheduler

Many users, a few cards

A continuous-batching scheduler adds and removes requests from the running batch every decode step, so the GPU never idles waiting on a slow reply and a one-line answer does not queue behind a long one. The paged KV cache stores each conversation's attention state in fixed blocks, so hundreds of sessions share a card without fragmenting its memory. Shared system prompts are cached once with prefix caching and reused across users.

Imagery in production
GPU runtime

Quantised, parallel, fast

The model runs in INT8 or FP8 kernels that roughly double throughput against FP16 at almost no quality cost on business tasks. Larger models shard tensor-parallel across NVLink-paired cards at full bandwidth. CUDA graphs capture the decode loop so per-token overhead stays flat as concurrency climbs, which is what holds first-token latency under 300 ms at load.

How the cluster is supplied

8B Default served model class
3 kW Draw per compute node
64 Concurrent sessions per node
< 300 ms First token, warm
Engineering questions

How it runs.

Which serving engine is it?

A continuous-batching, paged-attention engine of the vLLM class, hardened and packaged in the CalyOS serving stack. You get the throughput of modern batching without assembling and maintaining the stack yourself.

Can we run our own fine-tuned model?

Yes. Any Hugging Face-format model in the supported architectures loads from your model store, including your own fine-tunes and LoRA adapters. We help quantise and benchmark it on your node before it goes live.

What do our applications actually call?

A standard OpenAI-compatible base URL on your network. In most apps it is a one-line change from the public endpoint to yours, because the request and response shapes are identical, streaming included.

Specify it with us

Size it to your load.

Send us your peak concurrency and typical prompt and response length. We come back with a node count, a card and a first-token number you can hold us to.

Talk to engineering See all solutions
Performance

Latency you can feel.

Three numbers decide whether a private model is pleasant or painful: how fast the first token appears, how fast the rest stream, and how many people it serves at once before anyone waits. Here is where the cluster lands on each.

Measured on an 8B model
280 ms Time to first token, warm
45 tok/s Generation, single stream
3,000 tok/s Aggregate per GPU at batch
64 Concurrent streams before queue
The numbers

Three figures that matter.

What each one means for the person waiting on a reply.

Latency

Time to first token

How long from send to the first word appearing. On an 8B INT8 model over the LAN it is under 300 ms warm, so a chat feels instant rather than laggy.

Throughput

Tokens per second

A single stream generates around 45 tokens a second, faster than a person reads, while the node pushes roughly 3,000 tokens a second in total across a full batch.

Concurrency

Sessions per node

One node holds about 64 active streams before requests start to queue. Add a node behind the balancer and that headroom stacks close to linearly.

The difference

Owned versus rented.

Calyraen on-premPublic API
Where prompts go Stay inside your building Cross the internet to a vendor
Cost model Buy the node once Metered per token, forever
Latency floor LAN, under 300 ms Internet round trip plus queue
Used for training Never, by construction Per policy, and policies change
Runs offline Fully air-gapped if you wish Dead without a connection
Model version You pin it Deprecated on their schedule
“We sized it for forty concurrent users and it holds first token under a third of a second at peak. Legal stopped asking where the prompts go, because they stopped leaving.”
Head of platform engineering, a regional bank
Data handling

Nothing leaves the rack.

The cluster has no outbound dependency. Prompts, completions, embeddings and logs are written only to storage you control, there is no telemetry calling home and no vendor account behind it, and the whole thing runs with its internet cut. Every request is authenticated with a per-team API key and logged with a request ID and token count for audit, and prompt content can be kept out of those logs by policy. Models load from your own store, so you know exactly which weights are answering and nobody swaps them under you. For regulated work this is the line between a tool your security team bans and one they sign off, because the honest answer to where does our data go is nowhere.

Running it

Boring to run, on purpose.

Imagery in production
Observability

You can see everything

The stack exports Prometheus metrics for per-model latency, tokens per second, queue depth and GPU utilisation, with ready-made Grafana dashboards. You watch load build and add capacity before users feel it, instead of guessing from a vendor status page you do not control.

Imagery in production
Updates

On your schedule, reversible

Engine and model images are signed and staged. You test a new model or engine on the spare node, promote it when the numbers look right, and roll back to the previous image in minutes if they do not. Nothing updates itself under a live workload.

Imagery in production
Failover

One node can fail

Nodes run N+1 with a hot spare. Health checks drain a node showing GPU or memory faults and reroute new sessions to healthy ones, so a failed card degrades throughput rather than dropping the endpoint. Power and storage are redundant inside each node.

Rollout

Live in a fortnight.

  1. Sizing

    We measure your peak concurrency and typical prompt and response length, then pick node count, card and model class to a first-token target.

  2. Install

    The rack is mounted and cabled, the serving stack installed, and your chosen model loaded and benchmarked on site.

  3. Integration

    Your apps swap one base URL to the cluster, we issue per-team API keys and set rate limits, and traffic moves off the public endpoint.

  4. Handover

    Dashboards, a runbook and the on-call path are handed to your team, who own the cluster from day one.

Compliance questions

Where the data stays.

Does it phone home?

No. There is no telemetry, no licence check and no vendor account. The cluster serves entirely from your network and can be operated with its internet connection physically removed.

Can we run it fully air-gapped?

Yes. Models, engine and updates can be delivered on media and applied offline, so the cluster runs in an isolated enclave with no route to the internet at all.

How does this sit with GDPR and data residency?

Prompts and outputs never leave the premises, so the data stays in your jurisdiction under your existing controls. There is no processor to add to your records, because there is no third party in the request path.

Do your engineers see our prompts?

No. We support the hardware and the stack, not your traffic. Prompt and completion content stays in storage you own, and support access is to systems and metrics, never to request contents.

Specify it with us

Bring it in-house.

If prompts leaving the building is why a model is banned in your organisation, this closes it. Tell us the workload and we size the rack that keeps them home.

Talk to engineering OEM & ODM
The library

Read before you buy.

Briefs, technical papers and worked cases. Open one to read the detail.

Brief
Imagery in production

Solution brief

The one-page case for on-prem serving.

2 pages For decision makers
Brief
Imagery in production

Solution brief

The one-page case for on-prem serving.

2 pages For decision makers

A two-page summary: why prompts leaving is a real risk, the design targets, the node options and the cost model against a metered public API. Written for the person who signs off, not the one who integrates.
White paper
Imagery in production

Sizing and concurrency white paper

How to pick node count and card.

16 pages For technical evaluators
White paper
Imagery in production

Sizing and concurrency white paper

How to pick node count and card.

16 pages For technical evaluators

The method behind the numbers: how concurrency, prompt length and response length drive KV-cache memory and throughput, how INT8 and FP8 quantisation shift the curve, and worked sizings for 20, 40 and 100 concurrent users on L40S and H100 nodes.
White paper
Imagery in production

API and RAG integration guide

For your own engineers.

22 pages For integrators
White paper
Imagery in production

API and RAG integration guide

For your own engineers.

22 pages For integrators

The OpenAI-compatible endpoints, streaming, per-team keys and rate limits, plus a reference pattern for private retrieval-augmented generation using the on-node embedding and vector store, so your documents stay on the same rack as the model.
Use case
Imagery in production

Contract and document review

Legal work on private weights.

Model14B Roleextraction and summary
Use case
Imagery in production

Contract and document review

Legal work on private weights.

Model14B Roleextraction and summary

A legal team runs clause extraction and plain-language summaries over contracts that must never touch a public API. A 14B model on a single H100 node serves the whole department, with every prompt and document staying inside the firm's own network and logs.
Use case
Imagery in production

Clinical note drafting

Patient data stays on site.

Model8B Roledrafting
Use case
Imagery in production

Clinical note drafting

Patient data stays on site.

Model8B Roledrafting

Clinicians dictate and the model drafts structured notes entirely on hospital hardware, so patient identifiable data never leaves the building. Air-gapped operation and per-team keys satisfy the information-governance sign-off that a cloud tool could not get.
Use case
Imagery in production

Internal code assistant

Source never leaves the network.

Model14B Rolecode completion
Use case
Imagery in production

Internal code assistant

Source never leaves the network.

Model14B Rolecode completion

Developers point their editor at one internal URL and get completion and review over a private model that has never seen the public internet. Proprietary source and secrets stay on the corporate network, and the assistant works offline inside a secure enclave.
Request the files

Get the documents.

PDFs sent on request after a short qualifying call.

SLM inference cluster datasheet Datasheet PDF, on request Sizing and concurrency white paper White paper PDF, on request API and RAG integration guide Integration PDF, on request
“Our developers point their editor at one URL and get an assistant that has never seen the public internet. Same workflow, nothing leaving.”
Engineering director, a defence supplier
Specify it with us

Start with your numbers.

Send your peak users and prompt sizes and we return a sizing, a first-token figure and a fixed price. No meter behind it.

Talk to engineering See all solutions