# AI Infrastructure — operated private inference & RAG for SRE

xeonerix operates private AI / GPU inference as production infrastructure for
SRE and infrastructure operations, under a hard requirement that sensitive
operational data never leaves the organization. The angle is deliberate: this
is an infrastructure engineer running LLM / RAG serving as an operated
platform — placement, isolation, routing, observability, capacity — not an ML
modeller and not a prompt-level AI generalist. This document describes the
approach and the engineering findings. (Specific hardware and topology are
generalized here.)

## Why private inference

SRE knowledge — runbooks, incident history, configs, postmortems — is exactly
the data that cannot be shipped to third-party APIs. That constraint makes
private, on-prem inference an infrastructure problem, not an "add an AI feature"
problem. It has to be placed, isolated, monitored and capacity-planned like any
other production workload.

## Platform shape

A private LLM/RAG platform operated as separate services rather than one demo
container:

- **Interface:** Open WebUI
- **Routing:** LiteLLM as an OpenAI-compatible gateway (virtual keys, model
  routing, rate limiting)
- **Serving:** local GPU inference for Qwen-family models via llama.cpp, plus an
  external NVFP4-class vLLM tier for larger models
- **Retrieval:** Infinity for embeddings/reranking, bge-m3 embeddings,
  cross-encoder reranking, Qdrant vector collections, hybrid search
- **Web:** controlled web search through a self-hosted SearXNG
- **Observability:** Langfuse tracing, VictoriaMetrics / VictoriaLogs, Grafana,
  Vector

Model routing, GPU serving, embeddings, reranking, vector search, traces,
metrics and logs are all handled as first-class operated services. The platform
is used across multiple engineering teams, with first external adoption via an
IDE coding-assistant (Cline) integration.

## Serving profiles

Different hardware classes suit different workloads:

- Memory-bandwidth-bound serving favors decode-heavy and shared-prefix
  throughput (RAG and agentic traffic with shared system prompts / tool lists).
- Cold-prefill-bound serving favors diverse, long, one-off prompts.

Routing traffic between a shared-prefix tier and a diverse-prompt tier is a real
production design decision, not a micro-optimization.

## Engineering findings

From benchmarking inference as a colocated infrastructure workload:

- **CPU-only viability.** Small quantized (3B-class Q4) models are usable on
  tuned CPU-only VMs with NUMA pinning, hugepages and CPU/cache isolation.
- **Colocation cost.** Inference next to memory-bandwidth pressure degrades
  inference decode by ~17% and the neighbor workload by ~8%.
- **Datastore co-tenancy.** With inference colocated next to a latency-sensitive
  datastore under NUMA/hugepage isolation: ~10% throughput loss and ~13% p50
  latency increase on the datastore.
- **Bandwidth capping isn't a silver bullet.** Memory-bandwidth capping does not
  universally protect real services; datastore-style workloads were limited more
  by L3 cache contention than by raw memory bandwidth.
- **Prefix cache is a design point.** For RAG and agentic systems with shared
  system prompts and tool lists, prefix-cache behavior is a major driver of
  effective throughput and should be designed for explicitly.

## Production interpretation

Private SRE inference needs the same discipline as any production service:

- NUMA-aware placement
- hugepages
- CPU and memory isolation
- explicit concurrency limits
- model-size and context-budget sizing
- monitoring of memory bandwidth, L3 contention and neighbor latency
- routing between shared-prefix and diverse-prompt tiers
- clear data-boundary guarantees

## Applied use cases

- AI-assisted incident response
- postmortem generation
- merge-request review
- runbook and knowledge retrieval
- infrastructure Q&A over private documents