About — AI Inference Platform

What this is

A self-hosted AI inference platform — the infrastructure layer that sits between an application and a model. The gateway, the worker, the queue, the observability, the benchmarks — all of it is here, running on real hardware, doing real work.

The platform handles two things: live inference, where a request comes in and a response comes straight back, and batch processing, where you submit a list of prompts as a job and a background worker processes them one by one, storing each result.

Why it exists

Most AI projects are built around the model output. This one is built around everything that makes the model reliable in practice — how requests flow in, how failures get handled, how batch work gets queued and processed, and how you actually know whether the system is healthy or struggling under load. Those pieces are what turn a model call into something you can actually depend on.

How to use this page

Two tabs. Single inference sends one prompt and shows the response. Batch jobs lets you build a list of prompts, submit them as one job, and watch the worker process them — status updates every few seconds until the job completes, then per-item results.

Both tabs have a simple/detailed toggle. Simple shows just the output. Detailed shows the request ID, which model answered, how long it took, and token counts.

The model selector shows what's available. The self-hosted option uses a local open-weight model served by vLLM — the model runs entirely on local hardware with no external API calls. The cloud options tap into Anthropic or OpenAI directly. Any unavailable options will be greyed out in the selector, and if the backend itself isn't live — taken down, not hosted, or not currently running — the offline banner will say so.

How it works

The gateway validates every request before the model sees it, calls the model server, and maps failures to honest HTTP status codes — a timeout is different from the server being unreachable, which is different from a malformed response. Each one has a distinct code so callers know what actually went wrong.

The worker runs a continuous loop, claiming batch items from Postgres one at a time with row-level locking so multiple workers can run concurrently without processing the same item twice. It commits before calling the model so the lock window stays short. Results are written back per item — one bad prompt fails only that item, not the whole job.

Observability runs through both. Prometheus metrics on the gateway and worker. Structured JSON logs with correlation IDs so any request or batch item is traceable across log lines. Grafana dashboards that provision automatically when the stack starts.

Why it matters

Running a model in production requires more than the model. You need a serving layer that handles traffic reliably, a way to process larger workloads without blocking live requests, and enough visibility to know when something is wrong before users do.

This platform demonstrates each of those pieces working together — measured under real load with real numbers. The benchmark results in the repo come from actual runs on actual hardware. The observability isn't decorative — it's what made the benchmarking story visible in the first place.

The self-hosted model path shows what it looks like to own the serving layer completely. The cloud provider path shows the same infrastructure routing to an external API. Both go through the same gateway, the same worker, the same queue — the model backend is swappable without touching the architecture.

The stack