The unglamorous layer: gateways, evals, sandboxes and traces

A team I visited in February had spent eleven weeks tuning prompts and had no idea whether they were improving anything. Every change was evaluated by three engineers reading twenty outputs and forming an impression. When we built a proper golden set — 200 real cases with human-graded answers — it turned out their most celebrated prompt revision had made accuracy worse by four points and cost 30 per cent more. Nobody had lied. They just had no instrument.

This is the layer nobody writes launch posts about and everybody needs. Four categories matter, and I will go through them in the order I would install them.

01 — The gateway

Put a proxy between your application and every model provider on day one. It gives you one API surface, per-team budgets, key rotation, automatic fallback when a provider degrades, caching, and a single place to log every request. Without it, switching models is a refactor; with it, it is a config line.

My default is LiteLLM: open source under MIT, supports 100-plus providers through an OpenAI-compatible interface, and the self-hosted proxy is genuinely free. The enterprise tier adds SSO and audit logging and is quoted annually. Portkey is the more polished managed option with guardrails built in, on a generous free tier and paid plans starting in the tens of dollars a month. OpenRouter is the easiest way to reach obscure models — it takes a small percentage on credits — and I use it for evaluation breadth rather than production traffic, because you are adding a hop and a third party to your data path.

02 — Observability and tracing

You cannot debug what you cannot see. A trace should capture the full prompt, the tool calls with arguments, retrieved documents, token counts, latency per span, and cost. When a customer complains about an answer from last Tuesday, you should be able to pull that exact run in under a minute.

Langfuse is my recommendation for most teams: open source, self-hostable, OpenTelemetry-native, with a free managed tier and paid plans starting around $29 a month. LangSmith is the smoothest option if you are already deep in LangChain or LangGraph, at roughly $39 per seat per month plus trace-volume charges — watch that second part, it is the line that surprises people. Braintrust is stronger on evaluation workflows than pure tracing and is priced for well-funded teams. Arize Phoenix is a solid free self-hosted option if you want no vendor at all.

03 — Evaluation

Evaluation is where I see the widest gap between mature and immature teams, and it is not really a tooling problem. It is a data-collection problem wearing a tooling costume.

Build the dataset before you buy anything. Two hundred real production cases, graded by someone who knows the domain, stored in version control as plain JSON. Then wire whatever harness you like on top. Use deterministic assertions wherever you can — did it return valid JSON, did it call the right tool, did it include the required disclaimer — because those are cheap, fast and never disagree with themselves. Reserve LLM-as-judge for genuinely subjective dimensions, and calibrate the judge against human labels before trusting it. In my experience an uncalibrated judge agrees with domain experts about 70 per cent of the time, which is worse than useless because it looks like a number.

Run the suite in CI on every prompt, model or retrieval change. If that sounds heavy, note that a 200-case suite on a mid-tier model costs pennies per run.

04 — Sandboxes and browsers

If your agent writes and executes code, it needs an isolated runtime. E2B and Modal both give you fast-booting sandboxes with per-second billing; Modal advertises sub-second cold starts and bills compute by the second, which suits bursty agent workloads far better than an always-on container. For agents that drive websites, Browserbase handles headless browser infrastructure, session recording and proxy rotation, priced per browser-hour. Rolling your own Playwright fleet is a fortnight of work and a permanent maintenance tax; I have watched two teams do it and regret it.

The stack I would install this week

For a team of five shipping their first production AI feature: LiteLLM self-hosted as the gateway, Langfuse self-hosted for traces, a 200-case golden set in the repo with a plain pytest harness, pgvector inside the Postgres you already run rather than a dedicated vector database, and E2B if you need code execution. Total marginal licence cost: zero. Total setup: about three days.

The pitch for a dedicated vector database gets compelling somewhere past ten million vectors or when you need sub-50ms filtered search at high QPS — Qdrant and Turbopuffer are the two I would shortlist. Below that, pgvector with an HNSW index is fine, and one fewer system to operate is worth real money. The most common mistake I see in this layer is buying scale you will not reach for two years, and paying for it in operational complexity every single week until then.