Making an LLM gateway is as cheap as making an HTTP server

There was a time when a reasonably proper, performant HTTP server was a thing that required a significant investment. Apache, Lighthttpd, Nginx. To make your own was a multi-year commitment, so you needed to be confident that it was going to be worth it. That changed when Go hit the scene, and suddenly Caddy had interesting, innovative, quirky, useful features. And any microservice you built could have it's own HTTP server right there in the stdlib.

Today LLM gateways are that within reach. There is real value to being able to add observability, routing, etc in a gateway. It's so fast and easy to build this yourself, either to understand how the systems work, or to build the right set of capabilities customized for your software. An enterprise class gateway is still a useful thing for a large scale deployment, but a custom in-app gateway is sometimes the cheaper and better thing to use. And like HTTP servers, they compose nicely.

Zag is a personal desktop gateway that I built over the last few weeks to answer questions I had about comparing latency between inference providers, understanding API shapes, measure cost efficiency between different agents, experiment with the new class of decision models, and greatly reduce the effort it takes me to test things. You can try it if you want (macOS only) https://kindlyops.github.io/zag-releases/. But I really recommend making your own instead.