Hippo: The Go LLM Client I Built Because AI Apps Need Memory, Budgets, and Tools

Hippo unifies Anthropic, OpenAI, and Ollama behind one interface, then adds the parts most clients skip: typed memory, cost-aware routing, tool calling, MCP support, privacy tiers, and an embedded interface for inspecting all of it. Read as a tool it is neat. Read as evidence it is more interesting, because the feature list is a map of every place a naive AI feature quietly breaks in production. The people who build the model call and stop have not yet hit those walls. The people who have hit them build something that looks a lot like this.

When a team skips the operating layer, it does not avoid the work. It grows the layer by accident. A provider wrapper here, a memory hack there, a YAML file nobody understands, a budget check buried in one code path, a tool loop copied from an example, a local fallback that only runs on one laptop. Each piece looks reasonable on the day it is added. Six months later the AI feature is not one system anyone can reason about. It is a pile of glue, and the person who wrote it has left. The rare skill is seeing that outcome from the start and designing against it, which is what naming memory, budgets, routing, and privacy as first-class concerns actually means.

Take memory, the place most AI products quietly fail. The common move is to throw everything into a vector database and call it memory. That is not memory. It is a search index. Real memory has shape: a working note that should decay in hours, an event from last month that should still be retrievable, a stable preference that should never disappear because it got old. Treating those as the same record is why so many assistants feel amnesiac one minute and cluttered the next. Hippo stores them as distinct typed kinds with their own decay behavior, and combines keyword recall with semantic recall while pulling in the conversation around a hit instead of one isolated line. None of that is exotic. It is simply what someone who has watched memory fail in the wild knows to build, and what someone who has not will not think to.

Cost and privacy are the same story. AI systems get expensive quietly, because one call is cheap but a background agent that retries, streams, calls tools, summarizes, and stores memory is a recurring bill nobody approved. Privacy fails the same way, treated as a paragraph in a README rather than something the system enforces. The non-obvious move is to make both of them routing decisions: a policy that says reasoning goes to a strong cloud model, private work stays local, cheap generation uses a small model, and nothing exceeds a set ceiling, all in one file you can inspect and change without editing code. Whether a given request should ever leave the machine stops depending on a developer remembering. It becomes a rule the router obeys. That is what it means to point AI deliberately instead of hopefully.

Trust is engineered the same way, in the open. When an agent does work, the final answer is the least interesting part. What you need to see is what it tried, which tools it called, what came back, where it failed, and what it cost, streamed as it happens rather than reconstructed from logs after a customer complains. A system you cannot watch is a system you cannot trust, and a system you cannot trust does not belong in front of a business. The model is the commodity in all of this. The operating layer is the difference between a demo that impresses a room and a system that survives contact with real users.