I recently saw some blog post or HN comment talking about having a different way to model LLM context, not through the usual chat history or tree, but with a searchable log. When you think about LLM interactions as conversations, it makes sense to have the current tree/chat structure. But it doesn’t have to be that way.

We can instead have a huge log of all the messages and conversations you had with the LLM, with the LLM querying the past on demand, loading whatever is relevant to your current question. Most likely your latest turns would be the most relevant, but a queryable log of all interactions with the LLM feels like it would be much better if you can get the execution right. Or it doesn’t even have to be all your interactions with the LLM, why not enrich the log with whatever data you want? Maybe sleep or weather data or whatever. Or you can let the LLM save some structured data to your log.

I use claude a lot when I’m in the gym, asking for load suggestions, or which exercise to substitute the current one with if a machine is taken. I put in the exercise logs from my exercise tracking app manually. It does a good enough job of remembering the last session, but I just used up all the context for that conversation now, so this approach could be much better. Same thing with my diet, I just put in what I ate, might be home cooked or delivered, and it looks through the history to guess the macros. Later in the day I can ask it where I’m at and whether I should eat more or not.

You can have a tiny search model to search through each message, maybe summarize each message and add tags real-time. Let the LLM come up with structured data or other data sources or whatever.

I will try to do this with a sqlite opfs browser app, let’s see how far I can get.

First implement a regular chat log, then add metadata (questions, tags, summaries, chunks?) for search etc to work. Looks like there is sqlite-vector which works with sqlite wasm. Then LLM provided structured data types. Or maybe user provided is a better start. Anyway just get the sqlite wasm working first, and a nice to look at UI.

I guess a super big downside of this is cache hit rate. But it can be optimized, I can probably check the prefix of the last request and prioritize the prefix. There has to be some way to optimize cache hits in this. Cache hit rate makes the tree a super sensible way to model the context. Especially summarizing branches and continuing like pi does. Although the model spits out so many tokens for each answer, it might even be better to just restructure the whole context like this. Well benchmark and see I guess. But how to benchmark something like this? I’ll just use it as my daily driver.