A Small Guide to LLMs: Memory

by Whale Z (on JAI and Saucepan)

Note: this guide briefly explains some basic concepts that will help you understand how the very tech you use everyday works, especially surrounding the 'memory' of a model. It's slightly (very slightly) technical, but nothing requires prior knowledge.

LLMs

Large Language Models. The type of AI models we're using to chat and are concerned with here. They deal with natural languages instead of images, audio etc.

Examples:
✅Deepseek, Claude, all the models you can use to chat on Saucepan are LLMs
❌Stable diffusion, nijijourney are not LLMs

"So how does it work?"

LLMs are, essentially, glorified autocomplete. They have been trained with massive amount of text and learned the patterns of how our languages work. And after reading the information you provided to it, it predicts what is mostly likely to come next and complete it, one word after another.

Tokens

'Token' is a unit of text. LLMs process text using this unit. A 'token', depending on tokenization method, might be a word, a punctuation mark, a part of a word(a subword), etc. In English, one word is on average around 1.3 tokens.

Examples: A tokenizer might break down the sentence "I work internationally." into tokens ['I', 'work', 'inter', 'nationally', '.']

Prompt

A prompt is simply the input or instructions sent to an LLM. Each time you send a message to your bot/companion, it's a prompt. It includes all the definition and chat history, your persona info, and other technical things like the temperature you set (or didn't set, if you just use the default).

Context Window

Also called 'context limit'. The maximum number of tokens you can input into a model, or how long your prompt can be. That's all there is to it.

You will see in many discussions that higher context window is often linked to performance of a model. This is true to a degree, as models that can handle more tokens effectively usually have a larger window. However, models cannot 100% process all the things within the window. In fact, the more they have to process, the dumber a LLM becomes. A lot of models will advertise how they have massive context window while omitting how models actually behave within those windows.

"What happens if we reach the context limit?"

This is the actual part where context limit matters. The website will only send information to LLMs within the context window.

First, two terminologies here:

  1. perm tokens, or permanent tokens, are tokens that are always included in a prompt. This include your bot definition, your persona info, and your advanced prompt.
  2. temp tokens, or temporary tokens, are your chat history (including initial message).
    Depending on the website, lorebook entries might be perm tokens or temp tokens, but usually even if they're temp tokens, they have priority.
    Note: the LLMs do not distinguish between perm and temp tokens once it receives the prompt, the website does (to decide what is going to be included in the prompt sent to LLMs).

As your chat increases in tokens, you will eventually reach the maximum context limit, and the website will then have to choose which information to abandon. It will always choose to abandon temp tokens, and always keep the perm tokens.

Let's say for example, your bot definition is 1500 tokens, your initial message/intro is 1000 tokens, and the model has a context window of 8000. This leaves 8000-1500-1000=5500 tokens for chat history. If your chat uses up all the 5500 tokens, the website will stop sending the exceeding tokens, starting from the initial message.

Put it plainly, a bot might not remember he kissed you 10 messages ago even if it's in the prompt, but if it's beyond the context window, there's absolutely no chance he will remember that at all. It simply isn't getting sent to the LLM. (The bot might still come up to "sure I remember that" even so, by pure guessing that you might want that answer)

Memory

So, what does matter, if context window doesn't equal performance?

You might have heard about how much a model can 'remember', but having seen what is written above, a problem surfaces quickly: What is the definition of 'memory' we're talking about here? Do you mean how many words you can input to a model, or actually, how much information they can effectively process?

A common problem is: we often want to know how much information a model can effectively process, but people talk about 'context window' all day, which in fact, does not mean much in terms of model performance. What we should focus on instead, is a model's long context ability.

Long Context Ability

You might have seen a lot of different words when it comes to models' ability concerning long context. Comprehension, coherency, etc. They are all slightly different aspects, but essentially all talking about how well a model behaves when a prompt has a lot of tokens.

A spoiler for everyone: they behave poorly, especially for creative writing tasks.

An example here: According to a benchmark testing reading comprehension where models are asked to answer factual questions about a story given to them, when the story gets to 8k long, Claude-opus-4 achieved a 66.7/100 score, while deepseek-v3.2-exp achieved a 52.8/100.

And those are what we call SOTA models, state-of-the-art models, let alone the smaller ones. For context, a deepseek v3.2 has a parameter of 685B whereas the free models ai websites provide are typically around or less than 16B. For people with premium subscription on Saucepan, that model is a fine-tuned deepseek iirc.

(The result improves drastically for thinking versions of those models, but bear in mind at 16k the top thinking models still get 20-30% of the story wrong.)

There are more benchmarks and research papers out there. It's safe to say that most models will struggle to even 'remember' all the facts of a story once you get to 10, 20 messages. Now, consider what a model have to do apart from simply understanding. They still need to think about plot, characterisation, style, etc.

LLMs don't process tokens uniformly

What does this mean? The natural assumption might be that the 1st word in a prompt is just the same to LLMs and gets the same treatment as the 1000th one, but that is not the case.

Researchers have observed this effect called "lost-in-the-middle effect". In simple words, the middle part of a prompt is most likely to get lost when the context is long. When relevant information is at the top or bottom part of a prompt, the model performs better. If the model needs to access something in the middle, then performance degrades.

Which means, if something is important, put it at the start or end of a prompt will work better. In terms of chatbots, that means at the start of a bot definition, or sent together with your message in a chat. It’s worth noting that the very end of the prompt (the most recent message) usually has the strongest influence on the immediate next response. So, what users do in the last message is very important.

"I use 16k bots and they totally work!"

Now what is the definition of 'working' here? That they can produce a response? Or they can actually remember all the relevant details and apply them when appropriate?

A response will always be given regardless, and a model can still convince you they remember they kissed you even if it's far gone beyond the context limit. They can just read the atmosphere and deduce a whole bunch of things.

Different bots naturally require different degrees of performance. People using fantasy RPG bots will be more affected than people using modern romance bots. And smut? You don't need to know much about a character to write good smut, but you do need to remember a LOT more to continue plot that involved 7 NPCs and a whole fantasy world setting, and make it coherent.

-# That being said, anyone who's tried nsfw with non-human OCs knows what kind of errors they have to deal with.

Conclusions? Solutions?

If you want optimal performance, slim down on the information provided to the models. The longer your prompt, the more a model will have to process and the less they understand. So, more tokens = dumber bots.

For users: there limited things you can do. The most important is making use of chat memory function. For every 20 messages or less, depending on the model, you should consider update a summary of what has happened, with all the important details to emphasis it to the bot. You can also use commands to guide the bot into certain directions.

Another thing: don't use bloated advanced prompt. 500 tokens is all you need. As we established, more tokens doesn't mean smarter. Also, LLMs are bad with complicated rules. Keep it as simple as possible.

✅Use chat memory frequently
✅If you use proxy, set your context to 16k
❌Don't use overly bloated or complicated advanced prompts
❌Don't make your persona super long

For creators: you can now see why people say a bot should have 2k perm tokens at most. I mean, forget about 8k, many medium to small size models will instantly fail to understand half of the content at only 4k, perm and temp included. If you want an optimal experience for everyone, and not just users who can afford claude opus, be careful about the size of the bot.

Core principle here is be direct and concise. Spell out things that might cause confusion, while save your tokens if it's obviously implied stuff (adjust by model performance). Also, more advanced models will typically need less description or instructions to achieve your goal.

Note: My advice here is suitable for beginners and conservative for reasons. If you're sure that you know what you're doing, feel free to just ignore or expand on it, but I just need people to know the consequences.

References:

  1. OpenAI helpdesk on tokens, https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
  2. IBM Think, What is generative AI?, https://www.ibm.com/think/topics/generative-ai#257779831
  3. Context Rot: How Increasing Input Tokens Impacts LLM Performance, https://research.trychroma.com/context-rot
  4. Lost in the Middle: How Language Models Use Long Contexts, https://doi.org/10.48550/arXiv.2307.03172
  5. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval, https://aclanthology.org/2025.findings-emnlp.1264.pdf
  6. NoLiMa: Long-Context Evaluation Beyond Literal Matching, https://doi.org/10.48550/arXiv.2502.05167
  7. Fiction-liveBench, https://fiction.live/stories/Fiction-liveBench-Mar-25-2025/oQdzQvKHw8JyXbN87
Edit

Pub: 10 Feb 2026 04:04 UTC

Edit: 13 Feb 2026 22:10 UTC

Views: 172