Photo by Janne Simoes on Unsplash
***WARNING, DIATRIBE***
“Context window”… Whoever coughed up that phrase is either trying to fool people, works in marketing, or both, because “context window” has absolutely nothing to do with context. It should just be called “processing window” because it’s just the amount of text that an LLM could process at a time. Wow-wee. Doesn’t sound fancy enough though, so “context window” it is even though it’s NOT.
In fact, LLMs don’t deal with context AT ALL. All the talk you hear about “context” in connection with LLMs? Forget them all, ignore them. They’re hogwash.
What do LLMs actually deal with? Just their datasets, basically spewing them back. “Parroting,” as some may put it.
If you’ve read my other writings, you’d say “of COURSE it doesn’t deal with context, when it doesn’t refer to anything at all!” Yes, while I did say that, this time I’ll put things in more concrete terms with a firmer picture.
One definite proof that LLMs just spit their data back regardless of context is how their performance absolutely TANKS in even elementary-level problems from just switching around the wording even a little bit:
we found existing cutting-edge LLMs unanimously exhibits extremely severe recitation behavior; by changing one phrase in the condition, top models such as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary school-level arithmetic and reasoning problems.
Oof. Why did it screw up so badly even though it’s supposedly dealing with the same context each time?… Because it’s parroting the data regardless of any actual context. When the pattern is even a little bit off, the model jumps the shark.
Are you using ChatGPT, Claude, Gemini, or any other bot that spits out answers without giving you links to check those answers? Well you better think twice… Let me show you why.
Here, I’m using Perplexity as an example (but you can also use Microsoft Copilot, they’re just different flavors of crap so it’s entirely up to personal taste) where I ask the question “how much fish sauce to add to Greek yogurt to enhance flavor”:
That’s great, and the results even includes all the yogurt sauce links I could check accuracy and relevance with… except I want to just eat the yogurt without using it as a sauce for something else. (Why fish sauce? Because I want to, dammit.) Let me change the question (ergo, change the CONTEXT) and ask again:
Now we’re cooking! …or not. No, I’m not cooking with yogurt sauce and just want to have a bit of fish sauce umami yogurt to eat. But hold on! All the reference ARE STILL POINTING TO YOGURT SAUCES! One look at the entire list of sources confirms this:
Included in the list is a page result from a site search for “fish sauce yogurt” that had nothing directly to do with what I wanted either, a link to some salad recipes that again had to do with sauces, and a whole load of other stuff that had nothing to do with just eating savory yogurt, much less fish sauce savory yogurt. It just keeps going to sauce-related stuff because THAT’S what shows up most in the data that the model ingested. Now if the dataset is the reverse (that 99% of everything just talks about how to eat savory yogurt BY ITSELF) you can bet that it would refuse to actually talk about yogurt sauces even when it LOOKS like it’s talking about yogurt sauces even when really just regurgitating from sources detailing yogurt-for-lone-consumption. That’s how these things work.
Sidebar: All other kinds of generative AI work the exact same way; See my ouroboros examples in one of my posts discussing this. It kept drawing anything BUT ouroboros when I asked for pictures of ouroboros because the BULK of the training data are NOT images of ouroboroses but those of “regular” dragons, snakes, worms, etc etc etc that DON’T eat their own tail. Now IF it’s the reverse- That the bulk of the data ARE ouroboroses, then the stupid image generators would keep drawing ouroboroses even when you ask for dragons, snakes, and worms! Yeah, the dumb “most likely crap in the shitbag” approach. Another demonstration is this surreal video posted as part of a LinkedIn post.
Now where the heck DID the engine get its answers from then? Yep it’s random shit from all the salad dressing / yogurt SAUCE stuff. As usual, the bot had trouble just saying “I DON’T FUCKING KNOW”.
But hey; At least I get to KNOW that because at least Perplexity lists source links. With something like ChatGPT you wouldn’t even know; You’d just treat the “answers” as if the bot actually got all of them from sites that DIRECTLY discuss the subject! 🤡 You’re practically getting clowned on if you do that.
(Why don’t Sam put links in ChatGPT so people could check the output? Doesn’t he want people to see how wrong the answers could g- Oh wait…)
Do you start to see what could possibly be wrong with the “answers” now?
These things don’t “give a hoot” about the context (well, it’s not conscious nor thinking so of course not but) because it doesn’t deal with context AT ALL.
LLMs have been clowning all of us this entire time. The "answers" are coming from everywhere that just happens to have the pattern that's in a certain proximity with the preceding pattern (e.g. your prompt) and proximity ISN'T CONTEXT. The answer could just be coming from anything that HAPPENS to contain that pattern.
Edit: Sordid tale from a victim of evil clowning below.
Edit2: Another tale of clowning, but fortunately it was spotted in time:
