Google Gemini 1.5 now comes with a massive one million token context window, dwarfing its direct competition in ChatGPT, Claude, and other AI chatbots.

It sounds like a massive upgrade and could set Gemini apart. It's a little difficult to grasp its full extent—but Gemini's enormous context window could be a game changer.

What Is a Context Window?

While responding to your queries, like explaining a concept or summarizing a text, AI models have a limit on how much data they can consider to generate a response. The limit on the text size it can consider is called a context window.

Here's another way to look at it. Let's say you go to a grocery store to get groceries without your grocery list. The limit on how many groceries you remember when shopping is your context window. The more groceries you can remember, the higher the chances of not messing up your shopping plans. Similarly, the larger the context window of an AI model, the higher the chances of the model remembering everything it needs to provide you with the best results.

At the time of writing, Anthropic's Claude 2.1's 200k context window is the largest context window of any generally available AI model. This is followed by GPT-4 Turbo with a 128k context window. Google Gemini 1.5 is bringing a one million context window, four times larger than anything in the market. This leads to the big question: what's the big deal with a one million token context window?

Why Gemini 1.5's Context Window Is a Big Deal

To put it in a clearer perspective, Claude AI's 200k context window means it can digest a book of around 150,000 words and provide answers to it. That's massive. But Google's Gemini 1.5 would be able to digest 700,000 words at a go!

When you feed a large text block into AI chatbots like ChatGPT or Gemini, it attempts to digest as much of the text as possible, but how much it can digest depends on its context window. So, if you have a conversation that runs into 100k words on a model that can only handle 28k and then start asking questions that require it to have complete knowledge of the entire 100k words worth of conversation, you're setting it up to fail.

Imagine only watching 20 minutes of a one-hour-long movie but being asked to explain the entire movie. How good would your results be? You either refuse to answer or simply make stuff up, which is exactly what an AI chatbot would do, leading to AI hallucinations.

Now, if you are thinking that you've never had to feed 100k words into a chatbot, that's not the whole consideration. Context window transcends just the text you feed an AI model in a single prompt. AI models consider the whole conversation you've had during a chat session to ensure their responses are as relevant as possible.

So, even though you are not feeding it a 100k word book, your back-and-forth conversations and the replies it provides all add to the context window calculation. Wondering why ChatGPT or Google's Gemini keeps forgetting the things you've told it earlier in a conversation? It likely ran out of context window space and started to forget stuff.

A larger context window is particularly important for tasks requiring a deep understanding of the context, such as summarizing long articles, answering complex questions, or maintaining a coherent narrative in the generated text. Want to write a 50k-word novel that has a consistent narrative throughout? Want a model that can "watch" and answer questions on a one-hour video file? You need a larger context window!

In short, Gemini 1.5's larger context window may significantly improve the performance of its AI model, reducing hallucination and significantly increasing accuracy and ability to follow instructions better.

Will Gemini 1.5 Live Up to Expectations?

If everything goes as planned, Gemini 1.5 could potentially outperform the best AI models in the market. However, considering Google's many failures at building a stable AI model, it's important to err on the side of caution. Bumping up the context window of a model alone doesn't automatically make the model better.

I've used Claude 2.1's 200k context window for months since its release, and one thing is clear to me—a larger context window can indeed improve context sensitivity, but problems with the core model performance can make larger context a problem of its own.

Will Google Gemini 1.5 give us a game-changer? Social media is currently filled with glowing reviews of Gemini 1.5 from early-access users. However, most 5-star reviews stem from rushed or simplified use cases. A good place to check how Gemini 1.5 would perform in the wild is inside Google's Gemini 1.5 technical report [PDF]. The report shows that even during "controlled testing," the model couldn't retrieve all the tiny details of documents well within the size of its context window.

A one million token context window is indeed an impressive technical feat, but without being able to retrieve the details of a document reliably, then a larger context window is of little practical value and could even become a cause of declining accuracy and hallucinations.