The Two Lives of an LLM Request: Prefill and Decode

The strange pause

Okay, so lets say, you open a chat app and you paste a long document. Then, you ask, "Hey, can you summarize this?"

You hit enter.

Nothing happens. One second. Two seconds. Maybe three.

Then words start to flow. Fast and smooth. One after another.

Have you ever wondered about that pause? Why does the model sit silent at first, and then talk without stopping?

The answer is simple. Your request lives two lives. Each life has its own job. Each life stresses the GPU in a different way.

The first life is called prefill. The second is called decode.

By the end of this blog, you will know both. You will know why one is limited by math and the other is limited by memory. And you will see why this one idea shapes almost every trick in LLM serving.

Let's begin.

First, how does an LLM write?

An LLM does not write a whole answer at once. It writes one token at a time.

Okay, now what is a token?

A token is a small piece of text. Often a word. Sometimes part of a word.

Why one at a time? Think about how you write a sentence. You don't know the last word when you write the first one. You pick a word. You look at what you have. Then you pick the next.

An LLM works the same way. At its core, it is a very good guesser. Give it some text, and it guesses what token should come next. It makes one guess at a time. And each guess needs the guess before it.

Take a tiny example. The text so far is "The sky is". The model guesses "blue". Now the text is "The sky is blue". The model reads that, and guesses the next token. Maybe a full stop. Then it reads again.

That back-and-forth is the whole trick. Let's write it down as a loop.

Here is the loop:

Look at everything written so far.
Pick the next token.
Add it to the text.
Go back to step 1.

That's it. The model repeats this until it decides to stop.

So when you send a prompt of 1,000 tokens, the model must do two things:

First, read your 1,000 tokens.
Then, write the answer, one token per loop.

Reading is prefill. Writing is decode.

Keep this picture in mind. We will come back to it again and again.

Meet the kitchen

To understand why these two phases feel so different, we need to look at the GPU. Let's use a kitchen.

Imagine a giant kitchen. It has thousands of cooks. All of them work at the same time. These cooks are the GPU's compute units. They do the math.

Now imagine a huge recipe book. It is so big that it does not fit in the kitchen. It sits in a pantry next to the kitchen. This book is the model's weights. A 7 billion parameter model in 16-bit precision is about 14 GB. That's a very thick book.

You may ask: why not keep the book inside the kitchen? Simple. The kitchen is small. Very small.

A GPU has two kinds of memory. One is tiny and super fast. It sits right next to the cooks, like the kitchen counter. It holds only a few tens of megabytes. The other is big and slower. It is called HBM. It is like the pantry. It holds tens of gigabytes.

A 14 GB book cannot fit on a counter of a few tens of megabytes. It is hundreds of times too big. So the book stays in the pantry. The cooks can only borrow a few pages at a time.

To cook anything, the cooks need pages from the book. So someone must carry pages from the pantry to the kitchen. The speed of this carrying is called memory bandwidth.

Now we have two speeds:

How fast the cooks can cook. (Compute speed.)
How fast pages can be carried in. (Memory bandwidth.)

The kitchen runs well only when both are busy. If the cooks wait for pages, they are wasted. If pages pile up waiting for cooks, the carriers are wasted.

Hold on to this kitchen. It explains everything.

Life one: Prefill

Your 1,000-token prompt arrives.

Here is the key fact. The model already knows all 1,000 tokens. You gave them to it. It does not need to guess any of them.

So it does not need to go one by one. It can process all 1,000 tokens at the same time.

In kitchen terms: the head chef grabs a few pages from the book. Then he hands the same pages to the cooks, and gives each cook a different token to work on. All the cooks are busy. Everyone is cooking.

This is prefill. The model reads the whole prompt in one big parallel pass.

What comes out of it?

A first predicted token. This is the first word of the answer.
A set of notes about every prompt token. We call this the KV cache. We'll meet it soon.

Prefill is why the pause exists. The model is busy reading. The longer your prompt, the longer the pause.

The time from pressing enter to seeing the first word has a name. It is time to first token, or TTFT. Prefill is what decides it.

Life two: Decode

Now the model has to write the answer.

Here is the problem. It does not know the next token yet. It must create it.

And token 2 depends on token 1. Token 3 depends on token 2. There is no way around it. You cannot write the 10th word before the 9th.

So the model cannot go parallel anymore. It must go one token at a time.

Each loop, the model handles just one new token. That's all.

But look at what this means for the kitchen. For each single token, the head chef must still go to the pantry and bring the book's pages. All of them. Every layer. The entire 14 GB.

And then he hands those pages to... the cooks. But there is only one token to cook. So one cook works. The other thousands of cooks stand around.

Then the token is done. The pages are put back. And the next loop begins. Same trip. Same 14 GB. Again.

This is decode. It is a long walk to the pantry for a tiny bit of cooking.

Why can't the GPU keep the weights from the last token and skip the trip? Because its fast memory is tiny. It holds only a few tens of megabytes. The weights are 14 GB. A model also has many layers. The GPU works through them one by one. It copies a small chunk of a layer's weights from the big memory (HBM) into fast memory. It uses that chunk. Then it replaces it with the next one. By the last layer, the weights of the first layer are long gone. Nothing is deleted. HBM always keeps the full copy. The code decides the order, and it is known in advance, because layers run in a fixed order.

Now the next token arrives. It must pass through every layer again, using the same weights. But they are no longer in fast memory. So the GPU must fetch them all again. And there is no shortcut. Token 2 cannot start until token 1 is done, so we cannot share one trip between many tokens. Prefill could do that. Decode cannot. (At least for one user. Later, we will see how serving many users together changes this.) So every single token pays for one full trip through the 14 GB of weights.

The time between one output token and the next is called time per output token, or TPOT. Decode decides it.

The notebook: KV cache

Wait. If the model handles only one new token per loop, how does it remember the old tokens?

This is where the KV cache comes in.

Inside every layer, the model uses something called attention. Attention lets the new token look back at all earlier tokens. To do that, every earlier token leaves behind two small vectors. A key and a value. That's the K and V.

Now here is the clever part. Those keys and values for old tokens never change. So why compute them again?

The model does not. It saves them in a notebook. This notebook is the KV cache.

During prefill, the model fills the notebook for all prompt tokens.
During decode, each new token reads the notebook, does its work, and adds one new line.

Without this notebook, every loop would re-read the whole text from scratch. That would be painfully slow.

A tiny idea: work per byte

Now we get to the heart of the story. Which phase is compute bound? Which is memory bound? And why?

To answer, we need one simple idea. Ask this question:

For every byte I carry in from memory, how much math do I do with it?

This is called arithmetic intensity. Math done, divided by bytes moved.

High intensity: lots of math per byte. The cooks are the bottleneck. This is compute bound.
Low intensity: very little math per byte. The carriers are the bottleneck. This is memory bound.

Every GPU has a tipping point. Take an H100, for example. Roughly:

It can do about 1,000 trillion math operations per second (in 16-bit).
It can move about 3.3 trillion bytes per second from memory.

Divide the two. You get about 300 operations per byte.

So here is the rule. If your work does more than about 300 operations for every byte it loads, the cooks are the limit. If it does less, the carriers are the limit.

Now let's test both phases.

Why prefill is compute bound

Take a 7B model. Its weights are 14 GB.

In prefill, we load those weights once. Then we use them on 1,000 tokens at once.

Each weight gets used for 1,000 different tokens. That is a lot of math for every byte we carried.

Let's count. Each token needs about 2 operations per parameter. So:

Math: 1,000 tokens × 2 × 7 billion = about 14 trillion operations.
Bytes moved: about 14 GB.
Work per byte: roughly 1,000 operations per byte.

That's far above our tipping point of 300.

So in prefill, the carriers finish early. They sit and wait. The cooks are the ones working at full speed. The GPU's math units are the bottleneck.

Prefill is compute bound.

There is one more thing. In math terms, prefill is a big matrix times matrix multiplication. The weights are a matrix. The 1,000 tokens form another matrix. Big matrix multiplications are exactly what GPUs love. They reuse each loaded number many times.

And attention in prefill adds even more math. Every token looks at every earlier token. That cost grows with the square of the prompt length. So long prompts make prefill heavier still.

But wait. If all 1,000 tokens go in at the same time, how does each token still see the ones before it?

Here is how it works.

Inside attention, every token makes a question. It asks, "Which earlier tokens matter to me?" (The paper calls this the query.) Every token also carries a label that says what it holds. (This is the key.) The model matches each question against the labels. It does this for all 1,000 tokens together, as one big block of math.

But there is one strict rule. A token can only look backward. Token 5 can see tokens 1 to 5. It cannot see token 6. The model hides the future behind a mask. This is called the causal mask.

So the tokens enter together, yet each one sees only its own past. The order is kept. Nobody peeks ahead.

This is also why the cost grows so fast. Token 1 looks at 1 token. Token 1,000 looks at 1,000 tokens. Add it all up, and you get roughly half of 1,000 × 1,000 matches.

And as each token is processed, its key and value are saved in the notebook. That is how the KV cache gets filled.

Why decode is memory bound

Now the same model, in decode.

We still load the weights. All 14 GB. But now we use them on just one token.

Let's count again.

Math: 1 token × 2 × 7 billion = about 14 billion operations.
Bytes moved: about 14 GB.
Work per byte: roughly 1 operation per byte.

One. Not a thousand. Not even close to 300.

So in decode, the cooks finish their tiny job almost at once. Then they wait. They wait for the next pages to arrive.

Let's see how bad it is, using real time:

Time to do the math: 14 billion ÷ 1,000 trillion = about 0.014 milliseconds.
Time to move the weights: 14 GB ÷ 3.3 TB/s = about 4 milliseconds.

The moving takes roughly 300 times longer than the math. The math units are idle more than 99% of the time.

That is why decode is memory bound. The speed of writing is set by how fast the GPU can read its own memory. Not by how fast it can calculate.

In math terms, decode is a matrix times vector multiplication. Each loaded number is used once, then thrown away. The GPU cannot reuse anything.

There is a neat side effect. You can estimate the top speed of decode with one division. Memory bandwidth ÷ model size. For our example: 3.3 TB/s ÷ 14 GB is about 240 tokens per second. That is the ceiling for one user on one GPU. No matter how fast the cooks are.

The KV cache adds to the load too. Each new token must also read the notebook of all earlier tokens. The longer the chat, the bigger the notebook, the more bytes to carry. That makes decode even more memory hungry.

What this means in real life

Now the pause at the start makes sense.

Long prompt? Prefill takes longer. You wait longer for the first word.

Long answer? Decode takes longer. The text keeps streaming for a while.

Many users at once? Here is where it gets interesting.

Remember, decode wastes the GPU. One token uses a tiny fraction of its power. But the weights are loaded anyway.

So what if we serve many users in the same step?

Load the 14 GB once. Then use it for 32 users' tokens together. The carrying cost is the same. But now we get 32 tokens out of it.

This is called batching. It is the single biggest trick for decode. The work per byte goes up. The cooks finally have something to do. Throughput jumps, while the time for each user barely changes.

The story, in one minute

You send a prompt.

The model first reads it. All at once. Every token in parallel. The kitchen is full, and every cook is busy. This is prefill. It is compute bound. It decides how long you wait for the first word.

The model also writes a notebook as it reads. This is the KV cache.

Then the model writes. One token at a time. Each token needs the whole recipe book carried in from the pantry, just to do a tiny bit of cooking. The cooks mostly wait. This is decode. It is memory bound. It decides how fast the words stream.

Prefill asks: "How fast can we calculate?"

Decode asks: "How fast can we read memory?"

And that is why the same model, on the same GPU, can feel like two different machines.

Next time you see that small pause before the words flow, you will know what is happening behind it. The model is reading. And then, it starts to write.

Rough numbers in this post use a 7B model in 16-bit and an H100-class GPU. Real systems differ, but the shape of the story stays the same.