In modern LLM applications like RAG and Agents, the model is constantly fed new context. For example, in RAG, we retrieve relevant documents and stuff them into the prompt.

The issue is that this dynamically retrieved context doesn't always appear at the beginning of the input sequence. Traditional KV caching only reuses a "common prefix," so if the new information isn't at the very start, the cache hit rate plummets, and your GPU ends up recomputing the same things over and over.

CacheBlend changes the game by allowing for the reuse of pre-computed KV caches regardless of their position in the input sequence.

This makes it possible to achieve a 100% KV Cache hit rate in applications like RAG. The performance gains are significant:

• Faster Time-To-First-Token (TTFT): Get your initial response much quicker.
• More Throughput: Serve significantly more users with the same hardware.
• Almost lossless Output Quality: All of this is achieved with little degradation in the model's generation quality.

CacheBlend works by intelligently handling the two main challenges of reusing non-prefix caches:

• Positional Encoding Update: It efficiently updates positional encodings to ensure the model always knows the correct position of each token, even when we're stitching together cached and new data.
• Selective Attention Recalculation: Instead of recomputing everything, it strategically recalculates only the minimal cross-attention needed between the new and cached chunks to maintain perfect generation quality.

An interactive CacheBlend demo is available at: github.com/LMCache/LMCache-Exa…

cross-posted from: lemmy.ml/post/32561522

The violence last week in Kafr Malik, in the West Bank, comes amid a surge in assaults by Israeli settlers. It also set off a chain of violence in the area.

By Fatima AbdulKarim
July 1, 2025 Updated 1:56 p.m. ET

[A Palestinian friend who lived in my neighborhood in the US is there now, in his family's home. He said it was like 3 nights in a war zone.]

"The attackers threw another firebomb into the bedroom where Mr. Afeef’s newborn nephew was being lulled to sleep, scorching furniture and leaving blackened marks on the floor and walls, the family said. The damage was visible when Times reporters visited on Friday.
[...]
Soon after, Israeli forces arrived and opened fired at Palestinians instead of stopping the rioters, according to multiple witnesses.

The soldiers killed three people, according to the Palestinian health ministry. [...]Nine others were injured, some gravely..."

archive.ph/j6Mce

Peter Link

2025-07-01 22:52:06

In an Attack at Sunset, Israelis Set a Palestinian Village Ablaze

The violence last week in Kafr Malik, in the West Bank, comes amid a surge in assaults by Israeli settlers. It also set off a chain of violence in the area.

By Fatima AbdulKarim
July 1, 2025 Updated 1:56 p.m. ET

"The attackers threw another firebomb into the bedroom where Mr. Afeef’s newborn nephew was being lulled to sleep, scorching furniture and leaving blackened marks on the floor and walls, the family said. The damage was visible when Times reporters visited on Friday.
[...]
Soon after, Israeli forces arrived and opened fired at Palestinians instead of stopping the rioters, according to multiple witnesses.

The soldiers killed three people, according to the Palestinian health ministry. [...]Nine others were injured, some gravely..."

archive.ph/j6Mce

https://www.nytimes.com/2025/07/01/world/middleeast/israel-west-bank-palestinians-attack.html

Explanation (which might be wrong, since I’m writing this after banging my head against a wall. Please do correct me if I’m wrong):

In regular numbering systems (i.e., decimal), we exhaust all 10 digits (0–9) before we reach two-digit numbers. The first number to require 3 digits is 10². The first to use 4 is 10³, and so on.

In music intervals, there is no “0”. The interval c’–c’, for instance, is called a prime (1). This has the funny consequence that moving by a fifth and then by a fourth doesn’t land you on the ninth, but the octave (8). Moving by an octave and then another octave gets you to the 15th, not the 16th.

In Excel, shit hits the fan when you need to convert column names (A, B, C…) to numbers (0, 1, 2…). Since we use 26 characters as our ‘digits’, we’re in the hexavigesimal system. Knowing what I told you in the first paragraph, you’d expect the first double-digit column (AA) to be 26. And you’re right.

However, when do we need 3 digits? Which column is column AAA? A sane person would say it’s 26², so 676. Ha! No. Column number 676 is actually ZA. What gives? Well, we only ditch the zero for single digit numbers. All subsequent columns actually use 27 different characters, the ‘empty character’ being one of them. That’s where we get the ‘single digit’ – there actually is a second digit, only it’s empty.

So the column AAA actually has index 702, or 26×27. Which index does the column AAAA have? 26×27². The system of adding powers of the base works, only we changed bases midway through.

You can see the lopsidedness in the index lookup table (I’m not displaying all characters for brevity). Sane number systems have square tables. Excel’s is 26×27 (shown are 4×5).