An LLM generates one token at a time, and without a cache it recomputes the whole sequence at every step. We generated the same text with a local Qwen2.5-0.5B with the cache on and off: identical tokens, 4.5x faster at 256 tokens. The price is memory, which we calculate for the same model.
An LLM writes one token at a time. To choose token 100, it needs information from tokens 1 to 99. Without a cache, the model reads all 99 again at every step and recomputes the same numbers it computed one step earlier. The KV cache keeps those numbers and reuses them. It is the same trick as memoizing Fibonacci in our dynamic programming post: compute each piece once, look it up afterwards.
Three classic dynamic programming problems drawn cell by cell in a small pygame app. Memoization cut Fibonacci(8) from 67 calls to 15, the LCS table picked a different answer when the strings were swapped, and the knapsack table found 44 where a value-per-weight greedy got 32.
Dynamic programming (DP) has one idea: when a problem breaks into smaller problems that repeat, solve each one once and reuse the answer. The repetition is easy to say and hard to picture, so this post draws it. The three lessons come from the same pygame app as the path search post, and every number below came out of an actual run:
We embedded 134k docstring paragraphs with a local model and compared exact nearest-neighbour search with an HNSW index. Exact search grew from 0.07 to 0.8 ms per query; HNSW stayed near 0.17 ms at 97% recall. And with only one candidate, HNSWâs greedy walk found the true nearest neighbour less than half the time.
Semantic search turns every text into a vector and returns the vectors closest to the query (embeddings explained). The simplest way to find them is to compare the query with every vector. Vector databases use an approximate index instead, most often HNSW (Hierarchical Navigable Small World). The question is when the approximation is worth it.
Four grid searches run in a small pygame app on fixed maps. DFS found a 25-step path where BFS found 13, BFS walked through a swamp that Dijkstra avoided, and A* reached the same optimal cost as Dijkstra after expanding 22 cells instead of 267.
All four algorithms âfind a pathâ. They do not promise the same path. On four fixed maps in a small pygame app, measured with the appâs own search code:
TabPFN-3.5âs reported benchmark lead is a reason to evaluate, not a universal replacement rule. A local LightGBM demo scores 1.0000 AUC on random rows and 0.4748 on unseen groups; no TabPFN model is run.
A perfect test score can disappear before you change models. In our constructed example, one LightGBM baseline scored 1.0000 ROC AUC with random-row splitting and 0.4748 when testing unseen groups. The change was the evaluation question, not a new foundation model.
Language models read tokens, not letters: â strawberryâ is a single token. We asked two small local models to count letters in three words, with and without spelling them out first. The answers were inconsistent; code was not.
A language model never sees the letters of âstrawberryâ. It sees tokens, and inside a sentence the whole word ` strawberry` (with its leading space) is one token. To count the râs, the model has to recall how that token is spelled, then count, all from patterns it learned. Sometimes that works. In our test it was not reliable, and spelling the word out first did not fix it. If you need a letter count, count with code.
A synthetic task-selection example flips a 32.5% time reduction into a 10% increase without changing any task. Use it to understand why METRâs follow-up is not a simple verdict on AI coding productivity.
You can get âAI made work fasterâ and âAI made work slowerâ from the same task list without changing a single completion time. Change which tasks enter the study, and you change the question the result answers.
We rounded a small modelâs weights to 8, 4, 3 and 2 bits and measured perplexity and correct answers. 8-bit changed nothing; 4-bit dropped the 0.5B model from 11 to 4 correct answers, while a 1.5B model kept all 12.
TL;DR â Quantization stores a modelâs weights with fewer bits, so the model takes less memory. In our test, 8-bit weights were as good as the original. At 4 bits, a 0.5B model answered only 4 of 12 handbook questions correctly (down from 11), while a 1.5B model still got all 12. Below 4 bits, the small model broke down completely.
A dropped SSH connection sends SIGHUP to your jobs. We simulated one: a plain background job died, while nohup, disown and tmux kept running. Here is when to use each, and why a clean exit behaves differently.
TL;DR â When an SSH connection drops, the shell gets a hang-up signal (SIGHUP) and passes it to its jobs, which kills a plain python train.py &. Use nohup (or disown) for a fire-and-forget job with output in a log file. Use tmux when you want to come back later and see the live terminal. All three survived our simulated disconnect.
On standard Python 3.15, four threads sped up NumPy about 3x but not a Python loop. On the free-threaded build the Python loop scaled about 3x, single-threaded code ran about 14% slower, and NumPy did not change.
In our ordinary Python 3.15 build, four worker threads reduced a NumPy workload from 25.1 ms to 8.3 ms. The GIL was still enabled. A pure Python workload did not get faster.
