Build a language model from scratch in pure Python — the exact path Andrej Karpathy uses to teach LLMs. Start by counting letter pairs, end by writing self-attention and generating text from your own tiny GPT. Only numpy, no PyTorch, no magic.
Large language models feel like magic. They are not. Underneath, an LLM is a stack of ideas you can build yourself in an afternoon — and in this project you do, one small, testable step at a time, following the same ladder Andrej Karpathy climbs in his legendary Neural Networks: Zero to Hero series.
You'll train a character-level language model on names — a model that, given the characters so far, predicts the next one. We start at the absolute beginning: a tokenizer that turns letters into integers, then a bigram model you build by literally counting how often each pair of letters occurs, sampling brand-new names from those counts, and measuring quality with a loss. Then comes the leap that powers everything modern: instead of counting, you learn. You build the neural toolkit (one-hot, softmax, cross-entropy), train a one-layer neural bigram by gradient descent — writing the backward pass by hand and checking it against finite differences — and watch it rediscover exactly what counting found. From there you widen the context with an MLP and character embeddings (Bengio 2003), and finally build the idea behind every modern LLM: a single causal self-attention head, where each token looks back at the ones before it, assembled into a tiny GPT block that generates text.
The whole thing is written in plain Python with numpy — the one tool that maps almost 1:1 onto Karpathy's PyTorch tensors — so nothing is hidden behind a framework. Every challenge is graded by running real unittest suites against your code, including hand-derived gradient checks and a causality test that proves your attention can't peek at the future. By the end you won't just use LLMs — you'll know, line by line, how one is built.