Decoder-only language model with no high-level LLM libraries — BPE tokenizer, causal multi-head attention, scheduled training, then sampling with temperature, top-k and nucleus.
Using language models daily without being able to derive what happens between a token and a logit felt like a gap. Building a decoder-only transformer with nothing above raw tensor operations closed it.
Me, as an applied-AI engineer
Reason about context length, attention cost and sampling behaviour from first principles.
The RAG and assistant work
Know why a temperature or top-p change moves output the way it does.
Byte-pair encoding trained on the corpus — count adjacent pairs, merge the most frequent, repeat to a target vocabulary. Written directly, which makes vocabulary size a decision with visible consequences for sequence length rather than a config value.
Queries, keys and values projected per head, scaled dot-product scores, an upper-triangular mask so a position can never attend forward, softmax, then a weighted sum and output projection. The mask is what makes it a language model rather than an encoder.
Attention and a position-wise feed-forward network per block, each wrapped in residual connections and layer norm. Without those the stack stops training past a handful of layers — the fastest way to learn why they are there is to leave them out once.
Cross-entropy on next-token prediction, learning-rate warmup then decay, gradient clipping and checkpointing. Loss curves and held-out samples together — the loss can improve while the text does not.
Greedy decoding, then temperature, top-k and nucleus sampling implemented over the logits. Seeing the same model swing between repetitive and incoherent as top-p moves makes those parameters stop being magic numbers.