Technical explainer · July 2026
How Small Transformer Models Work
An illustrated introduction to the operations inside a transformer, from vectors and attention to training and inference, using language and wrist-sensor examples.
Download PDF · 41 pagesOpen up the model
The explainer follows the path from a neuron and multilayer perceptron to embeddings, attention, residual connections and a complete transformer block. Its examples connect the same mathematical operations to words and short windows of sensor data.
Connect the mathematics to the hardware
Later sections explain training, fine-tuning, memory use and the cost of generating outputs. The goal is to help a reader understand the practical choices involved in using a small model.
A foundation for the wearable research
Prepared as study material for the facilities-sensor work, this note provides background for the broader Ichne research programme. It explains possible architectures rather than reporting a completed production model.