Technical explainer · July 2026

How Small Transformer Models Work

An illustrated introduction to the operations inside a transformer, from vectors and attention to training and inference, using language and wrist-sensor examples.

Download PDF · 41 pages

Open up the model

The explainer follows the path from a neuron and multilayer perceptron to embeddings, attention, residual connections and a complete transformer block. Its examples connect the same mathematical operations to words and short windows of sensor data.

Connect the mathematics to the hardware

Later sections explain training, fine-tuning, memory use and the cost of generating outputs. The goal is to help a reader understand the practical choices involved in using a small model.

A foundation for the wearable research

Prepared as study material for the facilities-sensor work, this note provides background for the broader Ichne research programme. It explains possible architectures rather than reporting a completed production model.