Free-hand sketches are deceptively hard for a machine to represent well. A sketch isn't just a picture; it's a sequence of strokes, drawn in a particular order, often messy, and wildly variable in length a quick doodle of a cat might take a dozen strokes, while a more elaborate drawing of the same category might take five times as many. Any system that wants to classify sketches, search for similar ones, or generate new ones needs an embedding that captures the structure of that sequence, not just the shape it happens to leave on the canvas. That turns out to be a genuinely different problem from representing a photograph, and for years it wasn't clear which family of network architecture was actually suited to it.
“Sketchformer: Transformer-based Representation for Sketched Structure,” presented at CVPR 2020 by Leo Sampaio Ferraz Ribeiro and Moacir Ponti of the University of São Paulo together with Tu Bui and John Collomosse of the University of Surrey and Adobe Research, argues that the answer is the transformer architecture from language modeling, not the recurrent networks that had been the default choice for sketch sequences up to that point. The paper's central claim is a comparative one: a transformer-based embedding trained on the same data as prior recurrent models pulls further and further ahead as sketches get longer and more structurally complex which is exactly where the earlier approaches were theoretically weakest.
Why Rasters and RNNs Fall Short
Two families of prior work set up the problem Sketchformer is responding to. The first converts sketches to pixels and applies a convolutional network, the same toolkit used for photographs. That approach works, but it throws away information for free: a sketch captured as a sequence of pen strokes already encodes drawing order and stroke boundaries, and rasterizing it discards all of that, forcing the network to re-derive spatial structure from a flat image it never needed to lose in the first place.
The second family keeps the sketch in its original vector form a sequence of pen movements and processes it with a recurrent network, typically an LSTM. SketchRNN, a sequence-to-sequence variational autoencoder trained on Google's QuickDraw corpus, is the best-known example, and later work built triplet-loss search systems on top of similar LSTM embeddings. Keeping the stroke sequence intact is the right instinct, but LSTMs have a well-known limitation carried over from language modeling: their effective memory of earlier time steps decays as a sequence gets longer, which caps how much structural complexity they can actually encode. A short, simple sketch is no problem; a long one, with strokes drawn many steps apart that are nonetheless part of the same visual structure, is where recurrence starts to lose the thread. Transformers had already solved an analogous problem in language modeling by replacing recurrence with attention, letting a model connect any two positions in a sequence directly rather than through a chain of hidden states which is the idea Sketchformer imports into the sketch domain.
Building Sketchformer: A Transformer Autoencoder for Strokes
At its core, Sketchformer is an encoder-decoder transformer, built on the original architecture introduced for machine translation, but resized and adapted for stroke sequences rather than sentences. The authors scale the network down considerably from the original design four multi-head attention blocks instead of six, and a feed-forward dimension of 512 instead of 2048 reflecting the fact that a sketch vocabulary is a much smaller, lower-dimensional problem than natural language.
One architectural gap had to be closed before the transformer could serve as a sketch representation at all: a standard transformer encoder outputs a full sequence of vectors, one per input step, not a single compact embedding. For classification, search, and generation, what's actually needed is one fixed-size vector per sketch. Sketchformer closes that gap with a self-attention pooling layer placed on top of the encoder output, which learns a weight for every time step in the sequence and sums the weighted per-step vectors into a single embedding. That embedding is then handed to the decoder, which has to reconstruct the original stroke sequence purely from it. To make sure the embedding actually has to do that work rather than the decoder quietly inferring sequence length or position from masking cues the authors also strip out the padding mask that a standard transformer decoder relies on, forcing reconstruction to depend entirely on the compact bottleneck rather than any side information about how long the original sketch was.
Three Ways to Feed In a Stroke
A sketch has to be turned into numbers before a network can see it, and the paper's most distinctive experimental choice is to test three different ways of doing that, rather than committing to one upfront.
The first, continuous representation, keeps the format used by SketchRNN: each point is stored as its offset from the previous point, plus a categorical pen state marking whether the pen is drawing, lifting, or the sketch has ended. This is the most direct encoding, with no discretization at all.
The second, dictionary-learned tokenization, builds a vocabulary of 1,000 “motion words” by clustering a large sample of pen movements with k-means, then represents every stroke as a sequence of the nearest vocabulary entries, with special tokens marking the start and end of the sketch and the boundaries between strokes. Because it's built from relative motion, this representation is compact and naturally invariant to where on the canvas a sketch was drawn but quantization error can accumulate over a long sketch, gradually drifting the reconstructed strokes away from their intended positions the further into the sequence you go.
The third, spatial grid tokenization, takes the opposite approach: it divides the canvas itself into a 100-by-100 grid and represents each point by the absolute cell it falls into. This avoids the drift problem, since every point is anchored to its actual position rather than accumulated from a chain of relative offsets, but it comes at the cost of sensitivity to translation shift a sketch a few pixels and its token sequence changes and a much larger vocabulary, since it scales with the square of the grid resolution.
Training the Embedding
Sketchformer is trained with two losses applied together. A classification loss sits directly on the pooled embedding, pushing it to preserve enough semantic information to tell sketch categories apart. A reconstruction loss, applied through the decoder, ensures the embedding also preserves enough detail to regenerate the original strokes: for the continuous variant this is a distance loss on the predicted offsets plus a classification loss on the pen state, and for the tokenized variants it's a standard next-token classification loss over the learned vocabulary. Combining both losses in one embedding is what lets a single Sketchformer representation serve classification, generation, and retrieval simultaneously, rather than needing a separate embedding trained for each task.
Classification, Generation, and Search
The authors evaluate Sketchformer on QuickDraw50M, a corpus of over 50 million crowd-sourced sketches across 345 categories drawn casually and at speed, which makes it considerably messier and more diverse than earlier, smaller sketch datasets. They train on a 2.5-million-sketch subsample and test on the public 862,000-sketch partition, additionally splitting sketches by stroke-sequence length into short, medium, and long percentile groups so that the effect of complexity can be measured directly rather than only reported in aggregate.
For 345-way sketch classification, the dictionary-tokenized variant of Sketchformer comes out on top, reaching about 78% accuracy against roughly 73% for the LiveSketch LSTM baseline and 68% for SketchRNN itself a clear but not dramatic margin. A secondary experiment here is genuinely surprising: when the authors shuffled the order of strokes within each sketch and retrained from scratch, classification accuracy barely moved. That suggests a meaningful share of the model's discriminative power comes from the overall spatial layout of the strokes rather than the precise temporal order in which they were drawn, which is a useful caution against assuming a sequence model is necessarily exploiting sequence order.
The reconstruction and interpolation experiments are where the paper's core argument shows up most clearly, and they were evaluated with a human preference study on Mechanical Turk rather than an automatic metric, since fidelity of a redrawn sketch is inherently a matter of visual judgment. For short and medium-length sketches, human raters actually preferred reconstructions from the older LiveSketch LSTM embedding it won roughly 60% of preference votes at those lengths, against the dictionary-tokenized transformer's 24–28%. But the ranking flips sharply for long, complex sketches: there, the dictionary-tokenized transformer was preferred 51% of the time, more than the LSTM baseline's 28%. Interpolation between pairs of sketches, both within the same category and across categories, tells a related but slightly different story the dictionary-tokenized transformer actually wins at every length tested, and its margin over the LSTM baseline widens substantially as sketches get longer, reaching roughly a two-to-one preference advantage on the longest sketches. Both results point the same direction: the transformer's advantage isn't uniform, it's concentrated exactly where LSTMs were expected to struggle.
Cross-modal retrieval results are more mixed. For sketch-based image retrieval using a sketch to search a 10-million-image photo corpus the dictionary-tokenized transformer embedding delivers the best mean average precision of the methods compared, ahead of both the LSTM-based LiveSketch baseline and an earlier raster CNN retrieval model. But for matching sketches to other sketches across the vector and raster domains, the gains over the LSTM baseline are much narrower, on the order of one to three points, and in that particular experiment it's the continuous (untokenized) variant of Sketchformer that performs best rather than either tokenized version.
Conclusion
Sketchformer's contribution is really two findings stacked together. The first is architectural: attention-based encoding, adapted with a pooling layer to produce a proper fixed-size embedding, is a workable and in several respects superior alternative to recurrent networks for representing stroke sequences. The second is about tokenization, and it's easy to undervalue: how a sketch is discretized before it ever reaches the network continuous offsets, a learned motion vocabulary, or a spatial grid turns out to shift performance by as much as the choice of architecture itself, and different tokenization schemes win at different tasks rather than one being universally best.
The more durable lesson, though, is in where the transformer's advantage actually shows up. It isn't a flat improvement across the board; it's concentrated in exactly the regime long, structurally complex sketches where the limited effective memory of recurrent networks was always going to be the bottleneck. That's a reminder that architectural comparisons are often incomplete without asking not just “which model wins” but “wins under which conditions,” since the answer to the second question is usually where the actual insight lives.