All projects
Oct 20, 202612 min read

AI models learning AI

Exploring the latent space of a 30M autoencoder trained on 50K machine learning paper abstracts.

Parameters
30M
Abstracts
50K
Latent size
256-d
Similarity
Cosine

We train a text autoencoder on machine learning abstracts from 50,000 papers accepted at machine learning conferences over the past 10 years. The following are results from peeking inside.

Setup

The model has 30M parameters. The encoder reads a sentence and compresses it into a single 256-dimensional vector, z. The decoder sees only z and must reproduce the original text token by token. The model is trained from scratch on a T4 GPU for till it achieves ~94 validation accuracy on reconstruction of text. About 0.62 of the validation text is accurately reconstructed while cosine similarity peaks at 0.87

reconstruction loss · cross-entropyABSTRACTxENCODERx → zz · 256-DlatentDECODERz → x̂ABSTRACT′x̂
The autoencoder pipeline: abstract, encoder, latent space z, decoder, reconstructed abstract, with reconstruction loss feeding back.

With a 50k training set, the vocabulary of the field dominates the vocabulary of English. An embedding trained this way is a representation of this corpus, not a general-purpose text representation. We conduct experiments to further investigate the usefulness of this model.

Linear Interpolation

Our first experiment is to interpolate between two machine learning concepts by decoding points along the line segment between their vectors z. We observe that decoded text transitions gradually rather than switching abruptly. The following demo illustrates concepts morphing on a linear line in the latent space. We observe that decoded midpoints introduce target words gradually while trying (often failing) to keep overall sentence meaning. Click on the bars to see the evolution from A to B, and switch between examples to probe different pairs of concepts.

Step 1 of 26
Only in A In both Only in B

Deep reinforcement learning agents often need millions of interactions with an environment before they learn a useful policy.

What the space does not capture

  • Quantities vanish first: exact numbers and dataset names are traded for the syntax the decoder reuses across abstracts. The space is unreliable wherever α = 0.01 versus α = 0.1 matters.
  • Negation barely registers: “improves” and “fails to improve” differ by stop-word distance, so they land as neighbours.
  • The model knows only these 50k abstracts. Novel terminology is decomposed into familiar subwords and lands on the nearest older concept.

Takeaways

Reconstruction alone yields a space where topic and contribution type are readable from geometry — a sufficient baseline for search and clustering over a narrow corpus. Its failure modes are legible: quantitative detail drops, negations fold, novel vocabulary anchors to older concepts. A 30M-parameter autoencoder makes probing for these failure modes cheap.

Next: Universal sorting