What Is the 'Minimal Essence of Experience' for Machine Learning?

A New Lens on Concept Formation, Learning, and Unlearning in AI

What does it take for a mind — biological or artificial — to understand something?

I was fascinated by the concept of ‘machine unlearning’ presented in a recent 6.3950 lecture. Even after the model was told to forget a concept, it could regenerate the concept because representations are distributed across many layers and weights, and because the remaining data still contains patterns that indirectly encode the concept.

This implies the existence of a deeper structure behind neural networks, the same way that there is a ‘minimal core’ behind developmental psychology, theory of mind, language acquisition, and classical ideas like sufficient statistics. This begs the question:

What is the minimal essence of experience – a small number of high-leverage experiences – required to form and reform a concept?

This question sits at the heart of both human and machine cognition.

---

Across disciplines, a similar pattern appears again and again:

• In statistics, sufficient statistics represent the smallest data summary that fully determines a model.

• In developmental psychology, small sets of early experiences define stable concepts like object permanence, phoneme categories, attachment patterns, and trust.

• In machine learning, certain rare, early, or contrastive examples can disproportionately shape a network’s representation.

• In linguistics, Chomsky argued that children possess an innate structure that makes language learnable and only need a small set of experiences to activate this structure.

All of these perspectives hint at a simple idea:

Concepts may be anchored by small sets of core experiences,

while the majority of data is redundant.

---

Unlearning flips the usual direction of epistemology.

Instead of asking how a mind learns, we ask: given that a mind has already learned something, what can be removed without collapsing the concept? Once learned, is it even possible to fully unlearn certain concepts?

If we can understand the minimal experiences that anchor concepts, we might uncover a deeper theory that unifies how learning works across biological and artificial systems.

And perhaps, in the process, we’ll learn something about ourselves too.

Keep up with my project here: https://github.com/thunderingluck/regrow-unlearning