LONG VERSION
Introducing Supersensing as a promising path for human-level vision
There have been a few posts on this paper already, but I didn’t really dive into it yet. I am genuinely excited about the philosophy behind the paper. Given how ambitious the goal is I am not surprised to learn that Yann LeCun and Fei-Fei Li were important contributors to it.
➤Goal
We want to solve AI vision because it is fundamental to intelligence. From locating ourselves to performing abstract mathematical reasoning, vision is omnipresent in human cognition. Mathematicians rely on spatial reasoning to solve math problems. Programmers manipulate mental concepts extracted directly from visual processing of the real world (see this thread).
➤What is Supersensing?
This paper introduces the concept of “Supersensing”. It’s not an actual architecture, but a general idea of what we want AI to be able to do. Supersensing is essentially vision++. It is the ability to not only achieve basic perception feats (describing an image…) but also meta-perception like the ability to understand space and time at a human level.
We want AI to see beyond just fixed images and track events over long video sequences (this is the temporal part). We also want it to be able to imagine what’s happening behind the camera or outside of the view field (the spatial part).
With supersensing, a model should be able to understand a scene globally, not just an isolated part of it. It’s a bit like how a human can watch a movie and be queried on the gist of it while remembering the intense moments with more detail.
➤Where are we at?
Here are the perception levels
1- Basic perception
Basic descriptive abilities (“there is a dog in a park”).
2- Event perception
Tracking basic actions or surface-level events (“the dog is running in the park”).
3- 3D perception
Robust 3D understanding (“the park is now seen from a different point of view”, “the dog’s frisbee drifted upwards and landed in a higher plane of the park”)
4- Predictive world modeling
The pinnacle of reality understanding. It implies having a model of the physical reality overall, not just random isolated scenes (“the park is located in a modern city, with a few other parks around at a walking distance. I am an agent living in a minuscule part of all that”)
Models today show limited ability to perform level 2 on very short videos.
➤Main idea(s)
Generally speaking, models today treat their incoming data as equally worthy of attention. When watching a video, they consider all parts of it equally and the longer the video goes, the more compression is applied across the board. There is no concept of “surprise” or “important information” for the most part.
Video models also often fixate on pixels and struggle to see more broadly.
These two problems (lack of attention to surprise and pixel fixation) contrast drastically with humans. We remember the general look of familiar people around us, how they broadly behave, maybe a few details that stood out in certain rooms but that’s it. Sleep filters out everything else during the night.
To solve these problems, the team designed an architecture, “Cambrian-S”, that predicts in a simplified space (instead of the space of pixels), and use surprise to update its memory. This is done at runtime, not just during training!
When the model makes an incorrect prediction i.e high level of surprise, it stores information around that surprising event. Both the event and the surrounding context that led to it (think a few seconds before and after the surprising point) is stored in an external memory system to be used as information later on if needed.
Information is only stored when it’s deemed important. This is in contrast with current models that try to remember everything at once and with equal importance.
➤The Architecture (in general)
The architecture used to implement these ideas and get closer to “Supersensing” is called “Cambrian”. Cambrian constantly compares its predictions at run time with what actually happened, and stores surprising events in an external memory system for re-use during evaluation phases (for example).
The model also only processes videos in a compressed space. So both its predictions and the information it remembers do not contain actual pixels but only a “mathematical summary”, which is more human-like
➤The Architecture (technical details)
This paper is just a concept paper, so the implementation is kept to the simplest form possible.
In short, Cambrian-S = multimodal LLM + new component.
The LLM is pretrained on high-quality videos but has as robust a world model as you would expect from an LLM => not very good!
Note: this is a simplification. In reality, the LLM has a special training for video understanding that makes it better than most LLMs, but it’s not important to understand this paper.
Once the LLM is trained, they connect it to the core component of this architecture. It’s a predictive module capable of guessing the next frame at an abstract level (i.e. a simplified space that doesn’t remember all the pixels). They call it “Latent Frame Predictor (LFP)”. It runs at test time and constantly compares its predictions with reality.
The margin of error is used as a surprise signal: the higher the error, the more surprising what just happened is (thus the more important).
Important events are memorized with much more detail than the rest of the video. Even more: they are used as cutting points to segment the model’s experience of the video in terms of events.
This is based on a well-known phenomenon in psychology called the “doorway effect”. When humans enter a room or change environnment, our brains like to do a renitialization of our immediate memory context. As if to tell us “whatever you are about to experience now is novel and may have very little to do with what you were doing or watching right before”
Cambrian-S aims to do the same thing but in a very rudimentary way.
Here is how it works
0- Cambrian is trained (the LLM part)
1- A video is fed to Cambrian. We are now at runtime/test-time
2- Cambrian’s frame predictor tries to keep track of important events guided by surprise. The parts of the video that stood out the most are stored in an external memory
3- After watching the video, Cambrian is asked questions about the gist of it. The system is then allowed to extract information from its memory (hopefully, the frames associated with the question)
➤Supersensing/Cambrian vs JEPA
While both avoid pixels to make their prediction in a compressed space, Cambrian uses surprise/incorrect prediction at test-time (to organize and update its memory) while JEPA uses it only to train itself.
➤World Models need (way) better benchmarks
The researchers show that current video models have extremely shallow video understanding. The benchmarks used to test them are so easy, that it’s possible to get high scores simply by fixating on one specific frame of the video or by taking advantage of information inadvertently provided by the questions.
To fix this, the team designed new benchmarks that push these models to the brink. They have to watch videos sometimes hours long (and without knowing what they’ll be asked about) then are asked about important events. The tasks may be basic tracking over long timeframes but also much more difficult ones like counting how many times a specific item appeared in the video.
Note: for a human, these tasks aren’t as difficult as you would think because they revolve around obvious and surprising events, things that the human brain excels at (if you were queried about counting how many times you saw a dragon flying over a football field, you probably wouldn’t struggle!).
Ironically, another team of researchers managed to prove that as crazy as it sounds, even the benchmarks introduced by this paper (with questions about 4h-long videos) CAN be hacked, which reall stresses how difficult the art of designing benchmarks is.
➤Critique
This paper was critiqued by another research team shortly after its publication, so let’s talk about it.
They attempted to show 2 things:
1- The newly introduced benchmarks DO NOT test supersensing and can be easily hacked
They took a standard LLM without the frame predictor and showed that simply by telling it the questions in advance (before watching the video), the LLM can cheat and find enough cues to know what part of the video to look at to answer (essentially, finding one specific frame to fixate on rather than considering the entire video).
2- The architecture along with its suprise mechanism, actually doesn’t help it better understand videos
One of the things the supersensing team mentioned is that AI should have a good enough spatial sense to be able to detect when the same scene is presented with 2 different views. So if the AI is asked to count an item, it shouldn’t count it twice just because the same room is shown twice from a different pov.
The critics showed that if you repeat some scenes enough, Cambrian’s performance collapses to zero!
➤Response
The supersenting team responded to both points, and in my opinion, convincingly well:
1- Providing the model with context BEFORE watching the video goes directly against the purpose of the benchmark.
It’s the researchers who used the benchmark incorrectly, not the model that found a loophole in the test to score better
2- This paper is an introduction to a concept and a rudimentary first attempt at implementing it.
Cambrian is a very simple architecture because it’s a proof of concept so of course its spatial sense isn’t developed enough to resist aggressively misleading inputs.
➤Quick point on AI research
Many believe that “research” implies that we have to reinvent the wheel altogether every time. I don’t think it’s a good view.
The Transformer architecture is a strength to be used. If you read this paper, you’ll realize that the entire architecture is still structured around a Transformer-based LLM. They simply added a few modules to test their ideas.
While breakthroughs emerge from ambitious ideas, they are often still implemented over previous methods. Progress is, in some sense, incremental.
Something also has to be said about looking for “research directions” instead of “architectures”. The best way to avoid making architectures that are just mathematical optimizations of previous methods is by seeing larger and probing for fundamental problems. Truly novel architectures are a byproduct of those research directions.
---
➤SOURCES
Paper: https://arxiv.org/pdf/2511.04670
Video: https://www.youtube.com/watch?v=denldZGVyzM
Critique: https://arxiv.org/pdf/2511.16655v1