The Supersensing paper was critiqued by another research team shortly after its publication thus I have to talk about it.
➤Critique
The critics attempted to show 2 things:
1- The newly introduced benchmarks DO NOT test supersensing and can be easily
hacked
They took a standard LLM and found that simply by telling it the questions in advance (before watching the video), the LLM can cheat and find enough cues to know what part of the video to look at and get 100% on the test without any global understanding whatsoever.
2- The architecture along with its suprise mechanism, actually doesn’t help it better understand videos
For instance, during counting tests, the critics showed that if you repeat some scenes multiple times, Cambrian’s performance collapses to zero because it doesn't understand that it's the same scene from a different view (and thus counts the same object multiple times)!
➤Response
The supersensing team responded to both points, in my opinion, convincingly well:
1- It's not the model that found a loophole in the test. It's the researchers who used the benchmark incorrectly.
Providing the model with context BEFORE watching the video goes directly against the purpose of the benchmark.
2- This paper is an introduction to a concept and a rudimentary first attempt at
implementing it.
Cambrian is a very simple architecture because it’s a proof of concept so of course its spatial sense isn’t developed enough to resist aggressively misleading inputs.