Banner

A review published by @FlyScholarsCS (F.A.Q.) - July 2, 2021

Paper

"MLP-Mixer: An all-MLP Architecture for Vision" (v4)
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy
https://arxiv.org/abs/2105.01601v4

TL;DR of the review

MLP-Mixer is a very simple deep learning architecture for vision, with results on par with the best ConvNets and Vision Transformers when trained on sufficient data. However, some claims in the paper would benefit from clarifications and refinements, namely that (1) the proposed spatial mixing layer has a linear computational complexity in the number of patches and (2) that MLP-Mixer does not rely on convolutions. In addition, I would have appreciated a demonstration of the necessity of the global spatial mixing layer proposed, as opposed to a more traditional convolutional kernel. Finally I propose that models trained on such large datasets might essentially act as a "big look-up table", and propose an experiment to test this hypothesis.

Summary of the paper

This paper proposes a very simple deep learning architecture for vision, MLP-Mixer. This architecture essentially replaces the self-attention layers of the Vision Transformer (ViT) [1] with a one-hidden-layer MLP acting on all spatial locations (which we refer to as "the spatial mixing layer"), and acting separately on every channel with weight-tied kernels (see video [2] for a pedagogical explanation of the architecture). Trained on large enough datasets, such as the Google proprietary dataset JFT-300M which contains 300 million images and 1 billion labels, or with appropriate augmentations and regularizations (RandAugment, mixup, dropout, and stochastic depth) on smaller datasets such as ImageNet-1k and Imagenet-21K, MLP-Mixer is competitive or only slightly worse than Vision Transformers and ConvNets when looking at different trade-offs between prediction accuracy, training time (as measured by TPUv3 core-days), and throughput at inference time (as measured by img/sec/core).

Contributions & Strengths

A simplified architecture for vision on large datasets. MLP-Mixer is undoubedtly a simplified architecture compared to attention-based models such as ViT, and it is a remarkable finding that the attention modules of ViT seem to only play a minor role in performance as the size of the dataset increases. It is too early to say whether simpler architectures inspired by MLP-Mixer will eventually replace Transformers and ConvNets for large datasets (the state of the art is currently a transformer with 90.45% top-1 ImageNet accuracy followed by a ConvNet architecture at 90.2% [3]). However, given that ConvNets have been highly optimized over the years, the fact that a much simpler architecture works so well on large datasets without much exploration is very encouraging.

On the benefits of simpler architectures. Simpler architectures are promising for several reasons. First, they might come with fewer inductive biases (i.e. biases restricting the space of funtions that can be learned) and therefore scale better to larger datasets. Indeed, the paper shows that when trained on 3 billion (!) images, MLP-mixer starts showing a small advantage compared to ViT (fig. 2). Second, simpler architectures might be widely applicable to many datasets and provide an all-purpose architecture for deep learning. Indeed, a similar architecture has concurrently been proposed for language processing, which replaces attention layers by a Fourier Transform with partial success [4]. Another concurrent work proposing an architecture very similar to MLP-Mixer but limiting the training to ImageNet-1K [5], shows that this type of architecture can be competitive to the best Transformers and ConvNets in terms of the different trade-offs between accuracy, training time and throughput, athough they are still worse than the best ConvNets in absolute performance (79.7% top-1 ImageNet accuracy for the best MLP-based model vs 86.5% for the best ConvNet in this paper).

Limitations & Weaknesses

A subtler model than it may seem. On the smaller datasets studied (ImageNet-1K and ImageNet-22K), the simplicity of the architecture of MLP-Mixer must be put in perspective with the fact that many hyper-parameters and engineering went into the regularisations (dropout, stochastic depth), augmentations (RandAugment, mixup), and optimizer used (described in detail in Appendix B). These choices and hyper-parameters should be kept in mind when comparing the complexity of MLP-Mixer with other models for smaller datasets.

Clarity Concerns

The paper would greatly benefit from clarifications and refinements on claims about the computational complexity of the proposed spatial mixing layer replacing attention, and on the presence of convolutional inductive biases in MLP-Mixer.

  • Linear complexity of the spatial mixing layers. The paper claims that the the "computational complexity of the network is linear in the number of input patches, unlike ViT whose complexity is quadratic". This would be correct if the output dimension of the spatial mixing layer was truly independant of the input dimension (i.e. number of patches). Yet, when changing the resolution of the input image for fine-tuning, the output dimension of the spatial mixing layer is scaled to match the larger number of input patches. Thus, the complexity of this layer seems effectively quadratic in the number of patches. It is therefore unclear that the complexity of MLP-Mixer is truly better than the one of ViT, and this part would benefit from more explanations.
  • Convolutional inductive bias of MLP-Mixers. The paper claims that "while convolutions and attention are both sufficient for good performance, neither of them are necessary" and that "MLP-Mixer architecture [...] does not use convolutions". Yet, MLP-Mixer undoubtedly presents strong convolutional inductive biases. It is true that the network is not sensitive to the spatial relationship between pixels within a patch and between concommitent patches, as shown in an experiment where pixels within patches as well as patch locations are randomly permuted (Fig. 4). However, the network incurs a severe drop in performance when all pixels of the images are scrambled (45% performance drop, Fig. 4), whereas the performance of a strict MLP would not be affected. This discrepancy might seem hard to reconcile at first sight with the fact that the spatial mixing layers look at the entire image. A first evident departure from a pure MLP model is the patchify stem which treats pixels within a patch together. A second important departure from a pure MLP model could be due to the residual connections in the spatial mixing layers. By seeding each spatial token with a specific location, the residual connections could play the role of a powerful spatial inductive bias. This hypothesis might seem at odds with the visualization of the filters of the spatial mixing layers presented, which reveals global filters which do not seem convolutional (Fig. 5). However, note that this vizualization only reveals the filters of the first layer of the spatial mixing layers, as opposed to the effect of the full one-hidden layer MLP operation. Moreover, in the concurrent work [5] where they use a simple linear layer for the spatial mixing operation, they observe the emergence of simple convolutional and mostly localized filters in most layers.

Questions & Suggestions for future work

Importance of the global spatial mixing layers. An important difference between MLP-Mixer and more traditional convolutional architectures lies in the global spatial mixing layers used in MLP-MIxer. However, it is not totally clear that these fully-connected layers are even necessary to obtain good performances (see Clarity Concerns). It would be important to demonstrate the benefit of the these global spatial layers compared to more traditional convolutional kernels. This could be done by replacing in the architecture the global spatial layers with simple convolutional layers (at least in the early layers) and oberve whether the modified architecture would perform as well as MLP-Mixer on large datasets such as JFT-300M. Note that an advantage of this more traditional convolutional architecture would be that the complexity of these spatial mixing layers would truly be linear in the number of patches.

Could MLP-Mixer be replaced by a big look-up table (which would not require any training)? The drastic architecture simplification proposed by MLP-Mixer is reminiscent of The Bitter Lesson of Rich Sutton [6]: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin". However, the number of images in the training set of JFT-300M and JFT-3B becomes so large that models trained on these datasets might essentially act as big look-up tables. Therefore, it could be interesting to revisit the idea of a simple look-up table approach on these large datasets. For example, an experiment could consist in pushing the entire training data through a simple pre-trained ResNet-50 to obtain a feature representation of all images. A nearest neighbor approach in feature space could then be used to classify new images. One caveat of this approach is that since the classes of JFT-300M are not the same as the classes of ImageNet, a solution should be found to recast the JFT labels into ImageNet labels.

References

[1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby
https://arxiv.org/abs/2010.11929

[2] MLP-Mixer: An all-MLP Architecture for Vision (Machine Learning Research Paper Explained)
Yannic Kilcher
https://www.youtube.com/watch?v=7K4Z8RqjWIk

[3] Leaderboard of state of the art on ImageNet
https://paperswithcode.com/sota/image-classification-on-imagenet

[4] FNet: Mixing Tokens with Fourier Transforms
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon
https://arxiv.org/abs/2105.03824

[5] ResMLP: Feedforward networks for image classification with data-efficient training
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, Hervé Jégou
https://arxiv.org/abs/2105.03404

[6] The Bitter Lesson
Rich Sutton
http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Pay Attention to MLPs
Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le
https://arxiv.org/abs/2105.08050
TL;DR: Concurrent work showing that attention layers in transfomers can be replaced by a simpler gating mechanism, for both language and vision applications.

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
Luke Melas-Kyriazi
https://arxiv.org/abs/2105.02723
TL;DR: Another concurrent work with similar findings than MLP-Mixer, with a good explanation of the current quadratic complexity of spatial mixing layers.

Early Convolutions Help Transformers See Better
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, Ross Girshick
https://arxiv.org/abs/2106.14881
TL;DR: Replacing the patchify stem with a traditional convolutional stem in ViT improves training stability and performance (for ImageNet-1k to ImageNet-21k).

Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
Wieland Brendel, Matthias Bethge
https://arxiv.org/abs/1904.00760
TL;DR: Similarly to Mixer-MLP, they show that networks without spatial inductive biases can perform well on ImageNet

Revisiting Spatial Invariance with Low-Rank Local Connectivity
Gamaleldin F. Elsayed, Prajit Ramachandran, Jonathon Shlens, Simon Kornblith
https://arxiv.org/abs/2002.02959
TL;DR: An alternative way to relax the spatial invariance of a convolutional network layer, where the filter bank applied at each position is constructed as a linear combination of basis set of filter banks with spatially varying combining weights.

Reviewer

~anonymous

Did you enjoy this open review? Write your own reviews! https://rentry.co/faq_fs

Edit

Pub: 01 Jul 2021 20:03 UTC

Edit: 20 Jul 2021 11:14 UTC

Views: 954