
A review published by @FlyScholarsCS (F.A.Q.) - July 9, 2021
Paper
"The Values Encoded in Machine Learning Research"
Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, Michelle Bao
https://arxiv.org/abs/2106.15590v1
TL;DR
Technical objectives typically pursued in machine learning, such as performance and efficiency, are commonly believed to be neutral for society. This study powerfully raises awareness on the possibility that these objectives might in fact be aligned with the further concentration of power by large technology corporations. However, the argumentation is one-sided and fails to acknowledge the potential benefits for society of these same objectives.
Summary of the paper
This paper manually analyzes the abstract, introduction, discussion, conclusion, affilation and funding sources of the 100 most cited papers at NeurIPS and ICML conferences (for the years 2008/2009/2018/2019). The main findings are:
- Top-cited papers are increasingly affiliated with and funded by large tech corporations.
- Papers are majoritarily motivated by technical "values" (i.e. objectives) as opposed to societal values, and typically do not state potential negative impacts for society.
- Among these technical values, "performance", "generalization", "efficiency", and "building on past work" are the most cited values.
- These particular values, although they may seem neutral, might in fact support the centralization of power by large tech corporations.
Contributions & Strengths
Raising awareness on the non-neutrality of technical values in ML. The most powerful contribution of this paper is that is raises awareness on the possibility that even purely technical research objectives, typically considered neutral by the ML community (neither good or bad for society), may in fact support the concentration of power by large tech corporations and accentuate the harms [1,2,3,4] done by technology to the general public and to minorities.
A widely accessible paper. The paper is easy to read and accessible for a broad audience, an impressive feat since it is situated at the intersection of many disciplines with different traditions and jargons (machine learning, epistemology, linguistics, sociology, philospophy). In particular, the paper is written in simple words, rarely uses jargon from these disciplines, mostly defines concepts which are specific to particular disciplines, is mostly self-contained and its methodology is described precisely.
A well-justified methodology. The methodology used to analyze the papers is well justified and rigorous. The analysis of ML papers performed is purely manual and controlled for inter-individual reliability, and does not rely on automatic keyword or machine learning annotation which would suffer from well-known biases and blindspots.
Limitations & Weaknesses
The argumentation on the harms for society of widely pursued technical objectives is one-sided. The crux of the paper relies on a one-sided argumentation that the objectives referred to by most top-cited ML papers ("performance", "generalization", "efficiency", and "building on past work") in fact support the concentration of power by large tech corporations. Here we analyze some of these arguments in detail and show that they do not paint the full picture.
- Performance:
- "First, we find performance values are consistently and unquestioningly operationalized as correctness averaged across individual predictions, giving equal weight to each instance. However, choosing equal weights when averaging is a value-laden move which might deprioritize those underrepresentated in the data or the world, as well as societal and evaluee needs and preferences."
=> We note here that a perfect performance would also work for underrepresentated categories in the data. For utilizations of AI such as self-driving car [5], the vision system should be nearly perfectly reliable before being deployed, and performance is thus a crucial objective to consider. For other applications of AI such as biology and medical research [6], performance is also a crucial factor and a failure on edge-cases might not necessarily be detrimental to minorities. - "One common assumption is the availability of very large datasets. However, presupposing the availability of large datasets is power centralizing because it encodes favoritism to those with resources to obtain and process them"
=> It is a fact that most ML models work better when trained on more data, and thus it is expected that the science and engineering efforts concentrate on building and training models on large datastes. Note that the needs for larger datasets is not necessarily incompatible with the democratization of AI. For instance, Facebook AI and others release pre-trained models with MIT License which can be used for research and directly integrated by smaller companies into their product. - "Further overlooked assumptions include that the real world is binary or discrete, and that datasets come with a predefined ground-truth label for each example, presuming that a true label always exists "out there""
=> A lot of the effort in AI consists in using unsupervised learning approaches which do not rely on binary labels to learn representations. We note that the paper does not comment on this positive aspect of unsupervised learning for society.
- Generalization: "Typically, a paper shows that a model generalizes by showing that it performs well on multiple tasks or datasets. [...] Critical scholars have advocated for valuing context, which stands at the opposite side of striving for generalization. Others have argued that this kind of totalizing lens (in which model developers have unlimited power to determine how the world is represented) leads to representational harms, due to applying a single representational framework to everything"
=> This analysis fails to distinguish in-domain generalization (generalization to unseen samples from the same datasets as the ones used for training) and out-of-distribution generalization (unseen conditions during training). Out-of-distribution generalization is related to the discovery and disentanglement of the underlying causes shaping the data [7], a condition for creating fairer models. - Efficiency: "Abstractly, saying that a model is efficient typically means saying that the model uses less of some resource, such as time, memory, energy, or number of labeled examples. In practice however, efficiency is commonly referenced to imply the ability to scale up: a more efficient inference method allows you to do inference in much larger models or on larger datasets, using the same amount of resources."
=> Efficiency is the perfect exemple of a double-edged sword, which can indeed profit the powerful but also smaller organizations and minorities. For example, data efficiency can be extremely critical for medical applications where there is not a lot of data available [8]. Moreover, even when efficiency is obtained at scale and used by large tech corporations, it could be argued to be a net positive for society as it decreases the ecological footprint of widely used services. For example, it would be hard to argue today that we regret the energy efficiency of e-mails compared to traditional mail.
The composition of the team is one-sided. "The composition of our team confers additional validity to our work. We are a diverse, multi-racial, multi-gender team working closely, including undergraduate, graduate, and post-graduate researchers from machine learning, NLP, robotics, cognitive science, and philosophy. "
=> The team would have benefited greatly from a corporate researcher, since the crux of the paper relies on the demonstration that the technical objectives widely pursued by ML researchers mainly support the concentration of power by large corporations. We also note that most of the authors come from an elite university (top-50 QS), and yet the paper is speaking against concentration of power. This paper thus demonstrates that elite universities (and maybe even large corporations) can sometimes do genuine efforts to counteract harms of technology. It would be fair to acknowledge these efforts as well.
Clarity
Overall the paper is very clear and successfully bridges the culture gap between many different disciplines. Below are a few suggestions to make the paper even clearer:
- A definition of the words "operationalized", "intercoder consensus", "kappa coefficient" would benefit ML readers.
- "We have encouraged ourselves, and now encourage the reader, to remember that values once held to be intrinsic, obvious, or definitional have been in many cases transformed over time."
=> Some references and more explanations would be useful here. - For example, Abdalla and Abdalla [2] argue that big tech sway and influence academic and public discourse using strategies which closely resemble strategies used by Big Tabacco.
=> Some details on the similarity between Big Tech and Big Tobacco would be useful here, in the spirit of having a more self-contained paper.
Questions & Suggestions for future work
This work triggers many important questions which of course could not be adressed in a single study. We ask a few here:
- Would it be possible to identify technical objectives which are mostly beneficial to society and work against the concentration of power? A few suggestions are: causality, out-of-distribution generalization, robustness, data efficiency, energy efficiency, interpretability for users.
- Should all science be society-facing, or is it ethically OK to have most science focused on purely technical aspects and be destined only to other researchers? How does the fraction of purely technical objectives in ML compare to other fields such as Biology?
- What regulations should be put in place such that ML research would be less driven by economical incentives and more by societal considerations? What should be the proportion of ML research focusing on the current technical objectives such as perfomance, generalization, and efficiency?
References
[1] "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Shmargaret Shmitchell
https://dl.acm.org/doi/10.1145/3442188.3445922
NB: A paper showing the societal costs and dangers of large language models.
[2] "We read the paper that forced Timnit Gebru out of Google. Here’s what it says."
https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/
NB: Reporting on the termination of co-chief AI ethicist of Google.
[3] "He got Facebook hooked on AI. Now he can't fix its misinformation addiction."
https://www.technologyreview.com/2021/03/11/1020600/facebook-responsible-ai-misinformation/
NB: Reporting on the potential harms of Facebook recommendation algorithms.
[4] "YouTube Regrets"
https://foundation.mozilla.org/en/campaigns/regrets-reporter/findings/
NB: Mozilla report on the lived experiences of Youtube's recommendation engine users.
https://techcrunch.com/2021/07/07/youtubes-recommender-ai-still-a-horrorshow-finds-major-crowdsourced-study/
NB: TechCrunch coveragre
[5] "Andrej Karpathy Tesla Autonomous Driving Talk CVPR 2021"
https://www.youtube.com/watch?v=a510m7s_SVI
NB: Current challenges of developping AI at scale for autonomous driving.
[6] "DeepMind's AlphaFold 2 Explained! AI Breakthrough in Protein Folding! What we know (& what we don't)"
https://www.youtube.com/watch?v=B9PL__gVxLI
NB: Exemple of a breakthrough in AI for Biology by Google DeepMind which might benefit society as a whole.
[7] "On the Fairness of Disentangled Representations"
https://arxiv.org/abs/1905.13662
Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, Olivier Bachem
[8] "Rotation equivariant cnns for digital pathology"
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, Max Welling
https://link.springer.com/chapter/10.1007/978-3-030-00934-2_24
Related Material
"Normative roboticists: the visions and values of technical robotics papers"
Martim Brandao
https://www.martimbrandao.com/papers/Brandao2021-roman-visions.pdf
NB: A similar study on the values present in the Robotics litterature.
"Introducing the NeurIPS 2021 Paper Checklist"
Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan (NeurIPS 2021 Program Chairs)
https://neuripsconf.medium.com/introducing-the-neurips-2021-paper-checklist-3220d6df500b
NB: This checklist is a strategy to encourage authors to think about the ethical and societal ramifications of their research.
"NSF 101: Five tips for your Broader Impacts statement"
https://beta.nsf.gov/science-matters/nsf-101-five-tips-your-broader-impacts-statement
NB: It is interesting to note that the National Science Foundation (NSF) lists "Increased partnerships between academia, industry and others", "Improved national security" and "Increased economic competitiveness of the U.S." as acceptable objectives for a broader impact statement, in contrast with the defintion of socially beneficial values presented in the reviewed article.
Reviewer
~Anonymous
Did you enjoy this open review? Contribute your own reviews! https://rentry.co/faq_fs