1. Title of the Publication
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
(published at the 37th ICML, on July 2020)
2. Author Information
Esteban Real
Google Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043
ereal@google.com
617-595-5888
Chen Liang
Google Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043
crazydonkey@google.com
512-921-5166
David R. So
Google Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043
davidso@google.com
[will provide phone number later]
Quoc V. Le
Google Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043
qvl@google.com
650-353-1691
3. Corresponding Author
Esteban Real
ereal@google.com
4. Paper Abstract
Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by back-propagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.
5. Competition Criterion
F. The result is equal to or better than a result that was considered an achievement in its field at the time it was first discovered.
6. Statement Why The Result Satisfies Criterion F
Our method evolves code to solve machine learning (ML) problems from scratch within a fairly unbiased search space (please see point 9 for details). In this context, it discovers the following results, each of which was considered an achievement in the ML field when it was first found:
a) stochastic gradient descent
b) linear regression
c) bilinear models (as noted in Jayakumar et al., "Multiplicative interactions and where to find
them", ICLR 2020)
d) weight-averaging equivalent (similar to Polyak and Juditsky, "Acceleration of stochastic approximation by averaging", SIAM journal on control and optimization 1992; and Collins, "Discriminative training methods for hidden markov models ...", ACL 2002).
e) regularization through noise addition to the input (noted in Goodfellow et al., "Deep learning", MIT Press, 2016)
f) gradient normalization (similar to Hazan et al., "Beyond convexity: stochastic quasi-convex optimization", NIPS 2015; and Levy, "The power of normalization: faster evasion of saddle points", arXiv, 2016).
7. Full Citation
Esteban Real, Chen Liang, David R. So, and Quoc V. Le. AutoML-Zero: Evolving Machine Learning Algorithms From Scratch. 37th International Conference on Machine Learning (ICML 2020).
8. Prize Breakdown Statement
Any prize money, if any, is to be divided equally among the co-authors.
9. Required Statement Indicating Why this Entry Could be the "Best"
While many recent papers have used evolutionary algorithms to reach competitive results on specific machine learning benchmarks (e.g. ImageNet classification), AutoML-Zero aims to rediscover machine learning itself. As such, we believe our work represents an ambitious step for evolutionary optimization and AutoML.
In particular, AutoML-Zero does not assume a search space tailored to a particular problem. Instead, the search is only minimally constrained to a supervised learning paradigm: we directly evolve code that receives one example at a time. The primitives with which evolution must construct the code are restricted to basic operations such as scalar and vector arithmetic (note that gradients are *not* provided). Moreover, the search process starts from empty code; a whole algorithm must be found. Within this simple yet fairly unconstrained environment, AutoML-Zero is able to discover linear regression, simple neural networks trained with backpropagation, and various tricks invented by machine learning experts throughout the last few decades. In particular, while evolving on examples from 100 different image classification tasks on realistic data, a single evolution experiment discovers gradient descent, bilinear models, ReLUs, regularization through noise, gradient normalization, weight averaging, and the like.
We achieved this by evolving at scales that are unprecedented for automatic machine learning: a single experiment evaluated a trillion algorithms, each of which must be trained on data. This was permitted by a concerted engineering effort of code optimization and large systems design, enabled by the natural scalability of evolutionary search. We hope that as the world's computational capabilities reach biological scales, the approach used in this paper might become more prevalent.
10. Evolutionary Computation Type
Linear Genetic Programming
11. Publication Date
The date on the paper in the official ICML website appears as "2020". A more specific publication date would be July 2020, as the conference took place between July 13th and July 18th.