Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search (Paper Explained)

Neural Architecture Search is usually prohibitively expensive in both time and resources to be useful. A search strategy has to keep evaluating new models, training them to convergence in an inner loop to find out if they are any good. This paper proposes to abstract the problem and extract the essential part of the architecture to be optimized into a smaller version and evaluates that version on specifically custom learned data points to predict its performance, which is much faster and cheaper than running the full model. OUTLINE: 0:00 - Intro & High-Level Overview 1:00 - Neural Architecture Search 4:30 - Predicting performance via architecture encoding 7:50 - Synthetic Petri Dish 12:50 - Motivating MNIST example 18:15 - Entire Algorithm 23:00 - Producing the synthetic data 26:00 - Combination with architecture search 27:30 - PTB RNN-Cell Experiment 29:20 - Comments & Conclusion Paper: https://ift.tt/2ZJWfFl Code: https://ift.tt/3gYZ4Zx Abstract: Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly higher accuracy, especially when insufficient ground truth data is available. Our hope is that this work can inspire a new research direction in studying the performance of extracted components of models in an alternative controlled setting. Authors: Aditya Rawal, Joel Lehman, Felipe Petroski Such, Jeff Clune, Kenneth O. Stanley Links: YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher BitChute: https://ift.tt/38iX6OV Minds: https://ift.tt/37igBpB