#1. The complete title of one (or more) paper(s) published in the open literature describing the work that the author claims describes a human-competitive result: a) Evolving the Topology of Large Scale Deep Neural Networks b) DENSER: Deep Evolutionary Network Structured Representation #2. The name, complete physical mailing address, e-mail address, and phone number of EACH author of EACH paper(s): Filipe Assunção Department of Informatics Engineering Faculty of Sciences and Technology, University of Coimbra Pólo II - Pinhal de Marrocos 3030-290, Coimbra, Portugal fga@dei.uc.pt +351 239790016 Nuno Lourenço Department of Informatics Engineering Faculty of Sciences and Technology, University of Coimbra Pólo II - Pinhal de Marrocos 3030-290, Coimbra, Portugal naml@dei.uc.pt +351 239790016 Penousal Machado Department of Informatics Engineering Faculty of Sciences and Technology, University of Coimbra Pólo II - Pinhal de Marrocos 3030-290, Coimbra, Portugal machado@dei.uc.pt +351 239790052 Bernardete Ribeiro Department of Informatics Engineering Faculty of Sciences and Technology, University of Coimbra Pólo II - Pinhal de Marrocos 3030-290, Coimbra, Portugal bribeiro@dei.uc.pt +351 239790087 #3. The name of the corresponding author (i.e., the author to whom notices will be sent concerning the competition): Filipe Assunção #4. The abstract of the paper(s): a) In the recent years Deep Learning has attracted a lot of attention due to its success in difficult tasks such as image recognition and computer vision. Most of the success in these tasks is merit of Convolutional Neural Networks (CNNs), which allow the automatic construction of features. However, designing such networks is not an easy task, which requires expertise and insight. In this paper we introduce DENSER, a novel representation for the evolution of deep neural networks. In concrete we adapt ideas from Genetic Algorithms (GAs) and Grammatical Evolution (GE) to enable the evolution of sequences of layers and their parameters. We test our approach in the well-known image classification CIFAR-10 dataset. The results show that our method: (i) outperforms previous evolutionary approaches to the generations of CNNs; (ii) is able to create CNNs that have state-of-the-art performance while using less prior knowledge (iii) evolves CNNs with novel topologies, unlikely to be designed by hand. For instance, the best performing CNN obtained during evolution has an unexpected structure using six consecutive dense layers. On the CIFAR-10 the best model reports an average error of 5.87% on test data. b) Deep Evolutionary Network Structured Representation (DENSER) is a novel approach to automatically design Artificial Neural Networks (ANNs) using Evolutionary Computation. The algorithm not only searches for the best network topology (e.g., number of layers, type of layers), but also tunes hyper-parameters, such as, learning parameters or data augmentation parameters. The automatic design is achieved using a representation with two distinct levels, where the outer level encodes the general structure of the network, i.e., the sequence of layers, and the inner level encodes the parameters associated with each layer. The allowed layers and range of the hyper-parameters values are defined by means of a human-readable Context-Free Grammar. DENSER was used to evolve ANNs for CIFAR-10, obtaining an average test accuracy of 94.13%. The networks evolved for the CIFA--10 are tested on the MNIST, Fashion-MNIST, and CIFAR-100; the results are highly competitive, and on the CIFAR-100 we report a test accuracy of 78.75%. To the best of our knowledge, our CIFAR-100 results are the highest performing models generated by methods that aim at the automatic design of Convolutional Neural Networks (CNNs), and are amongst the best for manually designed and fine-tuned CNNs. #5. A list containing one or more of the eight letters (A, B, C, D, E, F, G, or H) that correspond to the criteria (see above) that the author claims that the work satisfies: (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. (E) The result is equal to or better than the most recent human-created solution to a long-standing problem for which there has been a succession of increasingly better human-created solutions. (F) The result is equal to or better than a result that was considered an achievement in its field at the time it was first discovered. (G) The result solves a problem of indisputable difficulty in its field. #6. A statement stating why the result satisfies the criteria that the contestant claims (see examples of statements of human-competitiveness as a guide to aid in constructing this part of the submission): (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. We tested DENSER on the evolution of Convolutional Neural Networks (CNNs) for the classification of the CIFAR-10. The highest performing CNN found during evolution reports an average (of 5 independent trains) test classification accuracy of 93.29%, which is superior to the performances reported by several other NeuroEvolution approaches [1,2]. Further this result is obtained without introducing prior-knowledge into the search space. For example, while Suganuma et al. [2] define building blocks (made of convolution and pooling layers) that are to be stacked by evolution, the experiments conducted with DENSER only define the basic primitives (i.e., each layer, and the parameters that need to be tuned along with their allowed ranges); the blocks have to found by evolution. In addition, the best performing networks discovered during evolution for the CIFAR-10 are applied to the MNIST, Fashion-MNIST, and CIFAR-100 datasets (without performing evolution again). The results on these datasets are of 99.70%, 94.70%, and 74.94%, respectively. To the best of our knowledge there are no peer-reviewed NeuroEvolution works that report performances on the MNIST and/or Fashion-MNIST; Snoek et al. [1] report an accuracy of 72.60% on the CIFAR-100. (E) The result is equal to or better than the most recent human-created solution to a long-standing problem for which there has been a succession of increasingly better human-created solutions. According to [3] who sampled 400 CIFAR-10 images, the human classification performance on CIFAR-10 dataset is of 94%. An ensemble formed by the two highest performing networks found by DENSER provides an accuracy of 94.13%. There are multiple standard hand-crafted CNNs that have successfully been applied to CIFAR-10: VGG [4] with an accuracy of 92.26% (1); ResNet [5] with an accuracy of 93.39% (1), DenseNet [6] with an accuracy of 95.90%. Despite the impressive performance of DenseNet it is important to note that this CNN has primitives that are not in the search space of the CNNs generated by DENSER. On MNIST we have found the best performance to be reported by Graham et al. [7]: 99.68%. On Fashion-MNIST the human performance is estimated to be 83.50% [8]; VGG reports an accuracy of 93.50%, ResNet an accuracy of 94.90%, and DenseNet an accuracy of 95.40%. On CIFAR-100 Graham et al. report an accuracy of 68.80% (with 1 test), and 73.61% (with 12 tests). The number of tests on Graham et al. work similarly to an ensemble, and are in fact the way in which our ensemble results are formed; DENSER with 5 tests reports an average test accuracy of 77.51%; to the best of our knowledge this is the best result reported by CNNs on the CIFAR-100. This is even more impressing considering that this is a direct proof that the CNNs evolved by DENSER generalise, and scale. (1) Results reported by Suganuma et al. [2]. (F) The result is equal to or better than a result that was considered an achievement in its field at the time it was first discovered. Deep Learning (DL) models, in particular CNNs, have proven to be extremely efficient in computer vision tasks, such as object recognition. This is noticeable by the results previously enumerated in (B) and (E). DENSER is not only able to generated CNNs, but it is highly competitive with the state-of-the-art, both in hand-crafted and automatically generated networks. In some of the experiments DENSER even reported results that are above the state-of-the-art, with CNNs that are robust, generalise, and scale. In addition to the references pointed out in (A) it is also to note that DENSER reports results superior to those of CoDeepNEAT [9] (not yet published in a peer-reviewed journal), which reports an accuracy of 92.7% on the CIFAR-10. (G) The result solves a problem of indisputable difficulty in its field. In DL often the difficulty is not the lack of models able to solve a task, but rather how to decide their structure and parameterisation. The search space is too large, and the decisions that have to be made are not independent of one another; e.g., the placement of a given layer may change the behaviour of the remaining ones; plus, it affects the training of the network. DENSER is able to deal with all the challenges concerning the automatisation of deep neural networks; the results show its effectiveness. Further, DENSER is highly generalisable due to its grammar-based formulation: to evolve networks of different types and/or with different layers, or to evolve networks to a different task one only needs to adapt the grammar that is input to the system. #7. A full citation of the paper (that is, author names; publication date; name of journal, conference, technical report, thesis, book, or book chapter; name of editors, if applicable, of the journal or edited book; publisher name; publisher city; page numbers, if applicable) a) @inproceedings{DBLP:conf/eurogp/Assuncao0MR18a, author = {Filipe Assun{\c{c}}{\~{a}}o and Nuno Louren{\c{c}}o and Penousal Machado and Bernardete Ribeiro}, title = {Evolving the Topology of Large Scale Deep Neural Networks}, booktitle = {European Conference on Genetic Programming (EuroGP)}, series = {Lecture Notes in Computer Science}, volume = {10781}, pages = {19--34}, publisher = {Springer}, year = {2018} } b) @article{DBLP:journals/corr/abs-1801-01563, author = {Filipe Assun{\c{c}}{\~{a}}o and Nuno Louren{\c{c}}o and Penousal Machado and Bernardete Ribeiro}, title = {{DENSER:} Deep Evolutionary Network Structured Representation}, journal = {CoRR}, volume = {abs/1801.01563}, year = {2018} } #7. A statement either that "any prize money, if any, is to be divided equally among the co-authors" OR a specific percentage breakdown as to how the prize money, if any, is to be divided among the co-authors: Any prize money, if any, is to be divided equally among the co-authors. #8. A statement stating why the authors expect that their entry would be the "best": DENSER is a general-purpose framework that aims at dealing with the automatisation of all aspects concerned with the tuning of deep networks: it can optimise the topology, learning, and data augmentation, aiding the selection of both the layers / algorithms, and their specific parameters. We have tested DENSER on the evolution of CNNs for the CIFAR-10; the results show that without any prior-knowledge DENSER can generate CNNs that are competitive with both human and automatically designed networks. The highest performing networks found during evolution for CIFAR-10 are applied to MNIST, Fashion-MNIST, and CIFAR-100; without further evolution the networks are able to hold impressive results, demonstrating that DENSER is able to find networks that are robust, generalise, and scale. To the best of our knowledge, the result on the CIFAR-100 establishes a new state-of-the-art on CNNs for that dataset. It is also important to mention that all these results are obtained with very limited computing power: the experiments were conducted on a server with only four 1080Ti GPUs. #9. An indication of the general type of genetic or evolutionary computation used, such as GA (genetic algorithms), GP (genetic programming), ES (evolution strategies), EP (evolutionary programming), LCS (learning classifier systems), GE (grammatical evolution), GEP (gene expression programming), DE (differential evolution), etc.: Genetic Algorithm (GA) and Dynamic Structured Grammatical Evolution (DSGE) #10. The date of publication of each paper. If the date of publication is not on or before the deadline for submission, but instead, the paper has been unconditionally accepted for publication and is “in press” by the deadline for this competition, the entry must include a copy of the documentation establishing that the paper meets the "in press" requirement: a) March 2018 b) June 2018 #References: [1] Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, Md., & Adams, R. (2015, June). Scalable bayesian optimization using deep neural networks. In International conference on machine learning (pp. 2171-2180). [2] Suganuma, M., Shirakawa, S., & Nagao, T. (2017, July). A genetic programming approach to designing convolutional neural network architectures. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 497-504). ACM. [3] Karpathy, A. (2011, April 27). Lessons learned from manually classifying CIFAR-10. Retrieved May 28, 2018, from http://karpathy.github.io/2011/04/27/manually-classifying-cifar10/ [4] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. [5] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). [6] Huang, G., Liu, Z., Weinberger, K. Q., & van der Maaten, L. (2017, July). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, No. 2, p. 3). [7] Graham, B. (2014). Fractional max-pooling. arXiv preprint arXiv:1412.6071. [8] Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747. [9] Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., & Hodjat, B. (2017). Evolving deep neural networks. arXiv preprint arXiv:1703.00548.