1. Hyunho Mo and Leonardo Lucio Custode and Giovanni Iacca. 2021. Evolutionary Neural Architecture Search for Remaining Useful Life Prediction. To appear on Applied Soft Computing. ------------ 2. Hyunho Mo, Via Sommarive, 9 - 38123 Povo (TN) , hyunho.mo@unitn.it, Tel. +39 3278 241660 Leonardo Lucio Custode, Via Sommarive, 9 - 38123 Povo (TN), leonardo.custode@unitn.it, Tel. +39 388 1767536 Giovanni Iacca, Via Sommarive, 9 - 38123 Povo (TN), giovanni.iacca@unitn.it, Tel. +39 389 8088796 ------------ 3. Corresponding author: Giovanni Iacca, giovanni.iacca@unitn.it ------------ 4. With the advent of Industry 4.0, making accurate predictions of the remaining useful life (RUL) of industrial components has become a crucial aspect in predictive maintenance (PdM). To this aim, various Deep Neural Network (DNN) models have been proposed in the recent literature. However, while the architectures of these models have a large impact on their performance, they are usually determined empirically. To exclude the time-consuming process and the unnecessary computational cost of manually engineering these models, we present a Neural Architecture Search (NAS) technique based on an Evolutionary Algorithm (EA) applied to optimize the architecture of a DNN used to predict the RUL. The EA explores the combinatorial parameter space of a multi-head Convolutional Neural Network with Long Short Term Memory (CNN-LSTM) to search for the best architecture. In particular, our method requires minimum computational resources by making use of an early stopping policy and a history of the evaluated architectures. We dub the proposed method ENAS-PdM. To our knowledge, this is the first work where an EA-based NAS is used to optimize a CNN-LSTM architecture in the field of PdM. In our experiments, we use the well-established Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset from NASA. Compared to the current state-of-the-art, our method obtains better results in terms of two different metrics, RMSE and Score, when aggregating across all the C-MAPSS sub-datasets. Without aggregation, we achieve lower RMSE in 3 out of 4 sub-datasets. Our experimental results verify that the proposed method is a reliable tool for obtaining state-of-the-art RUL predictions and as such it can have a strong impact in several industrial applications, especially those with limited available computing power. ------------ 5. List of claimed criteria: A, B, D, E, F, G ------------ 6. Justification of the claims: A) Several patents introduce methods for RUL prediction in PdM applications using deep learning (DL) models, especially CNN and LSTM, e.g. US20190235484A1, US20190057307A1, EP3407267A1, KR102178787B1, CN109726524A. This means that our DL network for RUL prediction obtained by successfully applying the EA may be patentable. B) The result we obtained can be summarized as follows: - Applying the EA to optimize the architecture of a multi-head CNN-LSTM model improves the state-of-the-art results on a well-established benchmark in the area of PdM (C-MAPSS). - The best model discovered for one sub-dataset is able to generalize to other sub-datasets. - The above achievements can be obtained by merely exploring around 4% of the total parameter space. Considering industrial contexts, a RUL prediction model for PdM has three main requirements: 1) accurate prediction, 2) robustness to shifting data distribution, and 3) limitation of computational resources. Several recent works have shown that data-driven DNN models are advantageous in terms of prediction accuracy. However their performance is highly dependent on their architecture. In fact, most DNNs are handcrafted by human experts. To go beyond the state-of-the-art results in terms of prediction accuracy, we employ an EA-based NAS to automatically explore the combinatorial parameter space of a multi-head CNN-LSTM model, in order to obtain a better RUL prediction. As a result, we obtain state-of-the-art RUL predictions using the best DNN found by the EA. Moreover, we find that the best model discovered for one sub-dataset shows a good performance on other sub-datasets although each sub-dataset has different degradation patterns under different operating conditions. This indicates that our results can be generalized to industrial applications that require a very stable performance across components under monitoring. Lastly, our results are obtained under limited computational resources, so that the proposed methods can be affordable also in industrial contexts that do not normally have access to expensive computing infrastructures. In summary, our work not only achieves statistically equal or better results in terms of prediction accuracy compared to the current state-of-the-art, but it is also adequate to industrial applications characterized by limited computational resources. D) The DNN obtained by means of the EA are publishable as new scientific results independently of the fact that they have been mechanically obtained. In fact, they could be published e.g. in industrial & engineering journals, regardless of the fact that they were obtained by means of evolutionary search, to advance the state-of-the-art in the area of remaining useful life (RUL) prediction for predictive maintenance (PdM) purposes. E, F) Predictive Maintenance is one of the key enabling technologies for Industry 4.0 and as such RUL prediction has attracted a great research interest, and a special attention from industry stakeholders. Especially, over the last few years, RUL prediction using DNNs has become very popular, based on the fact that in general DNNs show good results in pattern recognition tasks. Early works in this field merely proposed to use an MLP or CNN to recognize a degradation pattern in multivariate time series for solving a regression problem. They determined the architecture of neural networks empirically, without taking into account the industrial application issues. After that, various DNN models have been presented in the recent literature to get better results, but again those networks have been usually determined empirically, even though the architectures of these models have a large impact on the result. Therefore, a long-standing problem in this field of study is that handcrafted models (i.e., human-created DNNs) rely too much on the expertise of the researcher/engineer and on their knowledge of the dataset addressed. To tackle this problem, we present a custom EA specifically designed to optimize a DNN architecture used to predict the RUL accurately. The DNN optimized via the EA creates better results in prediction accuracy compared to recent works in this field (see Table 14 and 15 in the paper). Furthermore, as pointed out in B), our method finds the best model by searching around 4% of the total parameter space of a complex DNN architecture, while in practice humans cannot find by simple trial-and-error the best model by evaluating only 4% of all possible models. G) As discussed in B), E) and F), and highlighted by the various patents mentioned in A), finding manually the optimal architecture that yields the best RUL predictions in PdM applications is a problem of indisputable difficulty. Moreover, considering possible limitations on the computational resources available in industrial contexts (that do not normally have access to expensive computing infrastructures) makes this problem even harder. In those contexts, reducing the computational cost and execution time of the model (and of the process searching for it) still guaranteeing a high level of prediction accuracy, can have a direct impact on profits. To create a competitive solution, we reduce the computational cost of the evolutionary search using two mechanisms: 1) we enable an early stop condition on evaluating the validation error of each individual, based on learning rate decay, and 2) we reduce the number of overall evaluations during the evolutionary search using a history of the evaluated DNNs, that allow the algorithm to quickly converge to promising solutions. In fact, our solution could be available on free online services, such as Google Colab, or on cheap AWS cloud instances. ------------ 7. Hyunho Mo and Leonardo Lucio Custode and Giovanni Iacca. 2021. Evolutionary Neural Architecture Search for Remaining Useful Life Prediction. To appear on Applied Soft Computing. ------------ 8. Any prize money, if any, is to be divided in the following percentages: - Hyunho Mo: 50% - Leonardo Lucio Custode: 30% - Giovanni Iacca: 20% ------------ 9. We are confident that our entry would be the "best" for the following reasons: - We successfully applied a custom EA specifically designed to optimize a DNN architecture for real-world problems in industrial applications. - We proved that using our EA for NAS yields evidently better results compared to the previous works based on DNNs manually designed by human experts. - We showed that the proposed method is a reliable tool for obtaining state-of-the-art RUL predictions, based on the experimental results. - We introduced two mechanisms to shorten the EA evaluation time and save the computational cost, to make our approach suitable for real-world scenarios. - We verified that the DNN discovered by applying the EA on one sub-dataset can provide good results on other sub-datasets, while a model developed by human experts rarely produces good results across different sub-datasets. - Overall, our solution is human-competitive and it relates to a difficult industrial application. ------------ 10. The evolutionary computation technique used in our work is: GA (genetic algorithms) ------------ 11. The paper has been unconditionally accepted for publication on Applied Soft Computing. See the attached decision letter from the Managing Editor on behalf of the Handling Editor, Layth Sliman.