1. the complete title of one (or more) paper(s) published in the open literature describing the work that the author claims describes a human-competitive result; Empowering the Human as the Fitness Function in Search-Based Model-Driven Engineering -------------------------------------------------------------------------------------- 2. the name, complete physical mailing address, e-mail address, and phone number of EACH author of EACH paper(s); Francisca Pérez SVIT Research Group Universidad San Jorge Autovía A-23 Zaragoza-Huesca Km. 299 50830 Zaragoza, Spain email: mfperez@usj.es tel: +34 671 006 334 Jaime Font SVIT Research Group Universidad San Jorge Autovía A-23 Zaragoza-Huesca Km. 299 50830 Zaragoza, Spain email: jfont@usj.es tel: +34 976 060 10LL Lorena Arcega SVIT Research Group Universidad San Jorge Autovía A-23 Zaragoza-Huesca Km. 299 50830 Zaragoza, Spain email: larcega@usj.es tel: +34 976 060 100 Carlos Cetina SVIT Research Group Universidad San Jorge Autovía A-23 Zaragoza-Huesca Km. 299 50830 Zaragoza, Spain email: ccetina@usj.es tel: +34 976 060 100 -------------------------------------------------------------------------------------- 3. the name of the corresponding author (i.e., the author to whom notices will be sent concerning the competition); Jaime Font -------------------------------------------------------------------------------------- 4. the abstract of the paper(s); Abstract—In Search-Based Software Engineering, more than 100 works have involved the human in the search process to obtain better results. However, the case where the human completely replaces the fitness function remains neglected. There is a good reason for that; no matter how intelligent the human is, humans cannot assess millions of candidate solutions as heuristics do. In this work, we study the influence of using the Human as the Fitness Function (HaFF) on the quality of the results. To do that, we focus on Search-Based Model-Driven Engineering (SBMDE) because inspecting models should require less human effort than inspecting code thanks to the abstraction of models. Therefore, we analyze the impact of HaFF in a real-world industrial case study of feature location in models. Furthermore, we also consider a reformulation operation (replacement) in the evaluation because a recent work reported that this operation significantly reduces the number of iterations required in comparison to the widespread crossover and mutation operations. The combination of HaFF and the reformulation operation (HaFF_R) improves the results of the best baseline by 0.15% in recall and 14.26% in precision. Analyzing the results, we learned how to better leverage HaFF_R, which increased the improvement with regard to the best baseline to 1.15% in recall and 20.05% in precision. HaFF_R significantly improves precision because humans are immune to the main limitations of the baselines: vocabulary mismatch and tacit knowledge. A focus group confirmed the acceptance of HaFF. These results are relevant for SBMDE because feature location is one of the main activities performed during maintenance and evolution. Our results, and what we learned from them, can also motivate and help other researchers to explore the benefits of HaFF. In fact, we provide a guideline that further discusses how to apply HaFF to other software engineering problems. -------------------------------------------------------------------------------------- 5. a list containing one or more of the eight letters (A, B, C, D, E, F, G, or H) that correspond to the criteria (see above) that the author claims that the work satisfies; G, E -------------------------------------------------------------------------------------- 6. a statement stating why the result satisfies the criteria that the contestant claims (see examples of statements of human-competitiveness as a guide to aid in constructing this part of the submission); (G) The result solves a problem of indisputable difficulty in its field Feature Location is one of the fundamental tasks performed during the maintenance phase of a software product. It is a pre-requisite for most software engineering tasks such as code refactoring, bug fixing, or variability management. There are many approaches available in the literature to perform feature location [1,2] that have evolved over the years, which illustrates the complexity of the problem and the need for different approaches to deal with the different scenarios of feature location. In regards to feature location over models using textual comparisons, there has been an ongoing effort for many years to address the specific problems entailed by its application, such as the vocabulary mismatch or the tacit knowledge. The approach that constitutes our entry for "Humies" 2022 has been able to mitigate those problems thanks to the use of the Human as a fitness function and the design of intelligent operators that reduce the generations needed to reach an optimal solution. In the present work, a feature location problem is addressed in the context of a leading railway company (CAF) in business since 1917. Our previous work [3] estimated that a single engineer would need more than 30 years to manually locate the features present in the software of the company. The features located by our approach are influencing and reshaping how world-class industries currently produce software for their products. The features located by our work are currently being applied in the Train Control and Management software of CAF trains. In other words, software engineers of the top-six world manufacturers of trains prefer to use the features located by our approach instead of their own previous manually located features. Therefore, we consider feature location to be a problem of indisputable difficulty, and our approach addresses this problem for a specific scenario. [1] Rubin, J., & Chechik, M. (2013). A survey of feature location techniques. In Domain Engineering (pp. 29-58). Springer, Berlin, Heidelberg. [2] Dit, B., Revelle, M., Gethers, M., & Poshyvanyk, D. (2013). Feature location in source code: a taxonomy and survey. Journal of Software: Evolution and Process, 25(1), 53-95. [3] Pérez, F., Font, J., Arcega, L., & Cetina, C. (2019). Collaborative feature location in models through automatic query expansion. Automated Software Engineering, 26(1), 161-202. (E) The result is equal to or better than the most recent human-created solution to a long-standing problem for which there has been a succession of increasingly better human-created solutions. In one of our previous works [1], we conducted an experiment comparing manual feature location versus automated feature location in models of similar characteristics (software models used to generate source code). The comparison was based on common performance metrics such as the precision, recall, and F-Measure of the results obtained. Manual feature location outperformed automated in precision (27.79% better) and in F-Measure (by 19.05%) while the automated version obtained better results than manual for recall (by 32.18%). In the present work, the HaFF approach has been able to outperform both approaches (manual feature location and automated feature location) from [1]. HaFF combined with the reformulation operation (HaFF_R) outperforms the manual approach by 39.98% in precision and by 46.85% in F-Measure. In addition, HaFF_R outperforms the automated approach by 19.41% in recall. Before our work, our industrial partners used to manually locate the features, trying to use all of the tools available for them, e.g., spreadsheets or search tools. However, those methods were not scaling properly as the complexity and amount of software increased. Our approach has replaced their previous methods. Therefore, we consider the results produced by our feature location approach to be better than those produced by the humans. [1] Pérez, F., Echeverría, J., Lapeña, R., & Cetina, C. (2020). Comparing manual and automated feature location in conceptual models: A Controlled experiment. Information and Software Technology, 125, 106337. -------------------------------------------------------------------------------------- 7. a full citation of the paper (that is, author names; publication date; name of journal, conference, technical report, thesis, book, or book chapter; name of editors, if applicable, of the journal or edited book; publisher name; publisher city; page numbers, if applicable); Francisca Pérez, Jaime Font, Lorena Arcega, Carlos Cetina; "Empowering the Human as the Fitness Function in Search-Based Model-Driven Engineering," in IEEE Transactions on Software Engineering, doi: 10.1109/TSE.2021.3121253. -------------------------------------------------------------------------------------- 8. a statement either that "any prize money, if any, is to be divided equally among the co-authors" OR a specific percentage breakdown as to how the prize money, if any, is to be divided among the co-authors; Prize money, if any, is to be divided equally among the co-authors. -------------------------------------------------------------------------------------- 9. a statement stating why the authors expect that their entry would be the "best," and Traditionally, Search-Based Software Engineering has been applied to reformulate software engineering tasks into search problems, leveraging existing search strategies to address specific problems from software engineering. This resulted in many research efforts focused on adapting existing problems from the software engineering field to be solved by automatic means. Many tasks that were being solved manually or in semiautomatic ways were transformed into SBSE problems by leveraging a representation of the problem, a set of genetic operations to traverse de search space, and a fitness function to assess the candidates. This relegated the human to the background and put most of the burden on the search strategy. However, search strategies are means to efficiently explore large search spaces but do not excel in other tasks that could benefit from the ability of the human to exploit context information or make subjective assessments. Some works have attempted to address this by enabling the interaction of the human in the search process, mainly by providing scores to complement the fitness function. As part of our work, we have analyzed existing surveys about search-based software engineering applied to model-driven engineering [1] (covering works from 1998 to 2016) and about interactive search-based software engineering [2] (covering works from 1999 to 2017) and updated them (up to August 2020). None of the works included in the surveys, and the updates, have been able to successfully replace the fitness function with a human as our work does. The work presented as our entry to the "Humies" award is the first to propose a new dimension of the interaction between humans and search-based software engineering strategies: to completely replace the fitness function by the human. There is a good reason for not having attempted this before, humans cannot evaluate millions of candidate solutions in a reasonable time. However, our work achieves the replacement of the fitness function by a human leveraging two key aspects: (1) the raise in the abstraction level introduced by the use of software models as the individuals of the population; and (2) a set of intelligent operators that drastically reduce the number of generations needed by the search strategies to reach an optimal solution. As our related work shows, it seems that the community has been following a trend of systematically applying generic operations (mainly single-point crossover and random mutation); our work breaks that tendency and opens a path with promising prospects. In summary, the fitness function is effectively replaced by a human, yielding to better performance results (in terms of common metrics: precision, recall, and F-Measure). This has been validated on an industrial scenario, effectively replacing their former methods for solving this problem. The impact is such that our industrial partner is willing to provide firsthand account of the benefits obtained with our approach. This opens a new way to address software engineering problems, where the human can actively drive the search by applying its domain's knowledge. We expect that the use of humans as the fitness function results in a shift in the way problems from search-based model-driven engineering are solved. To encourage this change, our work includes a set of guidelines on how to apply HaFF to other software engineering problems different from feature location and how to adapt problems from other scenarios (less favorable than model-driven engineering) to apply HaFF. [1] Boussaïd, I., Siarry, P., & Ahmed-Nacer, M. (2017). A survey on search-based model-driven engineering. Automated Software Engineering, 24(2), 233-294. [2] Ramirez, A., Romero, J. R., & Simons, C. L. (2018). A systematic review of interaction in search-based software engineering. IEEE Transactions on Software Engineering, 45(8), 760-781. -------------------------------------------------------------------------------------- 10. An indication of the general type of genetic or evolutionary computation used, such as GA (genetic algorithms), GP (genetic programming), ES (evolution strategies), EP (evolutionary programming), LCS (learning classifier systems), GE (grammatical evolution), GEP (gene expression programming), DE (differential evolution), etc. GA -------------------------------------------------------------------------------------- 11. The date of publication of each paper. If the date of publication is not on or before the deadline for submission, but instead, the paper has been unconditionally accepted for publication and is “in press” by the deadline for this competition, the entry must include a copy of the documentation establishing that the paper meets the "in press" requirement. Published: 19 October 2021