1. Title of paper "Automatic generation of semantically rich as-built building information models using 2D images: A derivative-free optimization approach" 2. Authors Fan Xue KB512, Department of Real Estate and Construction The University of Hong Kong Pokfulam, Hong Kong SAR xuef@hku.hk +852 2219 4174 Weisheng Lu KB515, Department of Real Estate and Construction The University of Hong Kong Pokfulam, Hong Kong SAR wilsonlu@hku.hk +852 2859 7981 Ke Chen Department of Real Estate and Construction The University of Hong Kong Pokfulam, Hong Kong SAR leochen@connect.hku.hk +852 6486 7971 3. Corresponding Author Weisheng Lu 4. Abstract Over the past decade a considerable number of studies have focused on generating semantically rich as-built building information models (BIMs). However, the prevailing methods rely on laborious manual segmentation or automatic but error-prone segmentation. In addition, the methods failed to make good use of existing semantics sources. This paper presents a novel segmentation-free derivative-free optimization (DFO) approach that translates the generation of as-built BIMs from 2D images into an optimization problem of fitting BIM components regarding architectural and topological constraints. The semantics of the BIMs are subsequently enriched by linking the fitted components with existing semantics sources. The approach was prototyped in two experiments using an outdoor and an indoor case, respectively. The results showed that in the outdoor case 12 out of 13 BIM components were correctly generated within 1.5 hours, and in the indoor case all target BIM components were correctly generated with a root-mean-square deviation (RMSD) of 3.9 cm in about 2.5 hours. The main computational novelties of this study are: (a) to translate the automatic as-built BIM generation from 2D images as an optimization problem; (b) to develop an effective and segmentation-free approach that is fundamentally different from prevailing methods; and (c) to exploit online open BIM component information for semantic enrichment, which, to a certain extent, alleviates the dilemma between information inadequacy and information overload in BIM development. 5. Criteria List (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. (D) The result is publishable in its own right as a new scientific result ¡ª independent of the fact that the result was mechanically created. (G) The result solves a problem of indisputable difficulty in its field. 6. Statement on Criteria Criterion B: The vision understanding has been studied for decades. Typically, the automatic detection of semantic information (e.g., class label, geometry, material, texture, functions, and topology) from 2D images (even 3D point clouds) to form building information models (BIMs) is far from satisfaction for buildings and indoor scenes (e.g., furniture). We translate the task as an optimization problem of re-assembling the online open 3D model components, and apply an evolution strategy (i.e., CMA-ES) under the umbrella of derivative-free optimization (DFO) to solve the problem. The approach creates semantically rich BIMs with high success and automation. Though there is not a canonical problem dataset to serve as a benchmark, one can compare the results with machine learning (including deep learning) models which require tremendous efforts on training. Semantically rich BIMs are a critical information infrastructure for not only building research and the global smart city development but also future robotics and autonomous vehicles. By leveraging the power of evolutionary computation (CMA-ES), the approach in the paper is fundamentally different from the existing methods in the construction industry and computer vision community. The formulation in this paper also exposes the problem BIM creation (as well as vision understanding at large) to many more evolutionary computation algorithms. Criterion D: In addition to the significant improvement in detecting semantic information and creating BIMs, this paper proves and promotes a scientific routine of reusing existing, online open domain knowledge in the challenging tasks of BIM creation and vision understanding. The success also shows that evolutionary computation is competent in a field which used to be dominated by sophisticated machine learning models. Criterion G: It is no doubt an important and difficult problem to create semantically rich BIMs from 2D images of buildings and indoor scenes. Noteworthily, it is much more challenging when the images are limited to one or two low-resolution photos. The generalized 2D vision understanding problem, and numerous variations, are even more widely studied and significant. 7. Citation F. Xue, W. Lu, and K. Chen, 2018. Automatic generation of semantically rich as-built building information models using 2D images: A derivative-free optimization approach. Computer-Aided Civil and Infrastructure Engineering. To appear. 8. Prize Money Any prize money is to be divided evenly among the co-authors. 9. Authors' Statement to the Judges The relevance of problem: This paper exposes the real-world challenges of modeling and understanding the built environment to CMA-ES as well as many other evolutionary computation algorithms. As the knowledge dissemination media, the journal (Computer-Aided Civil and Infrastructure Engineering) is one of the top journals in both Computational Theory and Computer Science Applications according to scimagojr.com. A novel paradigm: Most, if not all, traditional methods that create BIMs from images 2D reply on explicit rules (e.g., "vertical planes are probably walls") and implicit machine learning classification models (e.g., SVM). This paper shows that evolutionary computation is competitive in recycling the existing, online open domain knowledge (BIM components) to create semantically rich BIMs. The paradigm alleviates the dilemma between information inadequacy (lacking semantic information in the model) and information overload (many online open BIM components) in BIM development as well as vision understanding. Multi-disciplinary impact: The approach can contribute to multiple disciplinaries and industries, such as architecture and construction (e.g., digital preservation of current conditions of built heritages), smart and resilient city development (e.g., facility and emergency management), computer vision (e.g., indoor/outdoor scene understanding), and autonomous robotics (e.g., vision-based locationing). 10. General Type ES : CMA-ES (Covariance Matrix Adaptation with Evolution Strategy) DFO (derivative-free optimization) : CMA-ES 11. The date of publication In press