1. Paper title: Data-Driven Reinforcement Learning–Based Real-Time Energy Management System for Plug-In Hybrid Electric Vehicles ------------------------------------------------------------------------------- 2. Author contact details: Xuewei Qi Department of Electrical and Computer Engineering, University of California-Riverside, Riverside, CA, 92507, USA xqi001@ucr.edu Matthew J. Barth Department of Electrical and Computer Engineering, University of California-Riverside, Riverside, CA, 92507, USA barth@ece.ucr.edu Guoyuan Wu Center of Environmental Research and Technology University of California-Riverside, Riverside, CA, 92507, USA gywu@cert.ucr.edu Kanok Boriboonsomsin Center of Environmental Research and Technology University of California-Riverside, Riverside, CA, 92507, USA kanok@cert.ucr.edu Jeffrey Gonder National Rewable Energy Laboratory, Golden, CO, 80401, USA jeff.gonder@nrel.gov ------------------------------------------------------------------------------- 3. Corresponding author: Xuewei Qi xqi001@ucr.edu ------------------------------------------------------------------------------- 4. Paper abstract: In recent years, transportation-related energy,consumption and air quality degradation problems have gained an increasingly amount of public concern. Recently,Plug-in hybrid electric vehicles (PHEVs) show great promise in reducing transportation-related fossil fuel consumption and greenhouse gas emissions. Designing an efficient energy management system (EMS) for PHEVs to achieve better fuel economy has been an active research topic for decades. Most of the advanced systems rely either on a priori knowledge of future driving conditions to achieve the optimal but not real-time solution (e.g., using a dynamic programming strategy) or on only current driving situations to achieve a real-time but nonoptimal solution (e.g.,rule-based strategy). This paper proposes a reinforcement learning– based real-time EMS for PHEVs to address the trade-off between realtime performance and optimal energy savings. The proposed model can optimize the power-split control in real time while learning the optimal decisions from historical driving cycles. A case study on a real-world commute trip shows that about a 12% fuel saving can be achieved without considering charging opportunities; further, an 8% fuel saving can be achieved when charging opportunities are considered, compared with the standard binary mode control strategy. ------------------------------------------------------------------------------- 5. Relevant criteria: A,G ------------------------------------------------------------------------------- 6. Statement: (A) A patent application has been filed by University of California, technology commercialization office for the proposed system in this paper. It is invented herein a data-driven, model-free, reinforcement learning-based, real-time energy management system (EMS) of plug-in hybrid electric vehicles (PHEVs). The proposed system is capable of simultaneously learning and controlling power-split operations of a PHEV’s powertrain in the most fuel-efficient manner in response of different driving situations (e.g., traffic conditions, roadway topology, charging opportunities and weather). A real-world case study on commute trips shows that about 12% fuel savings can be achieved compared to the conventional (binary mode) energy management strategy. The claimed novel features and advantages of this invention are: 1) Data driven. The invented system does not rely on PHEV’s powertrain model information once it is well trained, since all the decision variables can be retrieved without any model-involved calculation; 2) Resilient and customizable. The learning mechanism enables the system to be adaptive to complicated driving conditions in real-world and personalized driving behavior. The optimal or near-optimal decision is made by learning from historical experience; 3) On-line deployable. The invented system can be implemented in real-time without any prediction efforts, since the control decisions are made only upon the current system state. It achieves a good balance between real-time performance and fuel savings; 4) Charging opportunity considered. This system considers charging opportunities between trips in order to maximize the fuel efficiency of entire trip chain. medial coverage of this work: IEEE http://spectrum.ieee.org/cars-that-think/transportation/advanced-cars/hybrid-car-system-learns-fuel-efficiency Science Daily https://www.sciencedaily.com/releases/2016/02/160209132051.htm (G) The optimal power-split between engine power and vehicle battery power in a powertrain of plugin hybrid electric vehicle is believed to be a very complicated high dimensional optimization problem. For all the existing state-of-the-art methods for this problem, a short-term prediction is always required to predict and obtain the necessary future information to implement the receding horizon based optimization. However, the trip information (e.g., driving speed profile along a trip) is usually not available before the trip in real-world driving. The proposed reinforcement-learning based methods doses not need any previewed information when driving once the model is well trained with historical driving data. The testing of the proposed method with a set of real-world commute trip data shows that it is able to learn the optimal power-split control strategy from driving history with enough training. Hence the proposed model solves a real-world optimization problem that is theoretically impossible without predicted future driving information. --------------------------------------------------------------------------- 7. Citation: Xuewei Qi; Guoyuan Wu; Boriboonsomsin, K.; Barth, M.J.; Jeffery Gonder, “Data-Driven Reinforcement Learning-Based Real-Time Energy Management System for Plug-in Hybrid Electric Vehicles” Transportation Research Record (Journal of Transportation Research Board),vol,2572,pp. 1-8,2016. DOI: 10.3141/2572-01 ------------------------------------------------------------------------------ 8. Any prize money, if any, is to be divided equally among the co-authors. ------------------------------------------------------------------------------- 9. "Best" statement: We are confident that the proposed model in this work is among the most advanced models for the described engineering optimization problem at this moment (2016). It is very promising in real-world applications to help improve the fuel efficiency of plugin hybrid electric vehicles which is enhanced by the media coverage of this work and the fact that some major automakers (e.g., Honda) showed their interest in commercializing this proposed method in their hybrid vehicle models(e.g., Accord hybrid). ------------------------------------------------------------------------------ 10. Method used: Reinforcement Learning(RL).