混合動力汽車深度強化學習分層能量管理策略

戴科峰 胡明輝

摘要：為了提高混合動力汽車的燃油經(jīng)濟性和控制策略的穩(wěn)定性，以第三代普銳斯混聯(lián)式混合動力汽車作為研究對象，提出了一種等效燃油消耗最小策略（equivalent fuel consumption minimization strategy，ECMS）與深度強化學習方法（deep feinforcement learning，DRL）結合的分層能量管理策略。仿真結果證明，該分層控制策略不僅可以讓強化學習中的智能體在無模型的情況下實現(xiàn)自適應節(jié)能控制，而且能保證混合動力汽車在所有工況下的SOC都滿足約束限制。與基于規(guī)則的能量管理策略相比，此分層控制策略可以將燃油經(jīng)濟性提高20.83%～32.66%；增加智能體對車速的預測信息，可進一步降低5.12%的燃油消耗；與沒有分層的深度強化學習策略相比，此策略可將燃油經(jīng)濟性提高8.04%；與使用SOC偏移懲罰的自適應等效燃油消耗最小策略（A-ECMS）相比，此策略下的燃油經(jīng)濟性將提高5.81%～16.18%。

關鍵詞：混合動力汽車；動態(tài)規(guī)劃；強化學習；深度神經(jīng)網(wǎng)絡；等效燃油消耗

中圖分類號：U471.15????????? 文獻標志碼：A?????????? 文章編號：1000-582X（2024）01-041-11

Deep reinforcement learning hierarchical energy management strategy for hybrid electric vehicles

DAI Kefeng， HU Minghui

（College of Mechanical and Vehicle Engineering， Chongqing University， Chongqing 400044， P. R. China）

Abstract： To improve the fuel economy and control strategy stability of hybrid electric vehicles （HEVs）， with taking the third-generation Prius hybrid electric vehicle as the research object， a hierarchical energy management strategy is created by combining an equivalent fuel consumption minimization strategy （ECMS） with a deep reinforcement learning （DRL） method. The simulation results show that the hierarchical control strategy not only enables the agent in reinforcement learning to achieve adaptive energy-saving control without a model， but also ensures that the state of charge （SOC） of the hybrid vehicle meets the constraints under all operating conditions. Compared with the rule-based energy management strategy， this layered control strategy improves the fuel economy by 20.83% to 32.66%. Additionally， increasing the prediction information of the vehicle speed by the agent further reduces the fuel consumption by about 5.12%. Compared with the deep reinforcement learning strategy alone， this combined strategy improves fuel economy by about 8.04%. Furthermore， compared with the A-ECMS strategy that uses SOC offset penalty， the fuel economy is improved by 5.81% to 16.18% under this proposed strategy.

Keywords： hybrid vehicle; dynamic programming; reinforcement learning; deep neural networks; equivalent consumption minimization strategy

車輛傳動系統(tǒng)的電氣化是未來可持續(xù)發(fā)展中的重要環(huán)節(jié)。但就現(xiàn)階段而言，純電動汽車的電池技術還未實現(xiàn)突破；混合動力汽車（hybrid electric vehicle，HEV）的節(jié)油潛力也沒有得到充分發(fā)揮，設計良好的能量管理策略可以提高節(jié)油率。

混合動力汽車最優(yōu)能量管理的經(jīng)典數(shù)值計算方法有2種：一是基于系統(tǒng)模型的動態(tài)規(guī)劃（dynamic programming，DP）；二是龐特里亞金極值原理（Pontryagins minimal principle，PMP）[1]。其中，DP近似求解哈密爾頓-雅可比-貝爾曼方程以得到最優(yōu)控制問題在離散時間的最優(yōu)解。DP需要獲得完整的駕駛工況信息且計算負荷高，因此現(xiàn)階段僅用DP的離線計算來導出控制規(guī)則[2]。……