Most early studies, as well as current work, focus on direct intrusive control methods and frameworks. Early work that tackled aggregated modeling of TCLs can be found in 14 and 15. The solution computation and controller design of these approaches is considerably difficult, which represents a drawback for these approaches. These issues were mitigated in more recent works ^{5, 7, 16} using a different class of linear population-bin transition models based on Markov chains. Other approaches have proposed time-varying battery models with dissipation such as 17 or without dissipation as in 18. These approaches were used to compute near-optimal control trajectories with a reduced computational cost. Although optimal pricing for demand side management has been thoroughly studied in the literature ^{19– 21} , the price-based control of TCLs remains only briefly addressed in the literature. In 22, the operating reserve capacity of aggregated heterogeneous TCLs was evaluated using a TCL model that takes into consideration consumer behavior. The price-based approach was also addressed from the consumer perspective in 23. The objective of the proposed method was mainly to find the optimal set point change in response to electricity prices in other to minimize the increases in the electricity bill due to dynamic pricing. The power gain from this control scheme was then used for load following supply. Another approach was proposed to find the equilibrium between the electricity prices and the users’ comfort. Using a Stackelberg game approach, authors in 24 presented a unique Stackelberg equilibrium that maximizes the utility function and minimizes dissatisfaction cost of TCLs users. A similar approach was proposed in 25 and 26 using a mean-field game approach to find the best pricing scheme considering TCLs as price-responsive rational agents.

Deep learning and other machine learning methods are largely applied in DR programs ²⁷ . The implementation of a TCL cluster control program faces the problem of uncertainty and heterogeneity of the TCL units’ behaviors in response to control prices. Consequently, many researchers were interested in using machine learning models that can learn aggregate or individual behavior of TCL units under partial observability. A model-free reinforcement learning was early proposed in 28 for TCL control that gives similar results as model predictive approaches. Reinforcement learning approaches were also used in ²⁹ to control domestic water buffers according to a local photovoltaic production for the maximization of self-consumption. More recently, the success of deep reinforcement learning approaches has inspired more researchers to tackle the problem of direct TCL control using deep reinforcement learning. Authors in 30– 33 have used different deep neural architectures for automatic estimation of the TCLs’ state’s features in a batch reinforcement learning model. The same authors have later provided a comparison of the different architectures in 33, 34. The LSTM architecture has outperformed the other deep neural network architectures. These works focused only on deep Q-learning, which is based on the estimation of a quality function for every potential action before performing the optimization. In 35 Deep policy gradient method was explored along with deep Q-learning for an on-line energy optimization of the buildings.

Following the above-mentioned literature and the success of LSTM networks in mitigating the problem of partial state information and solving long-term dependency problem ^{13, 33, 34} , we propose a two-step pricing optimization method for the exploration of TCL flexibility in energy arbitrage. This paper addresses the need for new non-intrusive TCL control methods via electricity prices proxies, so far lacking in the scientific literature. The proposed method relies on LSTM networks learning individual TCL unit behavior and the prediction of individual responses to electricity prices. The individual predictions are aggregated to form an overall prediction model. This model is used in a genetic algorithm (GA)-based optimization algorithm to maximize a retailer’s profit considering grid and energy cost constraints. To the best of the authors’ knowledge, this is the first work that uses LSTM networks in a non-intrusive TCL control problem based on electricity prices within a DR program. The main contributions of this paper are the following:

The agent is only able to measure a partial observation of the true state i.e. no information about the indoor temperatures, resulting in a partially observable Markov decision problem. The observable state space X consists of two variables: the outside temperature, and the electric load:

x t = ( L t , T t ) ( 3 )

Since the observable state space only includes part of the true state, it is not possible to directly model future state transitions. Yet this remains convenient when following the results from 13 that we can predict the next step electric load L _{t +1} using the information of outdoor temperature T _t , the electric load L _t and the electricity price P _{t +1}. The state is extended with sequences of past observations of states and actions, which results in a non-Markovian state.

For each TCL, the electric load is approximated by:

L t + 1 ∼ g ( L t , T t , P t + 1 ) ( 4 )

We assume that the outside temperatures’ forecasts are available for every future timestep in the control horizon.

The control action u _t consists of the electricity price that the retailer announces for each time step of the control horizon. As mentioned earlier, even though the retailer is not controlling the TCLs directly, we assume that the TCLs react directly to electricity prices. Therefore, the electricity price controls the state by influencing the amount of energy consumed during a timestep t. The next state is then defined by:

x t + 1 = f ( x t , P t , w t ) ∼ ( g ( L t , T t , P t + 1 ) , T t + 1 ) . ( 5 )

According to the existing literature, the control of TCLs clusters can be performed considering different objective functions. For instance, the objective can be tracking a balancing signal or energy arbitrage. In this work we consider an energy arbitrage problem where a retailer is trying to maximize their profit. However, the framework and methods presented here might as well be applied to different objective functions. We consider the profit as the difference between the revenue and the cost function. We assume that the cost function C _t ( L _t ) is convex increasing in L _t for each timestep as formulated in 36.

C t ( L t ) = q L t 2 + p t L t + c ( 6 )

where, q > 0 is a constant, p _t > 0 is the electricity price in the wholesale market and c > 0 is a fixed cost.

In order to avoid overload during peak times, we introduce a maximum load capacity of the power network, denoted L _t,max at each timestep. Therefore, we have the following constraint:

L t = ∑ n L n , t ≤ L t , m a x , ∀ t ∈   H ( 7 )

The revenue is the bill that customers would pay for using the energy during the time window H:

R = ∑ t = 0 H ( L t * P t ) ( 8 )

Usually, there exists a total revenue cap, denoted as R _max , for the retailer. Therefore, we need to add the revenue constraint to improve the acceptability of the retailer’s pricing strategies, i.e., without such a constraint, the retail prices will keep going up to a level which is against energy regulations as well as financially unacceptable to the customers. As a result, we have the following constraint:

R < R m a x ( 9 )

Moreover, for each timestep t ∈ H, we define the minimum and maximum price that the retailer (utility company) can offer P _t,min and P _t,max , we have:

P t , m i n ≤ P t ≤ P t , m a x , ∀ t ∈ H ( 10 )

P _t,min and P _t,max are usually designed based on historical prices, market competition, customers’ acceptability, and the wholesale price. It is reasonable to assume that the price the retailers can offer is greater than the wholesale price for each hour, and there exists a price cap for the retail prices due to retail market competition.

Finally, the control problem defined the optimization of the price vector P, during the time horizon H, can be modeled as follows:

m a x P { R − ∑ t = 0 H C t ( L t ) } ( 11 )

subject to constraints:

R < R m a x ( 12 )

L t ≤ L m a x ( 13 )

P t , m a x ≤ P t ≤ P t , m a x , ∀ t ∈ H ( 14 )

LSTM networks are recurrent neural networks that consist of memory blocks. These memory blocks replace the summation units in the hidden layers in a standard recurrent neural network. The input vector and the hidden state vector are passed through the forget gate to determine the keeping rate of the cell state components. The same vector is passed through the input gate to determine how much of the new state candidate C can pass to the new cell state. Finally, the output gate will decide how much of the transformed state cell vector can be passed to the next hidden state vector h _t . Following 13, the proposed LSTM network consists of multiple layers of LSTM cells followed by a fully connected layer as illustrated in Figure 1. In the case of our model, the input I _n,t is a 2 x 3 matrix that consists of the electric loads, the temperatures and the electricity prices as follows:

I n , t = ( L n , t − 1 , T t − 1 , P t L n , t , T t , P t + 1 ) ( 15 )

The LSTM network recurrently uses the historical information of loads, temperatures and prices to predict electric load for an individual TCL n, in the next timestep. The aggregation of these predictions gives an approximation of g function mentioned in the previous section.

Initially, for each TCL agent n ∈ N we train an LSTM network based on the historical reactions of these TCLs to prices and temperatures. We assume that a DR program is implemented during a long period, enough to collect a sufficient amount of data related to the reactions of TCL agents to prices and temperatures.

Due to the discontinuous nature of the objective function and the complicated dependency between the function electric load L and the electricity prices P, the conventional nonlinear optimization methods are not usable for this problem. Therefore, GA-based optimization algorithms are more suited for this problem ³⁷ . The proposed GA algorithm uses rank selection and value encoding ³⁸ . Each chromosome represents a pricing policy P and consists of a vector of size H. We use uniform crossover ³⁹ and non-uniform mutation ⁴⁰ . The constraints are handled by the approach proposed in 41.

The proposed GA-based optimization algorithms for TCL pricing control are given in Algorithm 1 and Algorithm 2.

In Algorithm 1, we initialize a population of NP pricing policies at step 1. For each policy P we perform steps 2–6 to evaluate the fitness function and the feasibility for each policy. The evaluation of policies is performed using LSTM sequence prediction presented in Algorithm 2. The best policies are selected, and a new generation is created using crossover and mutation operations in step 10. This process is repeated until a stopping condition or maximum number of iterations is reached. At the end of the optimization process, the best pricing policy is selected, and prices are announced to TCL agents via two-way communications technology. After each control episode, the LSTM learning models are updated according to the new data collected from the actual response to the implemented electricity prices.

Following 13 the simulation data is generated using two fuzzy logic systems with the following assumptions:

TCL agents are operating during the day to maintain a comfortable temperature of the space while taking into consideration the electricity price in a given hour. Fuzzy logic is used in this problem because it can model non-qualitative concepts like “hot temperature” or “low price”. The combination of the two fuzzy logic systems delivers the load L _{n,t +1} using the outdoor temperature T _t and the electricity price P _{t +1}. The simulation is performed with different parameters to generate diverse data for 30 TCL agents. The temperature and price data used for the simulation are taken respectively from the Kaisaniemi observation station in Helsinki, available online in 42, and Elspot DA electricity prices in Finland ⁴³ for the period between 1 ^st January 2017 and the 7th September 2018. The generated dataset consists of 14,734 data points for each TCL agent.

The data generated from the above-mentioned simulations is used to train the LSTM networks to learn the behavior of each individual TCL agent. The hyperparameters and structure of the LSTM networks are chosen according to the results of 13 and summarized in Table 1.

The results are evaluated using validation data generated from the same simulations. Figure 2a illustrates the learning results for three TCL agents during different time periods with different temperatures and prices. Figure 2b illustrates the comparison between the real and predicted average power consumption of the 30 TCL agents cluster. The power curves show that the TCL agents’ responses to prices and temperatures are slightly different. In general, the power consumption is high when the temperatures and electricity prices are low and vice-versa. The comparison between the true load curves and the predicted load curves show a very small prediction error per hour in most cases. The true and predicted load curves have similar shapes and significant resemblances. The peaks and valleys are also predicted accurately in most of the cases, which gives a valuable insight for demand side management.

We run the GA optimization algorithm on a population of size 100 for 100 iterations. The parameters used for the optimization are summarized in Table 2. The optimization process is graphically presented in Figure 3. The learning process is measured by the fitness of the best individual in the population at each iteration. Figure 4 illustrates the results of the best pricing solutions for one day. Figure 4a is an illustration of the electricity prices fluctuations during the 24 hours. Figure 4b shows a comparison between the power consumption of the whole cluster under original prices and the power consumption under optimized prices. Figure 4c presents the revenue and profit that the retailer would make under original and optimized prices. Figure 4d presents daily bill of each user of the cluster under original and optimized prices.

The results show a general increase in prices throughout the day. However, this increase didn’t result in an increase in the daily electricity bills. Most of customers will be paying a slightly lower amount per day. This is a consequence of upper limit constraint on the revenue described in ( 12). The overall consumption of electricity was decreased comparing to the original pricing scheme which gives a good idea about the potential energy saving that an optimal pricing strategy can offer.

In order to validate the performance of the proposed algorithm, we consider a case where we have a full access to TCL units’ behavior, i.e. the exact electricity consumption of each TCL unit given temperatures and prices at each timestep. The optimization is performed with direct access to the simulation model described above, which provides full observability and perfect information about the TCLs. This theoretical setup can serve as a benchmark of our method. It can be seen as an upper limit on the profit possibly made by the aggregator without violating the constraints.

The results illustrated in Figure 5a–d, show that the proposed methods have performed very similarly to the benchmark. The hourly prices in Figure 5a, are only slightly shifted from the benchmark prices during most of the day. The difference is only significant in 2 to 3 points. The same observation can be made for the revenues and profits in Figure 5b and electricity consumption in Figure 5c. The comparison of daily bills under optimized prices and benchmark prices in Figure 5d shows a slight rise in the electricity bill in the benchmark model for most customers. This can be explained by the slight increase in prices illustrated in Figure 5a.

The daily revenues and profits under original, optimized and benchmark prices are compared in Figure 6. The comparison shows a closely similar revenue in the three cases. The optimized prices have given a slightly smaller revenue compared to the revenue from original and benchmark prices. However, the profit from original prices is considerably smaller than the profit from optimized prices. The latter is only slightly smaller than the benchmark’s profit. Numerically, the profit from the proposed methods is 95.97% of the optimal benchmark profit. This observation shows that an increase in the profit can be made without an increase in the revenue when the prices are optimized correctly.

Figshare: LSTM+GA data, https://doi.org/10.6084/m9.figshare.9746786.v1 ⁴⁴ .

This project contains the following underlying data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).