CityLearn MARLISA 2022¶

Author: Chia E Tungom
Email: bamtungom@protonmail.com
Date: Aug-11-2022

In this Notebook we outline how the Multiagent Reinforcement Learning Algorithm MARLISA can be modified for the 2022 CityLearn Challenge. For details on how MARLISA works, refer to the study and for the orignal implementation refer to the CityLearn git repository.

A brief of what the algorithm aims to achieve is a follows

1. Implement a Multi-agent version of Soft-Actor Critic (SAC) algorithm
   • Design a reward function with individual and collective goals
   • Agents share information amongs each other using a leader-follower scheme
     • Each building has it's own RL agent
2. Use the algorithm to
   • Plan and Control energy storage of a diverse set of buildings
   • Aim to reshape the aggregated curve of electricity demand
   • Decide how much heating and cooling is stored or released at any time
3. Measure performance on the following metric
   • Annual net electricity consumption in the district
   • Average daily peak demand
   • Annual peak demand
   • Ramping

Unlike the CityLearn environment where MARLISA was implemented in, the 2022 CityLearn Challenge environment has different environmnet features and optimization objective. In this Notebook we aim to show how MARLISA can be adapted to the 2022 CityLearn Challenge. We do so by doing the following

1. Setting Up the CityLearn 2022 Environment
2. Getting Building information (should be MARLISA compatible)
3. Some Data Overview
4. Generate Building State Actions requirements
5. Outline Modifications of MARLISA code
6. Run MARLISA2022

Note: I only aim to make it easier for those who want to get started with MARL(ISA)

To Run this Notebook, you will need to have the started git repo with the following folder

• Get the MARLISA python file from the intelligent-environments-lab git hub repo and place in the agents folder
• Get the common folder from the intelligent-environments-lab and place in the root folder of ur starter git folder
• Place this Notebook in the root folder as well
• YOU ARE GOOD TO GO !!!!

This Notebook was made while listening to MONALISA. Enjoy!!!

Lets Gooooooooooo!!!

1. Load Environment¶

We beginning by setting Up our RL environment for CityLearn2022. Some of the code in this section was taken from the local evaluation python file in the starter kit repo

2. Get Building Information¶

Building Information is needed as input in the MARLISA algorithm. Building information contains information like

• A bildings solar power storage capacity
• A buildings annual demand for cooling, heating and dhw (deep hot water)
• And other information

To get building information, you can run the command env.get_building_information(). The information we get does not contain keys like Building_1 which as needed but rather some sequence of random letters and numbers. We make modifications to rename the buildings in the order they appear (I am not sure this renaming is proper and if you have an idea please leave a comment below).

The Buildings are renamed the same way buildings are named in the CityLearn environment.

3. A Look at Building Data¶

Taking a Look at Building Data we see that the observations

• Indoor Temperature [C], Average Unmet Cooling Setpoint Difference [C] and Indoor Relative Humidity [%]: All have NaN values
• DHW Heating [kWh], Cooling Load [kWh] and Heating Load [kWh]: All have values 0

If this is the case when sampled in the environmet, we don't need this variables for learning a model as they don't provide any information about a building or state.

This is useful because inside MARLISA we can set which observations to be removed if need be

4. Generate Building States Action Requirements¶

This is an input to the MARLISA algorithm and is used to determine which states or actions are needed for each building. Unlike in the 2020 MARLISA where a buildings requirements file is nicely provided, the information in the 2022 challenge is in the schema.json file. The Requirements for each building state and action is provided (I assume it is same for all buildings) but we need to modify it for our inpute to suit the algorithm.

In the file a state with value True means it's required for the given building and an action with value True means that action needs to be taken for that building and False otherwise.

We see that only action electrial storage is needed.

NOTE: The order of the states is same as the order of the observations i.e if month is first then in observation it's in index 0. e.g Hour comes third which mean it's found in index 2:

After generating the file, you can save it in Json format for usage as input or give to your algorithm in the dictionary format. Here we use the later. If you use the later, comment line [37-38] where the file is read in.

5. Internal MARLISA Modification¶

MARLISA CODE Tips for modification for 2022 challenge

5.1. Initialization (init body)¶

[Line 58 to 65] abstract information to individual buildings

[Line 75-83] Energy size coefficient for every building (Not Needed in 2022)

[Line 86-111] Define Encoder: Set Regression Learner for every building, define Encoding for every observation (think of it as data column) and set target variable to be removed MODIFY ACCORDING TO THE CURRENT DATASET

[Line 131-145] Define Regression Encoder: for transforming states in regression model MODIFY ACCORDING TO THE CURRENT DATASET

[Line 149-164] Solar Capacity (remove variables if no solar PV): removes solar radiation related variables for houses without PV COMMENT THIS SECTION

5.2. Select Action (MARLISA method or function)¶

• Takes as inputs states and deterministic
  • states: the states of the buildings
  • deterministic: boolean can be true or false

**[Line 222-226] Initialize coordination variables: coordination variables are two dimesional for every building. MODIFY ACCORDING TO THE CURRENT DATASET (in 2022 we are asked to optimize for electricty cost and carbon emission)

• Capacity Dispatched: This is the toal amont of electricity already dispatched. it is related to the energy size coefficient at every time step
• **Electrical Demand: Related to the total electricty demand estimated by a prediction algorithm. Different in Information Sharing and Non Info Sharing Cases

AFTER the recommended modifications you can make a trial run as shown next

The REWARD FUNCTION can also be modified

6. RUN MARLISA 2022¶

To run Marlisa, we need to give it parameters that influence it's learning

The MARLISA Algorithm takes inputs that we can be classified into two categories

6.1. Environment Parameters¶

These are parameters specific to the reinforcement learning environment (CityLearn Version). They give information about the simulation envrionment that will be used. details about these environmental variables are explored above, to understanding the 2022 citylearn environment, check this notebook. A summary explanation can be laid down as follows.

• 'building_ids': These are the buildings identities in the environment written in the form "Building_id" where id is a building number a building_id can look as follows
  • ["Building_1", "Building_2", ... , "Building_n"]
• 'buildings_states_actions': This is a json file defining the different states and actions possible for a building e.g If a building has a
  • states {day : False, temp: True} it means there will be information for temp but not for day
  • "actions": {"cooling_storage": true, "dhw_storage": true, "electrical_storage": false}. this means there will be no action required or electric storage is absent in the building
• 'building_info': Gives valuable information about a building like
  • solar_power_capacity (kW)
  • Annual_DHW_demand (kWh)
  • Annual_cooling_demand (kWh)
  • Annual_nonshiftable_electrical_demand (kWh)
  • etc
• 'observation_spaces': This is information about the observation space of every building in the environment.
  • It contains n arrays where n is the number of buildings
  • Each array contains the lower and upper bound for the building observation along with it's dimension and datatype
• 'action_spaces': This is information about the actions_spaces of every building in the environment
  • It contains n arrays where n is the number of buildings
  • Each array contains the lower and upper bound for the building action along with it's dimension and datatype

6.2. Algorithm Parameters¶

These are parameters specific to our reinforcement learning algorithm. Details about these parameters can be found in the paper. The settings below are as provided in the original MARLISA implementation found here

• hidden_dim:[256,256],
• discount:0.99,
• tau:5e-3,
• lr:3e-4,
• batch_size:256,
• replay_buffer_capacity:1e5,
• regression_buffer_capacity:3e4,
• start_training:600, # Start updating actor-critic networks
• exploration_period:7500, # Just taking random actions
• start_regression:500, # Start training the regression model
• information_sharing:True, # If True -> set the appropriate 'reward_function_ma' in reward_function.py
• pca_compression:.95,
• action_scaling_coef:0.5, # Actions are multiplied by this factor to prevent too aggressive actions
• reward_scaling:5., # Rewards are normalized and multiplied by this factor
• update_per_step:2, # How many times the actor-critic networks are updated every hourly time-step
• iterations_as:2,# Iterations of the iterative action selection (see MARLISA paper for more info)
• safe_exploration:True

NOTE: Set the appropriate number of buildings as the environment. Ensure the BuildingsStatesActions generated have same number of buildings as the given environment

HOPE U LIKED THE SONG¶

Leave a comment for any questions, doubts or corrections