Best Practice Guide for Canadian Data Centres

Overview

Internationally, data transfers are expected to grow exponentially over the next few years. ^{Footnote 4} Worldwide, it is estimated that the transmission of data over networks consumes anywhere from 260 to 340 TWh or around 1% to 1.4% of the electricity used globally. Canadian-based data centres are following a similar growth path in terms of services and energy demand. Notably, a Natural Resources Canada (NRCan) market assessment ^{Footnote 5} estimated that data centres accounted for 1.4% of total electricity consumption in 2021 ^{Footnote 6} and increased by 50% to reach 2.1% in 2023. ^{Footnote 7} The International Energy Agency (IEA) has also estimated that data centres may double their 2023 demand levels by 2026. This growth path has the potential to challenge local electric grids in the near future. Currently, data centre (DC) services already account for a significant floor footprint ^{Footnote 8} and electric demand in the major Canadian markets of Toronto, Montreal, and Vancouver. Electric demand was estimated at 320 MW, 520 MW, and 46 MW for these three markets respectively.

While electricity typically has a low average emissions factor in some provinces, during periods of high electricity demand, data centres can put additional pressure on the power grid. This can force grid operators to use supplementary energy sources that are sometimes less clean to meet the increased demand. Moreover, data centres are equipped with fossil fuel-powered generators for backup power in case of main grid failures, leading to higher carbon emissions. Therefore, it is crucial to implement energy efficiency and management solutions to ensure reductions in carbon emissions, aligning with Canada’s environmental priorities. Canada is committed to achieving carbon neutrality by 2050, an initiative known as Net Zero 2050. Data centres can contribute to this goal while continuing to grow.

Governments and industry organizations around the world have recognized the need to address energy consumption in this energy-intensive industry. For example, the EU has developed a voluntary program to help data centre professionals to optimize energy use without jeopardizing essential operations in their facilities, with additional requirements soon to be established. Specifically, data centre operators located in the EU and with an installed IT power demand of at least 500 kW must submit reports to the European (EU) 2024/1364 database by 15 September 2024. ^{Footnote 9} Therefore, it is critical for NRCan to provide DC operators and users in Canada with useful information about international best practices for their consideration and adoption.

Objectives

The objective of this NRCan Best Practice Guide for Canadian Data Centres is to present the most up-to-date (for 2024) international best practices for optimizing DC energy use and essential operations. The best practices are suitable for use by operators and users of the different types of existing data centres in Canada. This Best Practice Guide (BPG) is intended to serve as an educational and reference document in advising and supporting Canadian data centre operators, owners, users, and other stakeholders in prioritizing and managing their processes and related energy consumption in a cost-effective manner without hindering the mission-critical functions of data centres. As this Guide is the first iteration of its kind, it also establishes a format for future iterations and ways to promote best practices within the industry and through possible collaboration with other stakeholders.

About this Guide (Who Is this Guide for?)

Broadly, this BPG is aimed at the Canadian data centre industry. Three primary research and analysis activities were carried out in the development of this BPG. An assessment of international documents, standards, methodologies, and other information sources was used to establish a comprehensive knowledge base of data centre operational best practices. One of the major documents used in the assessment is the well-established “2023 Best Practice Guidelines for the EU Code of Conduct on Data Centre Energy Efficiency,” (and the 2024 version) developed by the EU Joint Research Centre (JRC). This BPG also draws on the NRCan commissioned study Canadian Data Centres Energy and Technology Assessment (DC ETA) completed in 2021 and its 2023 update (also known as the “NRCan Market Assessment”). That NRCan study developed a consistent DC definition and categorization system and provided key characteristics on the different types of data centres operating in Canada today. Finally, another examination and reviews of best practice guides and other educational and outreach efforts were conducted. This latter effort helped to identify organizational strategies for guides as well as possible promotional approaches to highlight operational excellence and other means to educate, promote, and encourage the adoption of the best practices being used elsewhere. A graphical representation of the research and analysis methodology is shown below:

Much of the technical information in this BPG was drawn from the “2023 Best Practice Guidelines for the EU Code of Conduct on Data Centre Energy Efficiency” (2023 EU DC CoC) as well as the 2024 edition of the CoC. However, as the EU DC CoC contains information for all use cases, it was necessary to select and organize this information in a manner more appropriate to DC operators, users, and stakeholders operating in Canada today.

Based on the results of the NRCan market assessment, it is understood that the Canadian data centre industry is comprised primarily of the following categories or segments:

• Enterprise Data Centre: An enterprise data centre is a physical facility that an organization owns and operates to store and process data and support its IT needs. A typical enterprise data centre houses an organization’s IT equipment such as routers, switches, servers, firewalls, and other components that enable critical applications and store and process data.
  • Notes:
    • This category includes cryptocurrency data centres. The 2023 NRCan study found that, since 2018, a small number of large, high-density cryptocurrency data centres opened in Quebec and other provinces with low-cost electricity. However, the study findings suggest that cryptocurrency mining is unlikely to grow significantly, at least in two of the three major markets (Montreal and Vancouver).
    • It is also expected that many of the best practices that can be adopted for data centre operations should also apply to cryptocurrency data centres and vice-versa. ^{Footnote 10}
    • It is also expected that the widespread adoption of AI will have significant impacts on demand for DC services and can increase both the number of data centres and density of existing facilities to which many of the current best practices to optimize energy and operations still apply.
• Data Centre Colocation Providers: These consist of multi-tenant hosting facilities (also called colocation data centres) that include data centres owned by third parties, housing servers owned by companies, as well as data centres that sell server services to companies that want to outsource management of their own servers (managed hosting);
• Data Centre Colocation Users: Many businesses and public sector organizations are transitioning to managed services from owning their own servers and/or facilities to managed services through the use of colocation particularly to secure cloud computing with hyperscale providers.

It should be noted that the list of best practices provides a common terminology and frame of reference for describing an energy efficiency practice. However, this initial list is not comprehensive and may not be applicable across the abovenamed groups, or they may not be equally applicable depending on the application area or for other reasons. A different approach is used to present the relevant information for the Canadian data centre industry, as represented in Figure 2 below.

This different approach groups both the “enterprise data centre” and “colocation providers” categories into one group or industry category of data centre operator types, with a concentration of common practices for both, as well as separate practices applicable to one or the other.

Organization of this Guide

This Guide consists of this introductory section and two main technical sections, each corresponding to a category (or segment) of the Canadian data centre industry identified above. For each industry category, relevant operational practices are identified for stakeholder consideration. These practices are further classified into sections or practice areas as listed below:

• Metering and metrics: Focused on the measurement and analysis of data centre performance indicators;
• Management: Covers the strategies and practices for efficient data centre operations and administration;
• Cooling: Addresses the techniques and technologies for maintaining optimal temperatures within data centres;
• Resilience: Ensuring uninterrupted service delivery and mitigating risks associated with downtimes;
• General practices: Includes a variety of best practices that apply broadly across the data centre industry;
• Hardware: Focused on the physical components that make up the infrastructure necessary for storing, processing, and transmitting data;
• Data management: Deals with the processes and technologies for handling, storing, and protecting data within data centres.

For each section, the most important and impactful practices in terms of energy and management are identified. For example, colocation providers are guided through best practices in metering, management, cooling, resilience, and so on, which are appropriate for their situation. It should be noted that, while the practice areas may have the same names, the specific actions for each industry category/segment may differ. Moreover, the practice areas may be presented in a different order depending on their potential impacts or results.

How to use this Guide

How this guide is used depends on the data centre industry category or market segment of interest to the reader. An overview is provided in each section, followed by the identification of the most impactful practices that can be adopted along with details and recommendations or tools on how to implement the practices. Note: the EU DC CoC considers less “intrusive” practices as those applicable to existing data centre and IT equipment, retroactively where necessary. The EU DC CoC also includes a number of “inappropriate” practices – those that present an unnecessary burden when applied to an existing running data centre. These practices are identified as those to be implemented either when new IT equipment or software is sourced and deployed or during a retrofit of the facility. Many of these latter practices have the potential to provide substantial benefits and are intended to achieve efficiency improvements through the natural replacement cycle of equipment and facilities.

For NRCan, this Guide represents a first step in working with the DC industry and other stakeholders to highlight DC operational excellence. It should be noted that the EU currently has a program to recognize Participants to the Code of Conduct. Participants are those in the DC industry who have committed to a “useful and substantial level of energy saving effort”, of which measurement is the requisite first step. To reach Participant status, it is necessary to adopt and implement a subset of the best practices representing the expected minimum level of energy saving activity for a facility or location. It is also important to note that this program is expected to be the forerunner of a more stringent EU energy efficiency program aimed at addressing the projected growth in energy demand from this sector. ^{Footnote 11}

For the Canadian DC industry, it has been proposed that NRCan promote and support the first step – improve industry awareness about DCs’ impacts on local utility services as well as their overall environmental impacts. A recommended action is the systematic measurement and voluntary reporting of an indicative metric – such as the data centre power density (PD) metric. The data are voluntarily shared by operators to help improve industry knowledge and inform further actions.

As governments and industry organizations recognize the need to address DC operational efficiency and impacts, there will be more and structured initiatives to help data centre professionals optimize their operations and use of resources without jeopardizing the essential operations and performance of their facilities. With sufficient industry participation, NRCan may consider a more visible promotional campaign and program, perhaps in collaboration with provinces, utilities, and operators. Such a program can serve to recognize those DCs with the most efficient metrics, those that have adopted a significant number of practices, or those that have generated the most significant results, among others. ^{Footnote 12} Additionally, this Guide provides recommendations on the types of data to be collected, compiled, and shared to help improve industry and general knowledge as well as inform future actions. In the future, this role may be fulfilled by a non-governmental third party (such as an educational institution, utility, or others) whose objective is to promote energy and data handling best practices and can be relied upon to maintain the confidentiality of the data, similarly to the current practices by Green Grid, an industry organization.

This BPG is aimed at broadly serving as a reference document and practical manual to assist data centre operators in identifying and implementing measures that can improve operational efficiency. The actions listed herein also provide a common terminology and frame of reference for describing a best practice and to assist with their adoption. This BPG is also intended to be a dynamic document and will be updated on a regular basis with new or improved data centre best practices as they emerge or are refined and can be widely shared.

1.1Overview

The information in this section is targeted at DC operators – those who are responsible for the enterprise data centre from the physical building to its operations through to the delivery of IT services. It is also targeted at colocation providers – those who operate the data centre for the primary purpose of selling space, power, and cooling capacity to customers who will install and manage their own IT hardware and services. These two groups manage and/or own the data centre space, power, cooling, IT equipment, and some level of software for the purpose of utilizing or selling the IT services of these facilities. As such, they have operational oversight over many, if not all, of these aspects and are well positioned to invest in measures or practices that optimize the different aspects of or entire DC operations. To a certain extent, many of the practices included below may be adopted by those that operate their own server rooms and others where appropriate.

It is also expected that one or more of the practices listed below have been adopted by some operators. However, as the adoption of best practices is one of the lesser well-known aspects of the Canadian data centre market, the practices provided here are those that will provide the most impacts in terms of operational improvements.

Jump to the Data Management Part 1: Enterprise Data Centres section Jump to the Hardware section Jump to the Enterprise Data Centre and Colocation Providers section Jump to the Metering and Metrics section Jump to the Management section Jump to the Cooling section Jump to the Resilience section Jump to the General Practices section Jump to the Colocation Users section Jump to the Data Management Part 2: Colocation Providers section

1.2Quick Start & Category Specific Actions

Most DCs looking for a quick start should consider adopting one or more practices among the three listed below. It is believed that they are among the most impactful actions that operators of enterprise data centres and/or colocation providers can take:

• Conduct metering and use metrics to be aware of data centre consumption (and to identify a particular area or areas of consumption and key performance indicators – KPIs).
• Create an Approval Board to encourage group involvement and oversight in decision making process (management practices).
• Implement a comprehensive environmental, energy, and/or asset management plan (that can include near-term as well as long-term actions) as part of a holistic strategy and management approach and foster proactive commitment to sustainability and efficiency within the given facility.

1.3Metering and Metrics

1.3.1Importance

The ability to measure energy use and identify factors that can impact overall operations is a prerequisite to identifying and justifying improvements. Many data centres (and users) currently have little or no capabilities to measure energy use, environmental impacts, or other parameters; some do not even have a separate utility meter or bill and therefore lack the ability to monitor and finetune many aspects or impacts of their operations, of which energy and cooling are two of the most significant. ^{Footnote 13} With respect to energy use and operational optimization, it can be easier to address these while services and equipment are specified and installed in new data centres. However, there are also substantial opportunities to achieve energy and cost reductions from existing services and physical footprint reductions starting with establishing a more detailed understanding of various performance metrics through metering and other measurements. The specific action for this section is presented in Figure 4.

Jump to the Periodic Reports on Standardized Key Performance Indicators section Jump to the Temperature and Humidity Readings section Jump to the Cooling System Data Collection section Jump to the Improve Visibility and Granularity section Jump to the Business Relevant Dashboard section

1.3.2Specific Actions and Potential Impacts

Periodic Reports on Standardized Key Performance Indicators

Consider producing periodic reports on standardized key performance indicators (KPIs), including energy consumption and environmental metrics. KPIs such as power usage effectiveness (PUE), energy reuse effectiveness (ERE), carbon usage effectiveness (CUE), water usage effectiveness (WUE), IT Equipment Energy Efficiency for Servers (ITEEsv), IT equipment utilization savings value (ITEUSV), cooling efficiency ratio (CER), and renewable energy factor (REF) can be utilized for this purpose, as detailed in Subsection 1.3.3 below. An automated energy and environmental reporting console can serve to monitor energy use and enhance capability. Averaged instantaneous KPIs related to IT workloads can also be reported. ^{Footnote 14} The table below provides links to websites with more information or use cases on these metrics.

By implementing a system for periodic reporting, energy consumption and environmental impacts can be effectively monitored, which enables the identification of optimization areas leading to potential cost savings and reduced environmental footprints. ^{Footnote 15} An example of Navius Research’s Canada Energy Dashboard is included in Figure 5 below.

Temperature and Humidity Readings

Consider obtaining temperature and humidity data for reporting purposes and to assess trends. These measurements should be taken regularly, preferably during peak load periods. Implementing automated daily readings is recommended alongside automated hourly readings to enable a comprehensive evaluation of how temperature and humidity vary in response to IT workload fluctuations. Temperature and humidity readings can be taken using building automation systems (BAS), energy management systems, indoor environment data management platforms, and more.

Continuous monitoring of temperature and humidity levels can help identify potential equipment malfunctions or environmental issues early on, allowing for proactive maintenance to prevent costly downtimes and equipment failures. It also serves to identify areas of inefficiency and empowers organizations to make informed decisions regarding energy management strategies, resource allocations, and infrastructure investments.

Cooling System Data Collection

Collecting data from computer room air conditioning/computer room air handler (CRAC/CRAH) units on supply and demand air temperature enables data centre operators to optimize energy usage and ensure optimal performance. Also, collecting data on the use of free cooling is important for the data centre using it. This implies collecting data on the duration and energy consumption of mechanical refrigeration and compressor-based cooling systems compared to the utilization of free cooling.

Understanding the performance of cooling systems and their dependence on free cooling ensures constant temperature control and minimizes the risk of equipment overheating.

Improve Visibility and Granularity

Consider installing meters at the distribution board level (submetering) to monitor the amount of mechanical and electrical energy consumed. Enhancing the ability to see and understand exactly how energy is being used at different points in the facility is recommended.

Improved visibility and granularity enable better management and optimization of energy usage.

Business Relevant Dashboard

Develop useful business-specific metrics and potentially create a business-relevant dashboard. This dashboard provides a centralized and easily accessible platform for monitoring and analyzing important efficiency-related data. KPIs such as PUE, ERE, and CUE can be utilized for this purpose, as detailed in Subsection 1.3.3 below. Other specific measures can be considered, such as cost analysis of energy usage, potential savings opportunities, or energy consumption per unit of data processed or stored. These metrics are selected and defined based on the specific priorities, strategies, and objectives of businesses rather than relying on generic or standardized measurements.

The efficiency dashboard should be designed to provide insights and facilitate decision making aimed at improving energy efficiency and reducing overall energy consumption. It should also be customizable to meet the specific needs and preferences of businesses.

It should be noted that priorities and options may vary across Canada in terms of emission reductions, overall facility impacts, and fuel choices for electricity generation. Some of the tools presented below may be better than others in terms of providing insight regarding energy efficiency versus overall facility emissions and impacts. Where appropriate, these considerations should be part of discussions of facility committees on optimization and overall impact reductions.

1.3.3Tools and Metrics

This subsection lists currently available tools and metrics that are used by the DC industry to measure various aspects of operations. For the tools and metrics to be meaningful, it is important to have a fixed system boundary of the given data centre. Total energy consumption can be measured using the ISO/IEC 30134 standard.

Power Density (PD)

Power density (PD) refers to the electricity demand in kilowatts (kW) of all data centre end uses divided by either the data room floor area (in kW/ft² or kW/m²) or the number of server racks in the facility (in kW/rack). PD is calculated as follows:

• Power Demand includes the power consumed by the enterprise data centre facility.
• Data Room Area represents the total floor area specifically allocated for housing IT equipment within the data centre facility.
• Number of Racks represents the total count of server racks installed and utilized within the data centre facility.

Power density is a measure of space efficiency. The more power per rack or by square feet, the greater the IT workload that can be accommodated in a reduced space. It is perhaps one of the simplest metrics to establish. Calculating power density is useful as it is a more reliable indicator of data centre electricity demand than the floor area alone. Additionally, this metric can be adapted to changes in technology and data centre configurations and can be easily recalculated.

However, it also has drawbacks. One of these is that surface area and power can be difficult to define, which can be solved by establishing a robust calculation methodology. PD in kW/ft² has been more commonly used for a long time, but PD in kW/rack is increasingly favoured. The problem with kW/ft² is that it provides no indication of power variations within a population of computer cabinets. Furthermore, it is not clear how this metric is used in data centres whose growth plans evolve. ^{Footnote 16} In addition, PD is perhaps considered too simplistic as it does not capture the complexity of data centre operations, including variations in workload types, equipment energy efficiency, and other factors that can influence overall data centre performance and efficiency.

Therefore, this metric should not be used as a benchmark as it does not indicate performance. However, it can be useful for assessing DC space utilization.

Power Usage Effectiveness (PUE)

PUE is a ratio that quantifies how a data centre uses electrical energy to power IT equipment with the aim of identifying and reducing energy wastage. PUE ranges from a minimum value of 1.0 (most efficient) to a maximum value of infinity (less efficient). The PUE metric is unitless and is calculated as follows:

• Total Facility Energy includes the energy consumed by the enterprise data centre facility, including cooling systems, lighting, and other non-computing loads.
• IT Equipment Energy represents the energy consumed by the actual computing equipment such as servers, storage devices, and networking gear.

To calculate PUE, the ISO/IEC WD 30134-2 standard: Information technology – Data centres – Key performance indicators – Part 2: Power usage effectiveness can be referenced.

PUE has notable limitations. It fails to benchmark IT energy consumption against IT production, potentially overlooking inefficiencies within IT equipment itself, such as virtualization. Additionally, PUE does not recognize factors like energy generation or waste heat recovery and lacks geographic adjustments, leading to variations in values for data centres in different climates.

Despite these drawbacks, PUE offers a simple ratio of IT equipment energy to total energy use. It proves valuable primarily for assessing cooling system performance.

According to the Uptime Institute Global Data Centre Survey 2023, the global average PUE is 1.58. ^{Footnote 17} This KPI is the one most reported by data centres. A total of 71% of data centres report their PUE according to the Uptime Institute Global Data Centre Survey 2023. According to an NRCan data centre market assessment, around 22% of data centres report their PUE in Canada.

Data Centre Infrastructure Efficiency (DCiE)

The data centre infrastructure efficiency (DCiE) metric serves as the reciprocal of PUE. The DCiE is unitless and is calculated as follows:

While both metrics essentially measure the same thing, they offer different perspectives on energy allocation within a data centre. For instance, if a data centre’s PUE is 3, the corresponding DCiE would be 1/3, which translates to 33%, indicating that the IT equipment consumes 33% of the total power in the data centre.

Energy Reuse Factor (ERF) and Energy Reuse Effectiveness (ERE)

When energy is reused within a data centre, it has the potential to drive the PUE below 1. However, to accurately account for reused energy, two metrics have been introduced.

The first metric, known as the energy reuse factor (ERF), serves to evaluate the proportion of energy that is reused. The ERF ranges from a minimum value of 0, which indicates that no energy is reused, to a maximum value of 1, which indicates that all energy brought into the data centre is repurposed. The ERF metric is unitless and is calculated as follows:

• Reuse Energy includes the energy that is reused elsewhere in the facility, the total energy brought into the data centre from another location, and the energy from the data centre that is used in another location and reduces the energy the site would need to buy, procure, or generate.
• Total Facility Energy includes the energy consumed by the enterprise data centre facility, including cooling systems, lighting, and other non-computing loads. It is the same as the numerator of PUE.

To calculate the ERF, the ISO/IEC 30134-6:2021- Information technology – Data centre key performance indicators – Part 6: Energy Reuse Factor (ERF) can be referenced.

The energy reuse effectiveness (ERE) metric is designed to assess the effectiveness of energy reuse within a data centre. A higher ERE value indicates more energy reuse, meaning less energy is wasted or lost in the data centre operation process. The ERE metric is unitless and is calculated as follows:

The ERE value can also be obtained with the ERF and PUE as follows:

If the reused energy is zero, then the ERF is zero, and ERE = PUE.

Carbon Usage Effectiveness (CUE)

The CUE metric serves to evaluate the emissions caused by a given data centre. CUE is similar to PUE, but instead of using total facility power consumption, it uses the total CO₂ emissions and divides it by IT equipment energy consumption. The CUE is expressed as kg of CO₂ emitted per kWh of energy consumed [kg CO₂/kWh] and is calculated as follows:

• Total CO₂ Emissions include carbon emissions resulting from the use of electricity from the grid, for which data can be obtained from published sources specific to the region, as well as emissions associated with onsite electricity generation, for which estimates can be made.
• IT Equipment Energy represents the energy consumed by the actual computing equipment such as servers, storage devices, and networking gear.

Another way to calculate the CUE metric is to use PUE and incorporate electricity carbon emissions factor:

To calculate the CUE, the ISO/IEC 30134-8:2022 Information technology – Data centre key performance indicators – Part 8: Carbon usage effectiveness (CUE) can be referenced.

The CUE offers a comprehensive view of a data centre’s carbon footprint. When used alongside PUE, it provides a more accurate assessment of facility environmental impacts.

Water Usage Effectiveness (WUE)

WUE serves to assess water usage in a data centre. The WUE metric is expressed as litres of water consumed per kilowatt-hour (kWh) of equipment power and is calculated as follows:

• Annual Water Usage includes the overall use of water in data centre systems, including water loops, onsite evaporation for cooling, humidification, or energy production. It does not include domestic water usage.
• IT Equipment Energy represents the energy consumed by the actual computing equipment such as servers, storage devices, and networking gear.

To calculate WUE, the ISO/IEC 30134-9:2022 Information technology – Data centre key performance indicators – Part 9: Water usage effectiveness (WUE) can be referenced.

This metric provides insights into data centre water efficiency practices and environmental impacts related to water consumption. A lower WUE value can mean a more efficient way to use water resources, but it can also mean inefficient IT equipment. The average data centre has WUE of 1.8 L per 1 kWh according to a report by the US Department of Energy. ^{Footnote 18}

IT Equipment Energy Efficiency for Servers (ITEEsv)

The IT equipment energy efficiency for servers can be used to assess the energy efficiency of servers in data centres. The metric is expressed in tasks/kWh, and can be calculated as follows:

• Number of Tasks Performed refers to the tasks a server can handle within an hour.
• Energy Consumed by the Server represents server hourly energy consumption.

The aim is to measure the amount of computing work (such as processing data, serving applications, etc.) that can be performed per unit of energy consumed by a given server. This metric is focussed on potential rather than actual operational performance.

To calculate the ITEEsv, the ISO/IEC 30134-4:2017 Information technology – Data centres – Key performance indicators – Part 4: IT equipment energy efficiency for servers (ITEEsv) can be referenced.

IT Equipment Energy Utilization for Servers (ITEUSV)

The IT equipment energy utilization for servers (ITEUSV) describes the utilization of server equipment in data centres. First, the ITEUSV(t) is calculated, which represents the average CPU utilization of all servers in a data centre at time t:

• The numerator represents the sum of central processing unit (CPU) utilization ratio of a server at time t.
• Number of Servers Running is the number of servers in a data centre running at time t.

Then, ITEUSV can be calculated as follows:

• a is the number of ITEUSV(t) measurement intervals over a year (all intervals should be the same length).
• ITEUSV(t) is the average CPU utilization of all servers in a data centre at time t.

To calculate the ITEUSV, the ISO/IEC 30134-5:2017 Information technology – Data centres – Key performance indicators – Part 5: IT equipment utilization for servers (ITEUSV) can be referenced.

Cooling Efficiency Ratio (CER)

The cooling efficiency ratio (CER) serves to assess the efficiency of cooling operations within a data centre. The CER metric is unitless and can be calculated as follows:

• Heat Removed represents the amount of heat removed from the data centre by cooling systems.
• Energy Consumed by the Cooling System includes the annual energy consumed by all cooling systems in the data centre.

The CER is not used for comparing cooling systems across different data centres but rather serves as an internal benchmark to track efficiency improvements within a specific facility.

To calculate the CER, the ISO/IEC 30134-7:2023 Information technology – Data centre key performance indicators – Part 7: Cooling efficiency ratio (CER) can be referenced.

Renewable Energy Factor (REF)

The renewable energy factor (REF) is a metric to quantify the amount of renewable energy controlled and produced by a data centre. The REF metric is unitless and can be calculated as follows:

• Renewable Energy includes all renewable energy for which the data centre owns the legal rights. It is the renewable energy owned and controlled by the data centre.
• Total Facility Energy includes the energy consumed by the enterprise data centre facility, including cooling systems, lighting, and other non-computing loads.

The REF varies from 0, which represents no renewable energy produced on site, to 1, which means 100% of energy is renewable and produced on site.

To calculate the REF, the ISO/IEC 30134-3:2016 Information technology – Data centre key performance indicators – Part 3: Renewable Energy Factor (REF) can be referenced.

Summary of Key Energy and Environmental Performance Indicators (KPIs)

Metric	Equation	Standard
Power Density	$\mathbf{PD = \frac {\mathrm{Power Demand}~\left [ kW \right ]} {\mathrm{Data Room Area}~\left [ ft^{2}\right ]}$	–
Power Usage Effectiveness	$\mathbf{PUE = \frac {\mathrm{Total Facility Energy}~\left [ kWh \right ]} {\mathrm{IT Equipment Energy}~\left [ kWh \right ]}}$	ISO/IEC WD 30134-2
Data Centre Infrastructure Efficiency	$\mathbf{DCiE = \frac {1} {PUE} = \frac {\mathrm{IT Equipment Energy}~\left [ kWh \right ]} {\mathrm{Total Facility Energy}~\left [ kWh \right ]}}$	ISO/IEC WD 30134-2
Energy Reuse Factor	$\mathbf{ERF = \frac {\mathrm{Reuse Energy}~\left [ kWh \right ]} {\mathrm{Total Facility Energy}~\left [ kWh \right ]}}$	ISO/IEC 30134-6
Carbon Usage Effectiveness	$\mathbf{CUE = \frac {Total~CO_{2}~Emissions~\left [ kg~CO_{2} \right ]} {\mathrm{IT Equipment Energy}~\left [ kWh \right ]}}$	ISO/IEC 30134-8
Water Usage Effectiveness	$\mathbf{WUE = \frac {\mathrm{Annual Water Usage}~\left [ L \right ]} {\mathrm{IT Equipment Energy}~\left [ kWh \right ]}$	ISO/IEC 30134-9
IT Equipment Energy Efficiency for Servers	$\mathbf{ITEEsv = \frac {\mathrm{Number of Tasks Performed}~\left [ Tasks \right ]} {\mathrm{Energy Consumed by the Server}~\left [ kWh \right ]}$	ISO/IEC 30134-4
IT Equipment Energy Utilization for Servers	$\mathbf{ITEUsv = \displaystyle{\sum \left [} \frac {\sum CUS_{i}(t)~\left [ % \right ]} {\mathrm{Number of Servers Running}} \displaystyle{\right ]}$	ISO/IEC 30134-5
Cooling Efficiency Ratio	$\mathbf{CER = \frac {\mathrm{Heat Removed}~\left [ kWh \right ]} {\mathrm{Energy Consumed by the Cooling System}~\left [ kWh \right ]}}$	ISO/IEC 30134-7
Renewable Energy Factor	$\mathbf{REF = \frac {\mathrm{Renewable Energy}~\left [ kWh \right ]} {\mathrm{Total Facility Energy}~\left [ kWh \right ]}}$	ISO/IEC 30134-3

Other metrics

Server Energy Effectiveness Metric (SEEM)

The server energy effectiveness metric (SEEM) is a standardized measure to assess the energy efficiency of servers in data centres. It utilizes the SPEC SERT™v2 framework to evaluate server capabilities in running standardized workloads while considering server energy consumption. SEEM offers a standardized approach for quantifying server energy efficiency, enabling data centre operators to make informed decisions to optimize energy usage and reduce environmental impacts.

To calculate the SEEM, the ISO/IEC 21836. Information technology – Data centres – Server energy effectiveness metric (SEEM) can be referenced.

Benchmark Energy Factor (CSA Group™)

The benchmark energy factor (BEF) was introduced by the CSA Group™ as part of the CSA C510:21 standard that concerns the assessment of data centre energy performance. It serves as a valuable tool for data centre operators and managers to assess how efficiently their facilities use energy in comparison to established benchmarks or theoretical ideals. It is applicable to data centres with a minimum IT load of 100 kW and where the organization has control over the equipment, excluding colocation data centres.

The BEF is a dimensionless factor computed by considering various components of data centre energy consumption, including total electrical usage, cooling requirements, lighting, and other non-computing loads. A BEF value below 1 indicates that the data centre operates with higher energy efficiency than the benchmark or ideal, signifying exemplary energy efficiency performance.

Mechanical Load and Electrical Loss Components (ASHRAE)

The mechanical load component (MLC) and electrical loss component (ELC) are metrics introduced by ASHRAE in the 2016 Standard 90.4. This standard separated mechanical and electrical system efficiency, enabling data centre operators to have an alternative compliance path if either the mechanical or electrical system fails to meet the minimum efficiency requirements set by the standard.

Mechanical Load Component (MLC)

The MLC is a metric used to assess the energy efficiency of a data centre by quantifying the power required for cooling systems, fans, pumps, and heat rejection equipment, divided by the designed power capacity of the IT equipment. This measure helps account for the impact of partial load conditions on cooling systems by analyzing their energy consumption at various load levels (25%, 50%, 75%, and 100% of the IT load).

Electrical Loss Component (ELC)

The ELC serves to evaluate the efficiency of data centre electrical systems by identifying and measuring losses within the electrical distribution segments. It covers the incoming electrical service segment, which includes power supply through transformers and uninterruptible power supply (UPS) equipment, the UPS segment itself, and the IT equipment distribution segment from the UPS output to the connection points of IT equipment. This calculation is used to demonstrate the minimum electrical design efficiency by considering the most critical parts of each segment.

While MLC and ELC are good indicators for estimating the potential for energy performance improvement, they require precise information about mechanical and electrical systems.

1.4Management

1.4.1Importance

Data centre management involves overseeing and controlling all aspects of data centre operations and maintenance. This includes strategic planning to align data centre operations with business objectives as well as management of the physical infrastructure and IT equipment such as servers and storage, networking, cooling, and power systems.

Capacity planning and growth management are essential to anticipate and meet current and future IT demand and to do so in an energy efficient way. Overall, effective data centre management involves a holistic approach to coordinating and optimizing the various functions to ensure the reliability, security, and efficiency of an organization’s IT infrastructure. The specific actions are presented in Figure 6 below.

Jump to the Create an Approval Board section Jump to the Conduct an Audit section Jump to the Implement Service Charging Models section Jump to the Conduct a Life Cycle Assessment section Jump to the Implement Environmental, Energy, and Asset Management Plans section Jump to the Ensure High Quality Site Documentation section Jump to the Invest in Training and Development section

1.4.2Specific Actions and Potential Impacts

Create an Approval Board

Establish an approval board (or review committee) made up of representatives from several departments, including senior management, IT, monitoring and evaluation (M&E) engineering, applications/software, and procurement. This committee, which can be informal, should participate in addressing important decisions to ensure a thorough understanding of the implications and buy-in on effective solutions. Requiring the committee’s approval for significant data centre initiatives establishes a structured framework that mitigates the risks associated with unilateral decision making, thus fostering accountability and alignment with organizational objectives.

The potential impacts of implementing such a structure include heightened transparency and accountability in decision-making processes. With representatives from key departments involved, decisions are more likely to reflect a holistic understanding of data centre operational, financial, and strategic considerations.

Conduct an Audit

Performing a thorough audit of existing equipment is an essential step in maximizing the potential of data centre resources before considering new investments. By systematically assessing all components, organizations can identify areas for optimization, consolidation, and aggregation, thereby exploiting untapped capacity and avoiding unnecessary expenses.

The implementation of an Information technology infrastructure library (ITIL) type configuration management database and service catalogue can be considered. ITIL is a set of practices for IT service management (ITSM) that is focused on aligning IT services with the needs of businesses. Also, the international standard for asset management, ISO 55000 could be a useful reference to support ITIL implementation.

By proactively managing existing assets and making informed decisions based on audit findings, organizations can optimize resource utilization, minimize costs, and improve the overall efficiency of the data centre environment.

Implement Service Charging Models

This consists of adopting pricing structures and tariffs that incentivize best practices and enhance energy efficiency to encourage responsible resource consumption while discouraging wasteful practices. The selected pricing model depends on the types of data centre or service provider as well as the services provided (internal or external).

This service charging approach not only fosters a culture of accountability, but also motivates stakeholders to adopt strategies that prioritize energy conservation. Ultimately, leveraging service charging models empowers organizations to optimize resource allocation, minimize costs, and achieve long-term sustainability goals within the data centre ecosystem.

Conduct a Life Cycle Assessment

Implementing a comprehensive plan for life cycle assessment (LCA) aligned with ISO 14040 and ISO 14044 standards represents a significant step toward reducing the overall facility carbon footprint and enhancing sustainability within data centre operations.

By conducting a thorough assessment of the environmental impacts associated with every stage of a product life cycle, from raw material extraction to disposal, organizations gain valuable insights into areas where emissions can be minimized and sustainability improved.

By integrating LCA into strategic planning and decision-making processes, organizations can identify opportunities for emission reductions, resource conservation, and the adoption of greener technologies.

Implement Environmental, Energy, and Asset Management Plans

Implementing comprehensive environmental, energy, and/or asset management plans enables a proactive commitment to sustainability and efficiency within a facility. An environmental or energy management plan (EMP) is a site-specific plan developed to ensure that a facility is operated in an environmentally sustainable or energy efficient manner and understand the potential risks and impacts arising from the facility or site and the appropriate actions required to minimize those risks. ^{Footnote 19}

EMPs can be carried out using the ISO 14001, ISO 50001, or ISO 55000 standards. The adoption of ISO 14001 ensures that environmental risks are systematically identified, monitored, and mitigated, thus fostering a culture of environmental responsibility and compliance. Likewise, ISO 50001 facilitates the establishment of energy management systems, enabling organizations to set clear objectives, monitor performance, and continuously improve energy efficiency. Meanwhile, ISO 55000 provides a structured framework for asset management, ensuring that assets are strategically managed to achieve organizational objectives while minimizing costs and risks.

By integrating these management systems, data centre operators can enhance operational resilience, reduce operational costs, and drive continuous improvement in sustainability performance.

Ensure High Quality Site Documentation

Ensuring the availability of high quality, precise operation and maintenance (O&M) manuals, as-built records, commissioning records, schematics, and single-line diagrams serves to support the maintenance of all installed infrastructure and equipment at the intended design specifications.

High quality documentation facilitates effective maintenance by providing technicians with accurate information about equipment and systems. This enables the utilization of energy efficient functions and ensures that any changes to settings or equipment are properly recorded and updated. By maintaining accurate records, data centres minimize lack of communication and optimize energy usage.

Invest in Training and Development

A comprehensive training methodology should be established and implemented for all personnel involved in data centre management and energy efficiency to ensure competence and expertise. It is essential to provide the data centre manager and any appointed energy manager with appropriate training to effectively fulfill their roles in promoting energy efficiency and adhering to best practices. Similarly, it is crucial to educate both IT and facility personnel about energy efficiency best practices to support organizational strategies aimed at reducing IT energy consumption.

This practice eliminates departmental silos and fosters a common understanding of data centre operations and management, encouraging collaboration and efficiency.

1.5Hardware

1.5.1Importance

In a data centre, hardware refers to the physical components that make up the infrastructure necessary for storing, processing, and transmitting data. This includes a wide range of equipment such as servers, storage devices, networking devices, power distribution units (PDUs), UPS, and various other components. Cooling systems are more specifically discussed below.

Implementing best practices for data centre hardware is key to optimizing energy use. By prioritizing the process of selecting and deploying energy efficient hardware, data centres can significantly reduce energy consumption and the results costs. These practices also improve overall operational efficiency and resource utilization in the data centre environment.

In addition to energy consumption, the entire equipment life cycle should be taken into consideration. The materials used in the design, the end of life of equipment, and the disposal method are all elements that are factored into total equipment energy consumption in addition to greenhouse gas emissions. Emissions emitted outside facility operation, i.e. the life cycle stages before and after the equipment is installed in the data centre, are known as Scope 3 emissions. Scope 3 emissions can exceed those of the use phase itself, so they should not be neglected.

An informed choice of hardware is important not only to reduce energy consumption during equipment use, but also to reduce Scope 3 emissions. The specific actions are presented in Figure 8 below.

Jump to the Power Management and Control of Equipment Energy Use section Jump to the Removal of Unused Equipment and Consolidation of Existing Services section Jump to the Management of End-of-Life Equipment section Jump to the Selection of Lower Power Storage Devices section Jump to the Provision only for the Actual IT Power Usage Required section Jump to the High Efficiency Uninterruptible Power Supply (UPS) section Jump to the Operating Temperature and Humidity Range section

1.5.2Specific Actions and Potential Impacts

Power Management and Control of Equipment Energy Use

As hardware is deployed, it is important to change the deployment process to include the enabling of power management features. These power management features include BIOS (basic input/output system), operating systems, and driver settings. These functions enable the control and optimization of the energy consumption of hardware components such as processors, graphics cards, hard disks, and so on. For example, they can include sleep mode for unused peripherals.

However, using these features can cause issues with balancing the phases of electrical power distribution systems and can have negative impacts when redundant system components need to be removed from service for maintenance and repairs. It is therefore important to strike a balance between efficiency and availability requirements.

Selecting equipment with built-in energy control features facilitates external management of energy consumption and mitigates environmental impacts.

Removal of Unused Equipment and Consolidation of Existing Services

Consider removing and decommissioning all unnecessary equipment and regularly reviewing installed equipment to ensure ongoing necessity. Servers or services that cannot be decommissioned but are not used regularly should be virtualized and disk images archived on low-energy media. Additionally, operational services that do not fully utilize their hardware resources should be consolidated using resource-sharing technologies to optimize the efficiency of physical resources across servers as well as storage and networking devices.

This practice yields significant benefits including cost savings from reduced equipment purchasing, maintenance, and energy consumption.

Management of End-of-Life Equipment

Removed equipment should be disposed in line with the principles of the circular economy to minimize environmental impacts and maximize utility. Prioritize evaluating unused equipment to determine if it can be repurposed, sold, or recycled as part of the decision-making process. Explore opportunities for internal or external equipment reuse. If equipment cannot be reused, ensure proper recycling. Look for certified recycling centres with the necessary facilities to safely process electronic components.

Responsible end-of-life equipment management promotes a circular economy by enabling the reuse and recycling of hardware resources. This approach also helps reduce Scope 3 emissions by minimizing electronic waste and promoting sustainable equipment use.

Selection of Lower Power Storage Devices

It is crucial to prioritize energy efficiency when choosing storage hardware. This evaluation process should encompass factors like achieved performance, storage volume per watt, levels of data protection, availability, and recovery capacity required to meet service level agreements. Lower power storage devices also generate less heat, which means fewer resources are required for cooling. References to ENERGY STAR® specifications for data centre storage provide valuable guidance in this process.

Prioritizing lower power storage hardware reduces operational costs, minimizes environmental impacts, and ensures alignment with performance and data protection requirements, thereby enhancing overall sustainability and resource optimization.

Provision only for the Actual IT Power Usage Required

The provision of power and cooling should be aligned with the planned power draw of IT equipment as configured rather than the power supply unit (PSU) size or nameplate rating. The power draw of IT equipment can vary significantly based on factors such as workload, configuration settings, and hardware utilization. By aligning provisioning with the actual power consumption of the equipment in operation, data centre operators can ensure that resources are efficiently allocated.

This approach prevents oversizing of electrical infrastructure, ensuring efficient operation even at low loads. Adjustments to provisioning may be necessary with upgrades or changes in IT equipment performance.

High Efficiency Uninterruptible Power Supply (UPS)

For static AC UPS systems, prioritize selecting energy efficient UPS systems compliant with the IEC 62040 series to ensure they meet environmental operating conditions. UPS systems should perform as rated within specific ambient temperature and humidity ranges (specifically temperature: 0°C to +40°C; relative Humidity: 20% to 80%). Additionally, consider referencing and utilizing the current U.S. EPA ENERGY STAR specifications for UPS.

Operating Temperature and Humidity Range

Excess heat and humidity in data centres can damage devices and equipment, causing them to malfunction and stop working. The procurement of equipment capable of operating within ASHRAE Class A2 should be considered. If this is not possible, then equipment supporting at least ASHRAE Class A1 may be procured. Additionally, consider equipment that operates within a broader range of intake temperature and humidity levels as defined in ASHRAE Class A4 (which is broadly equivalent to ETSI EN 300 019-1-3 Class 3.1).

By respecting the operating temperature and humidity range, optimal equipment performance can be achieved. In addition, equipment that operates over a wider range of inlet temperature and humidity levels, as defined in ASHRAE Class A4, allows greater flexibility under varying environmental conditions.

1.6Data Management Part 1: Enterprise Data Centres

1.6.1Importance

Data management is an organization’s practice of collecting, organizing, protecting, and storing data so they can be handled and used appropriately. Today’s digital transformation means companies and organizations have access to and are responsible for more data than ever before. Data serve as a foundation of intelligence and need to be managed to meet organizational objectives. To ensure workers have the right data and guidelines for decision making, organizations must invest in processes that address data visibility, reliability, security, and scalability. Data management is also a necessary step in effective data analysis.

Data management and handling requirements differ depending on the type of organization handling the data (for example governments, NGOs, businesses, or education entities) as each is subject to sector-specific requirements. ^{Footnote 20} The specific actions are presented in Figure 9 below.

Jump to the Select or Develop Efficient Software section Jump to the Deploy Virtualization Technologies section Jump to the Eliminate Traditional 2N Hardware Clusters section Jump to the Reduce Total Data and Storage Volume section Jump to the Optimise Server, Network, and Storage Utilization section

1.6.2Specific Actions and Potential Impacts

Select or Develop Efficient Software

Pertaining to software or specific applications, efficiency is the amount of useful work a software system produces per unit of energy consumed. For example, the amount of data a web server sends to a client is the useful work, and the energy consumed is the energy the server draws to accomplish the task. On the other hand, software development efficiency refers to how software engineering teams improve efficiency through practices such as implementing continuous integration and delivery pipelines, adopting agile methodologies, and the use of tools such as code editors, debuggers, and performance profilers to reduce errors.

Consider making energy efficiency a priority in the software selection process or in the software development process. Software performance improvements should also be encouraged. The concept of energy efficient software incorporates several key characteristics aimed at minimizing energy consumption during operation, such as optimized algorithms, efficient resource management, and idle state management.

This practice is aimed at reducing energy consumption above all else, which saves money and reduces environmental impacts.

Deploy Virtualization Technologies

Deploying virtualization technologies involves establishing protocols that require the approval of senior business managers for any new service that requires dedicated hardware instead of utilizing shared resource platforms. This policy applies across the server, storage, and networking components of a service.

By implementing such procedures, organizations ensure a rigorous evaluation process for resource allocation, promoting the efficient utilization of hardware resources and avoiding unnecessary hardware purchases. This approach not only optimizes resource allocation, but also reduces infrastructure costs and enhances scalability by effectively leveraging virtualized environments.

Eliminate Traditional 2N Hardware Clusters

Eliminating traditional 2N hardware clusters involves replacing active/passive server hardware clusters with faster recovery approaches. This transition is guided by assessing the business impact of short service incidents for each deployed service. By adopting this strategy, organizations streamline their infrastructure and thereby reduce redundancy and improve resource utilization.

Reduce Total Data and Storage Volume

Organizations should implement robust data identification and management policies aimed at minimizing both the total volume of stored data and the number of data copies. Remove unnecessary data, and space saving functionalities like snapshots, data compression, and thin provisioning should be integrated into the storage subsystem where feasible.

This practice minimizes storage overhead and improves resource utilization, ultimately contributing to a more sustainable and cost-effective storage environment.

Optimise Server, Network, and Storage Utilization

Use available tools to measure the proportional use of overall network capacity, storage capacity, and processor capacity. This measure can be grouped either globally or by service/location. A basic level of reporting can be very informative and should take energy consumption into account. The definition of utilization may vary depending on factors such as available capacity and what is considered used.

Implementing this data centre best practice leads to optimized server, network, and storage utilization, ensuring that resources are used efficiently and reducing unnecessary energy consumption. This can result in cost savings, improved performance, and lower environmental impacts.

1.7Data Management Part 2: Colocation Providers

1.7.1Importance

The specific actions are presented in Figure 10 below.

Jump to the Deploy Virtualization Technologies section Jump to the Set Minimum Targets for the Utilization of IT Equipment section Jump to the Assess Impacts When Purchasing New Services section Jump to the Implement a Data Management Policy section Jump to the Separate Physical Data Storage Areas by Protection and Performance Requirement section Jump to the Reduce Total Data and Storage Volume section

1.7.2Specific Actions and Potential Impacts

Deploy Virtualization Technologies

Virtualization technology serve to virtually represent servers, storage, networks, and other physical machines. Virtual software mimics the functions of physical hardware to run multiple virtual machines simultaneously on a single physical machine. Virtualization removes all these limitations by abstracting physical hardware functionality into software, allowing for the management and utilization of hardware infrastructure like an application on the web. Deploying virtualization technologies involves establishing protocols that require the approval of senior business managers for any new service that involves new dedicated hardware instead of utilizing shared resource platforms. This policy applies across servers, storage, and networking components of the service.

By implementing such procedures, organizations ensure a rigorous evaluation process for resource allocation, promoting the efficient utilization of hardware resources and avoiding unnecessary hardware purchases. This approach not only optimizes resource allocation, but also reduces infrastructure costs and enhances scalability by effectively leveraging virtualized environments.

Set Minimum Targets for the Utilization of IT Equipment

For servers, networks, and storage, calculate and monitor the utilization levels. Tools can be used, such as IT equipment utilization for servers (ITEUSV) or server energy effectiveness metrics (SEEMs). Determine minimum targets for IT equipment utilization to ensure they are optimized. It should be noted that minimum targets may vary depending on the services and other conditions; thus, it may be necessary to have benchmarks within a particular industry for comparison. This practice maximizes the use of IT equipment and is more energy efficient.

Assess Impacts When Purchasing New Services

When making decisions about purchasing new cloud services or evaluating a cloud strategy, assessing impacts on equipment utilization is essential to prevent the overutilization of existing resources and avoid inefficiencies. This involves determining how such decisions will affect the amount of hardware resources needed for tasks like data storage and workload processing as well as anticipating changes in energy consumption. By quantifying potential increases or decreases in energy usage, organizations can better anticipate their hardware needs and optimize resource allocation. It is also important to consider any supplementary cooling requirements that may arise due to changes in equipment utilization.

Ultimately, the goal is to ensure that the organization’s IT infrastructure remains efficient and effectively supports its operations while minimizing unnecessary energy consumption and associated costs.

Implement a Data Management Policy

It is important to have a data management policy to establish a common understanding among users. The policy should define which data are to be retained, for how long, and at what level of protection. The impact of data retention on energy consumption needs to be assessed.

Standards like ISO/IEC 27001:2022 - Information security, cybersecurity and privacy protection or ISO 15489-1:2016 – Information and documentation – Records management can be used.

In the NRCan market assessment, it has been observed that businesses and institutions are increasingly inclined to place their servers and data in colocation centres that provide a level of protection and infrastructure management. So, data management is very important for colocation providers as they have a wide range of customers with different protection and performance requirements. The various organizations requiring different levels of protection are:

• Governmental organizations
• Commercial enterprises
• Nonprofit organizations
• Healthcare institutions
• Educational institutions
• Financial institutions

Some of these organizations use additional security categories as it would be too expensive and energy intensive to set all data to maximum security levels. Some organizations use the following categories to classify their data:

• Restricted: Restricted data deserves the highest level of security.
• Private: A reasonable level of security should be applied to private data.
• Public: Breach of public data exposes organizations to little or no risk.

Implementing a data management policy serves to establish clear guidelines and procedures for handling data effectively. Freeing up safe due to unnecessary data saves energy by reducing the storage load and minimizing the energy requirements associated with IT infrastructure. In addition, ensuring adequate data protection helps reduce the risk of data breaches or losses, which can have financial, legal, and reputational consequences for organizations.

Separate Physical Data Storage Areas by Protection and Performance Requirement

Organizations should establish a tiered storage environment that offers varying combinations of performance, capacity, and resilience. Clear guidelines on storage tier usage should be implemented and communicated to users, outlining different service levels for performance and availability.

Protection and performance requirements vary depending on the type of organization. For example, a financial institution may prioritize high-performance storage tiers with redundant backups to ensure data integrity and availability for critical transactions. On the other hand, a research institution may prioritize cost-effective storage solutions with lower performance tiers for archival purposes, thus focusing more on capacity and long-term retention.

This practice ensures that critical applications have the necessary performance levels while maximizing cost savings for less critical data, ultimately improving overall operational efficiency.

Reduce Total Data and Storage Volume

Organizations should implement robust data identification and management policies aimed at minimizing both the total volume of stored data and the number of data copies. Users should be encouraged to remove unnecessary data, and space-saving functionalities like snapshots, compression, and thin provisioning should be integrated into the storage subsystem where feasible.

This practice minimizes storage overhead and improves resource utilization, ultimately contributing to a more sustainable and cost-effective storage environment.

1.8Cooling

1.8.1Importance

The most essential mission for any data centre is to maintain workflows and deliver the necessary services at all levels. A related operational task is always maintaining the necessary environmental conditions suitable for information technology equipment (ITE) functions. Meeting these objectives requires the removal of heat produced by ITE and transferring this heat to elsewhere because high temperatures and humidity levels can damage IT equipment, causing them to fail.

Generally, heat removal for data centre cooling is collectively referred to as equipment, tools, systems, techniques, and processes that work together to maintain ideal temperatures and humidity levels within a facility. Data centre cooling is the process of removing excess heat from equipment and operations within a space. Data centre cooling is one of the largest energy uses in such facilities. Typically, cooling and ventilation systems account for between 30% and 55% of data centre power consumption and thus represents a significant improvement area.

1.8.2Specific Actions and Potential Impacts

According to the Uptime Institute, the data centre cooling process can be further grouped into server (or IT equipment) cooling and space cooling. To identify suitable actions and their potential impacts, it is necessary to review the cooling strategies currently in use for each group and practices that can improve their efficiencies. Each is discussed separately below. ^{Footnote 21}

1.8.3Server (IT Equipment) Cooling

ITE generates heat as the electronic components therein use electricity to operate. In air-cooled equipment, this heat is transferred from electrical components to the air surrounding the server. Server cooling fans draw air across the internal components, representing the first stage of heat transfer. It should be noted that liquids can perform this function more efficiently than air; thus, some systems use liquids to absorb and carry heat away from the equipment. However, liquid-cooled systems require more complex cooling setups and are less commonly used.

The most significant practices for server and equipment cooling involve the containment and directing heated air so that cool server intake air is not mixed with heated server exhaust air. Containment can provide several benefits such as:

• Providing more consistent inlet air temperatures;
• Allowing the temperature of the supplied air to the ITE space to be raised, improving options for efficiency;
• Returning air at higher temperatures to the cooling equipment, which can make CRAC/CRAH units operate more efficiently;
• Enabling higher equipment density.

The specific actions for server cooling are presented in Figure 12 below. They contain practices that can be adopted during the design, retrofit, or operational stages.

Jump to the Design – Hot/Cold Aisle Containment section Jump to the Design and Retrofit – Contained Hot or Cold Air section Jump to the Cabinet Air Flow Management – Blanking Plates and Other Openings section Jump to the Ensure Adequate Vents in Cabinet Doors section Jump to the Equipment Segregation section

Design – Hot/Cold Aisle Containment

As the required air flow volumes of server room equipment have increased, it is necessary to ensure that equipment can share an air flow direction within the server cabinet (rack), in adjacent cabinets, and across aisles. The hot/cold aisle practice helps to align equipment air flow to create "aisles" between cabinets that are fed cold air from which the equipment draws intake air in conjunction with hot aisles to which all equipment exhausts air.

• In a cold aisle containment system, cooled air from air handlers is contained, while hot server exhaust air is allowed to return freely to the air handlers. Most cold aisle containment environments are used in conjunction with raised floor ducting air delivery. It is also possible to use cold aisle containment with another air delivery system such as overhead ducting. Typically, cold aisle containment works best in instances wherein the ITE system load does not vary widely.

  

  Source: Effect of under-floor blockages and perforated tile openings on the performance of raised-floor data centres by B. Fakhim, N. Srinarayana, M. Behnia and S. W. Armfield, School of Aerospace, Mechanical and Mechatronic Engineering University of Sydney, New South Wales 2006, Australia

• In a hot aisle containment system, exhaust hot air is contained and returned to the air handlers directly via a ceiling return plenum without mixing it with ambient air. In an optimized hot aisle containment environment, equipment operations and requirements determine how much cooled air is needed. As cool air is used and removed from the space by server fans, lower air pressure results in more air flow. Thus, the hot aisle setup allows for more flexibility in equipment layout, density, and loading than a cold aisle system.

Design and Retrofit – Contained Hot or Cold Air

In addition to hot and cold aisles for containment, there are also practices whereby the containment and separation of cold air from heated return air on the data floor are the objectives, including:

• Contained cabinet supply;
• Contained room return and room supply;
• Contained cabinet return, with or without cabinet chimneys.

The containment of air flow result in both an increase in cooling efficiency and a reduction in risks. Variable demand in services means that the air flow and heat output of IT devices may change rapidly due to power management and workload allocation tools and can be better managed through containment.

Cabinet Air Flow Management – Blanking Plates and Other Openings

Installing blanking plates or blanking panels to cover openings on racks where there is no equipment as well as to fill other air gaps can reduce the volume of hot air recirculating through cabinet gaps. This reduces instances of heated air being recirculated and consequently increasing intake air temperature and reducing efficiency.

The installation of aperture brushes (draught excluders) also helps to eliminate all air leakage possibilities in each cabinet. Areas to address include:

• Floor openings at the base of cabinets;
• Gaps at the sides, tops, and bottoms of cabinets between equipment or mounting rails;
• Gaps around the perimeter of cabinets.

Ensure Adequate Vents in Cabinet Doors

ITE cabinets with solid doors should be considered for replacement with perforated doors to ensure adequate cooling airflow where cooling ventilation is necessary. ^{Footnote 22} Solid doors restrict airflow, causing recirculation within cabinets and increasing air inlet temperatures. According to ISO/IEC 14763-2, a minimum of at least 66% perforated area is needed, and other sources recommend 80% perforation.

Equipment Segregation

Equipment (including IT, mechanical, and electrical equipment) installed in a data centre with substantially different environmental requirements and/or equipment airflow direction should be deployed in its own separate areas. It is easier and more preferable to provide separate environmental controls and areas for equipment with different environmental requirements. This helps to address the issue of the data centre cooling plant settings being constrained by the equipment with the most restrictive environmental range, thus limiting the efficiency of the enterprise data centre.

1.8.4Space Cooling

In air-cooled data centres, the heated air extracted from equipment can mix with other air in the space before being returned to the cooling system. Typically, cooling and circulation are provided by a CRAC or CRAH unit. ^{Footnote 23} The heated air releases a thermal charge within the CRAC/CRAH unit that removes the heat and delivers it elsewhere. In older legacy DC designs, the CRAC/CRAH units return the cooled air into raised floor plenums that deliver cooled air back into the ITE space and the process repeats. Today, it is a challenge for such a raised floor design alone because it requires further enhancements to meet demand due to increased heat of newer ITE and the efficiency of newer, denser data centre designs. Best practices for space cooling include scalable or modular cooling equipment installation and operation to allow for unnecessary equipment to be shut down. The specific actions for space cooling are presented in Figure 15 below.

Jump to the Shut Down Unnecessary Cooling Equipment section Jump to the Review Cooling Before IT Equipment Changes section Jump to the Define and Review a Cooling Strategy section

Shut Down Unnecessary Cooling Equipment

Where a DC facility is not yet fully populated or when space has been cleared for consolidation, it is recommended that fixed speed fan CRAC/CRAH units be turned off in the empty areas. However, this practice should not be applied in cases where CRAC/CRAH units are more efficient, are equipped with variable speed drives, and can be operated more efficiently at lower loads.

Review Cooling Before IT Equipment Changes

Depending on the equipment types and requirements, changes in IT equipment can significantly affect cooling needs. Thus, the availability of cooling, including opportunities to optimize cooling, should be reviewed before any IT equipment changes.

Define and Review a Cooling Strategy

DC operators and managers should periodically review IT equipment and cooling deployment systems against a defined strategy to assess cooling performance, identify improvement opportunities, and carry out necessary corrections.

1.8.5Cooling Plant

Cooling plants account for a major portion of the energy used in data centre cooling. They are also DC areas with the greatest variation in technologies and equipment. Regardless of technology, an economical option for cooling plants is the use of free cooling/economized cooling that takes advantage of cool ambient conditions to meet part or all facility cooling requirements so that the need for any other form of cooling such as mechanical cooling (including the use of compressors) is reduced, resulting in significant energy reductions. Free cooling can be retrofitted to many existing facilities.

In cooler and dryer climates (such as in many parts of Canada) and where increased temperature set points are used, there are more opportunities for the utilization of free cooling. If a refrigeration plant can be reduced in size (or eliminated), operating and capital costs are reduced, including those for the required supporting electrical infrastructure. Free cooling technologies should be considered in all new designs/builds, retrofits, and upgrades. The specific practices for cooling plants are presented in Figure 16 below.

Jump to the Review CRAC/CRAH Settings section Jump to the Review and Optimize Chilled Water Temperature section Jump to the Review and Set Target IT Equipment Intake Temperature section Jump to the Review and Extend the Working Humidity Range section Jump to the Keep Separate Environment Zones section Jump to the Use Chillers with High Coefficient of Performance (COP) or EER section Jump to the Review Cooling System Operating Temperatures section Jump to the Install Variable Speed Drives for Compressors and Pumps section Jump to the Separate Data Centre Chilled Water System from Comfort Cooling System section Jump to the Non-IT Equipment Should not Dictate Cooling System Set Points section Jump to the Variable Speed Fans section Jump to the Control CRAC/CRAH Unit Supply Air Temperatures section Jump to the Ensure Proper Cooling Unit Sizing and Selection section

Review CRAC/CRAH Settings

To avoid cooling units working against each other in occupied areas, it is necessary to make sure that the CRAC/CRAH units have appropriate and consistent temperature and relative humidity settings and are properly calibrated. Many CRAC/CRAH units now have the option to connect their controls and run together when installed in the same area; thus it is necessary avoid any potential new failure modes or single points of failure that may be introduced.

Review and Optimize Chilled Water Temperature

The chilled water temperature set points of cooling plants should be reviewed and increased where feasible to maximize the use of free cooling and reduce compressor energy consumption. Seasonal impacts should also be taken into account as part of this review. Increasing the chilled water temperature difference can reduce the water flow and result in reduced pump energy consumption. It should be noted that water set point changes should be made in tandem with supply air flow set point changes to avoid reducing capacity. For direct expansion (DX) systems, evaporator temperatures should also be reviewed. Lastly, the use of electronic expansion valves (EEVs) allows for better control and permits higher evaporator temperatures (compared to thermostatic expansion valves – TEVs).

Review and Set Target IT Equipment Intake Temperature

Data centres should be designed and operated at the highest efficiency to deliver intake air to IT equipment based on ASHRAE Class A2 that stipulates an allowable range for data centres. When DCs operate within that range, they enable energy savings by reducing or eliminating overcooling.

Review and Extend the Working Humidity Range

Data centres designed within the ASHRAE Class A2 range should review and reduce the lower humidity set points (relative to non-Class A2) to decrease energy demand from humidification where applicable. At the same time, the upper humidity set point(s) of the data floor can be increased within the A2 humidity range can to the dehumidification loads within a facility.

Keep Separate Environment Zones

It is necessary to avoid the need to set the data centre cooling plant at the most extreme range and thus reduce or eliminate opportunities for efficiency for the rest of the installation. If IT equipment with more restrictive temperature or humidity ranges are required, that equipment should be identified and kept in zones or areas with separate environmental controls and should use separate cooling systems where feasible to facilitate cooling efficiency optimization in each zone. Some examples are:

• UPS units requiring tighter environmental controls to maintain battery capacity and lifetimes;
• Tape or drives requiring tighter environmental controls to meet archival criteria;
• Warranties requiring tighter environmental conditions for specific equipment.

Use Chillers with High Coefficient of Performance (COP) or EER

During the design and procurement of new or retrofitted plants, it is necessary to prioritize the selection of refrigeration plants with high coefficients of performance (COPs) or chiller systems with good energy efficiency ratios (EERs) throughout their likely working range as well as matching their capacities to cooling requirements.

Review Cooling System Operating Temperatures

Periodically review and evaluate opportunities to decrease the condensing temperature and increase the evaporating temperature of cooling systems. A reduction in the difference between these temperatures means less work is required from the cooling cycle, thus reducing energy consumption and improving efficiency.

Install Variable Speed Drives for Compressors and Pumps

Electrically commutated (EC) motors are significantly more energy efficient than traditional AC motors across a wide range of speeds and allow for variable speed control, which reduces energy consumption for compressors, pumps, and fans in partial load conditions in which they tend to be operated. Consider the use of variable speed control equipment during new or retrofit stages as well as include the ability to properly control the speed according to the demand on the systems.

Separate Data Centre Chilled Water System from Comfort Cooling System

In buildings that have cooling systems primarily designed to remove heat from technical spaces and IT equipment, chilled water systems should be separated from water used for human comfort cooling in other parts of the building. This is because the required temperature to achieve latent cooling for comfort cooling is substantially below that required for sensible cooling of a data centre. Combining the uses will compromise the efficiency of the data centre cooling system. Consider the use of heat pumps to provide either cooling or heating for office area comfort.

Non-IT Equipment Should not Dictate Cooling System Set Points

For spaces designed primarily for IT equipment, other equipment that shares the space and requires more restrictive temperature or humidity control ranges than IT equipment must be separated so that their requirements are not used to dictate the set points of the cooling system responsible for IT equipment (see the Separate Environmental Zones subsection above).

Variable Speed Fans

Air conditioning/air handling units within a computer room represent the second major component of most cooling systems. Many older CRAC/CRAH units operate fixed speed fans that consume substantial power and make it difficult to manage the data floor temperature. Variable speed fans are more suitable for this purpose, particularly when there is a high level of redundancy in the cooling system, during periods of low utilization of the facility, or for highly variable IT load levels. The control of these fans may be dictated by factors such as the supply or return air temperature or the chilled air plenum pressure.

Control CRAC/CRAH Unit Supply Air Temperatures

In older facilities, many CRAC/CRAH units are controlled return temperature. This practice is no longer considered appropriate. A better practice is to control supply temperature, which ensures an even supply air temperature independent of the load on each CRAC/CRAH unit.

Ensure Proper Cooling Unit Sizing and Selection

The air volumes required by data centre IT equipment can vary with utilization and are dependent on both the IT load (kW) and the IT equipment temperature differential. To properly size cooling unit design flow rates, all of these factors should be considered. In addition, the likely future utilization and bypass should also be taken into account. If the IT temperature is overestimated, it will result in undersized CRAC/CRAH air volumes and potential air management problems. ^{Footnote 24} Conversely, if the temperature differential is underestimated, the CRAC/CRAH air volumes will be oversized, which makes inefficient low partial load operation and air bypass more likely. ^{Footnote 25}

1.9Resilience

1.9.1Importance

Introducing resilience to data centre operations is important for ensuring uninterrupted service delivery and mitigating risks associated with downtimes. Resilience strategies tailored to business requirements and that enable or enhance the ability of data centres to remain operational and recover quickly from unexpected events, either man-made or natural disruptions, not only optimize resource allocations and minimize capital expenditures, but also enhance operational efficiency and flexibility. The specific actions are presented in Figure 17 below.

Jump to the Build Resilience to Business Requirements section Jump to the Design Infrastructure to Maximize Partial Load Efficiency section Jump to the Consider Multiple Levels of Resilience section

1.9.2Specific Actions and Potential Impacts

Build Resilience to Business Requirements

Implement resilience strategies tailored to business requirements to improve operational efficiency, profitability, and flexibility. Only the level of resilience justified by business requirements and impact analysis should be built. For example, 2N infrastructure (100% redundancy) should be carefully matched to business requirements.

Aligning resilience levels with business needs reduces unnecessary investment in oversized infrastructure, optimizes resource allocations, and minimizes costs. Avoiding unnecessary redundancies improves energy efficiency by reducing the operational load on non-critical systems.

Design Infrastructure to Maximize Partial Load Efficiency

Designing infrastructure to maximize partial load efficiency involves strategically configuring systems to operate efficiently even during periods of low demand, thereby leading to significant energy savings and operational benefits. This applies to cooling systems, electrical distribution systems, energy storage devices, and so on.

Energy consumption can be reduced by minimizing equipment use during periods of low demand, thereby lowering operating costs and improving sustainability. Efficient infrastructure design improves system reliability by minimizing component wear, thus reducing the risk of equipment failure and downtimes.

Consider Multiple Levels of Resilience

Implement appropriate levels of resilience in data centres, IT equipment, software, and network infrastructure to align with the service resilience and availability required by business demand. While high levels of resilience are crucial for ensuring continuous operation, it is important to consider the energy efficiency implications. It is possible to build a single data centre to provide multiple levels of power and cooling resilience to different floor areas.

Offering multiple resilience levels in a single data centre allows for greater customization to meet varying business needs as well as enhance customer satisfaction and retention. Also, providing optional resilience features can generate additional revenue streams for colocation providers while meeting the specific needs of different customer segments. Designing appropriate levels of resilience also improves energy efficiency by avoiding over-provisioning, leading to cost savings and environmental benefits.

1.10General Practices

1.10.1Importance

This subsection describes practices that relate to data centres in a more general manner and that can be applied to several aspects. The specific actions are presented in the Figure 18 below.

Jump to the Select ENERGY STAR^® Compliant Equipment section Jump to the Install Energy and Temperature Reporting Equipment section Jump to the Manage Air Flow section Jump to the Consider the Use of Free Cooling section Jump to the Locate the Data Centre Where Waste Heat Can Be Reused section

1.10.2Specific Actions and Potential Impacts

Select ENERGY STAR^® Compliant Equipment

Only data centre equipment that is as energy efficient as possible should be selected. Use ENERGY STAR programs as a reference for selecting servers when available for specific equipment classes. ENERGY STAR^® Program Requirements for Data Centre Storage, ENERGY STAR^® Program Requirements for Computer Servers, and ENERGY STAR^® Certified Uninterruptible Power Supplies can be used. Operators should determine hardware energy efficiency levels through advanced analysis and choose the most efficient equipment for their needs.

By prioritizing energy efficiency in the selection of all data centre equipment, operators minimize energy consumption and maximize resource utilization.

Install Energy and Temperature Reporting Equipment

Choose equipment with direct local measurement capabilities for energy consumption as well as temperature indication. Use standard reporting methods such as IPMI, DMTF Redfish, and SMASH where appropriate.

The aim of this practice is to ensure data centre energy and environmental monitoring across the entire infrastructure, with increasing levels of granularity.

Manage Air Flow

When air movement is used to cool IT equipment, care must be taken to ensure that the ceiling height is sufficient to guarantee the best use and efficiency of air-cooling technologies. Insufficient ceiling height often hinders the effectiveness of air-cooling technologies, such as raised floors, suspended ceilings, aisle confinement, or ducts. All unwanted openings in a raised floor should be closed, and unbroken rows of cabinets should be maintained to prevent air recirculation and minimize the risk of mixing hot and cold air streams. Additionally, optimizing placement and opening factors of vented tiles reduces bypass and enhances cooling efficiency.

Consider the Use of Free Cooling

The data centre should consider and use the free cooling method. Free cooling leverages natural elements such as air or water to cool IT equipment without the need for mechanical refrigeration. The physical layout of the building should not obstruct or limit the implementation of free cooling systems or other equipment with a free cooling/economy mode.

Implementing free cooling methods offers substantial energy savings and lowers operating costs. With the cooler climate in Canada, there is significant opportunity to capitalize on free cooling.

Locate the Data Centre Where Waste Heat Can Be Reused

Strategically position the data centre in a location where waste heat generated by the facility can be efficiently reused. For example, work with neighbouring communities to send waste heat to an offsite location that needs it.

This practice enables energy efficiency and minimizes environmental impacts.

1.11Summary and Recommendations

A combination of business and technology trends are driving the rapid growth of Canadian DCs. This continued growth in demand for DC services is very much in line with international trends. For example, as noted earlier, AI and cryptocurrencies – two of the most intensive growth areas in online services worldwide – are hosted in data centres and raise DC energy consumption. ^{Footnote 26} Taken together, both technologies are expected to double DC facility electricity usage by 2026, reaching more than 1,000 terawatt-hours (TWh) or roughly equivalent to the electricity consumption of Japan. ^{Footnote 27} The IEA also indicated that updated regulations and technological improvements will be needed to moderate the surge in energy consumption from data centres.

While there are no indications as yet about further governmental actions on data centres in Canada or clear trends on how cryptocurrency and AI developments will affect the Canadian data centre market, it is clear that energy and costs for other services are also increasing with economic progress. Therefore, owners, operators, and users of data centre services can increase their overall efficiency, manage costs, and prepare for uncertainties by adopting many of the best practices outline herein, which will increase their competitiveness, optimize their operations, and reduce future risks.

Looking Ahead

As of 2024, there are indicators as to how a number of governments are approaching the challenges of meeting emissions reduction commitments, and increasing energy demands from data centres. As indicated earlier, data centres located in the EU with an installed IT power demand of at least 500 kW are required to report to the EU database by 15 September 2024, per the Energy Efficiency Directive (EED). EU Member States may also change certain aspects of the reporting scheme, such as adding new data points, widening the scope of reporting entities, or setting up their own national reporting platform, for instance. ^{Footnote 28}

In addition, requirements of the EED also apply to mixed use buildings, so in addition to reporting on their DC portion of the building, operators are expected to have separate metering for different functions or sectors within the building (e.g., residential, commercial, and data centre areas. Buildings are required to adopt integrated energy management systems that allow for detailed and accurate reporting of energy consumption for different building functions (DC operators can, for example conduct measurements via the use of DCIM tool). Specific to co-location data centres, it is understood that reporting on ICT capacity and data traffic is especially difficult for colocation data centre operators, as they do not possess this information. For reporting on both of these categories, the Delegated Regulation provides that colocation operators may set up an anonymous internal reporting mechanism, to gather data from their customers strictly for compliance purposes.

Based on industry feedback received during the consultation phase prior to implementation, the EU has seen strong overall industry support for these regulations. To date, industry has only requested an extension or extended transition for the first reporting period, especially for colocation providers. Industry feedback also indicated support for inclusion of smaller data centres (i.e., data centres less than 500 kW), as they can be less efficient. Further information regarding the EED and reporting requirements in the EU for commercial buildings are contained in Appendix II of this document. Recommendations for DC actions and reporting based on these developments are provided below.

Recommendations

It is recommended that all DC stakeholders in Canada and elsewhere take the first step, which is to measure and examine various aspects of their operations using the tools provided herein. As the popular saying goes, you can only control what you can measure. Once indicators are obtained, results will help to point to areas of excellence as well as areas requiring attention. The next recommended step is to gather all relevant decision makers and consult experts in all relevant areas to help identify, select, and implement those practices that can deliver cost-effective impacts in DC operations. It should be noted that an effective course of actions is to consider the implementation of both short and long-term practices. The importance of energy efficiency in data centres cannot be overstated as these facilities consume significant amounts of energy, and optimizing their operations can lead to substantial cost savings and environmental benefits.

Below are key recommended metrics and practices for those who wish to gain a comprehensive overview of their data centre energy efficiency. Each recommended metric covers a different aspect of energy usage. For example, power usage effectiveness (PUE) is focused on the overall efficiency of data centres, while power density addresses the intensity of energy use in relation to the space occupied. When combined with carbon usage effectiveness (CUE), water usage effectiveness (WUE), and cooling system monitoring, these metrics collectively offer a substantial understanding of data centre energy efficiency. Monitoring these indicators not only improves energy efficiency, but also enhances the reliability and lifespan of IT equipment by maintaining stable environmental conditions. Operators and management boards should consider adopting a combination of these practices to achieve best results.

• Power Density (kW/ft²): This metric provides a clear understanding of how much power is being used relative to the floor space of the data centre. By monitoring power density, operators can identify areas of inefficiency and optimize the placement and utilization of IT equipment.
• Power Usage Effectiveness (PUE): PUE is a widely recognized metric used to compare data centre performance over time. PUE offers insights into how much energy is being used for cooling, lighting, and other non-IT functions.
• Carbon Usage Effectiveness (CUE) and Water Usage Effectiveness (WUE): CUE and WUE are metrics that serve to benchmark data centre carbon footprints and water consumption levels respectively. These metrics are critical for understanding the environmental impacts of data centre operations, particularly in regions where water scarcity and carbon emissions are of concern.
• Monitoring of Cooling Systems: The cooling system is one of the most energy-intensive components of a data centre, and its efficiency can have a significant impact on overall energy use. To ensure optimal performance, it is recommended that cooling systems be continuously monitored using advanced techniques and technologies. Some options for monitoring include:
  • Real-Time Temperature and Humidity Sensors: These sensors can be installed throughout the data centre to provide continuous feedback on environmental conditions. The data can be used to adjust cooling strategies in real time, thus ensuring that the HVAC system is neither overworking, nor underperforming.
  • Intelligent Control Systems: Implementing intelligent control systems, such as those that use machine learning algorithms, optimizes cooling based on actual loads and environmental conditions. These systems dynamically adjust airflow, temperature, and humidity levels, leading to significant energy savings.
  • Thermal Imaging and Computational Fluid Dynamics (CFD) Modeling: Regular thermal imaging and CFD modeling help identify hotspots and inefficiencies in airflow distribution. These tools serve to guide adjustments to the layout and operation of cooling systems, ensuring even cooling across all equipment.
  • Integration of Variable Speed Drives for Fans and Pumps: Integrating variable speed drives into fans and pumps allows the cooling system to adjust its output based on real-time demand, matching the cooling output to the exact needs of the data centre.

Finally, while it is not one of the recommended practices, stakeholders should consider collaboration with other organizations and stakeholders to promote successes, learn from others, and help further industry knowledge on this topic. It is likely that, due to the rapid pace of progress in the industry, there are best practices that may still be under development and are not yet highlighted or shared in any written documentation.

2.1Colocation User Overview

The information in this section is aimed at colocation users (“colo”), i.e. customers who install and manage their own hardware and IT services. These customers have control over hardware, data management, and some cooling measures. In addition, metering can be carried out to ensure good performance.

In certain cases, some organizations are transitioning from owning their own IT infrastructure to managed IT services and colocation particularly to secure cloud computing with hyperscale providers. For those customers, the primary area for improvement is how they manage their data.

It is expected that one or more of the practices listed below will have been adopted by some colocation users. However, as the adoption of best practices is one of the lesser known aspects of the Canadian data centre market, the practices presented here are those that will have the greatest impacts in terms of operational improvements.

Jump to the Colocation Users section Jump to the Hardware section Jump to the Data Management section Jump to the Cooling section Jump to the Metering and Metrics section

2.2Quick Start & Category Specific Actions

The three items below consist of measures that can be implemented quickly. These are considered to be the most effective actions that colocation users can take.

• Install IT energy consumption meters;
• Implement a comprehensive environmental, energy, and/or asset management plan;
• Conduct data management.

2.3Metering and Metrics

2.3.1Importance

Metering in a data centre refers to the systematic monitoring and measurement of energy consumption and other resources used by the equipment within a facility. Since colocation users do not have access to all data centre equipment, metering is primarily limited to hardware.

Measuring equipment energy consumption allows for accurate diagnosis of issues and justifies potential improvements. A more detailed understanding of various performance metrics offers significant opportunities to reduce energy consumption and costs. The specific actions are presented in Figure 20 below.

Jump to the Periodic Reports on Standardized Key Performance Indicators section Jump to the IT Energy Consumption Meters section

2.3.2Specific Actions and Potential Impacts

Periodic Reports on Standardized Key Performance Indicators

Consider producing periodic reports on standardized key performance indicators (KPIs) such as IT equipment energy savings values and IT equipment utilization savings, as detailed earlier.

By implementing a system for periodic reporting on KPIs, energy consumption and environmental impacts can be effectively monitored, which enables the identification of areas for optimization and leads to potential cost savings and reduced environmental footprints.

IT Energy Consumption Meters

Measure total energy supplied to IT systems using metering equipment to track energy consumption. Total energy can also include other power sources whereby non-UPS protected power is delivered to server cabinets. An automated energy and environmental reporting console can be used to monitor energy use and enhance capability.

2.4Data Management

2.4.1Importance

Data management is the practice of collecting, organizing, protecting, and storing data so that the data can be processed and used appropriately. Data management is also a necessary step for effective data analysis.

Data management and processing requirements may vary depending on the type of organization handling the data (e.g. governments, NGOs, businesses, or educational entities) as each is subject to sector-specific requirements. The specific actions are presented in Figure 21 below.

Jump to the Optimise Server, Network, and Storage Utilization section Jump to the Select or Develop Efficient Software section Jump to the Eliminate Traditional 2N Hardware Clusters section Jump to the Reduce Total Data and Storage Volume section

2.4.2Specific Actions and Potential Impacts

Optimise Server, Network, and Storage Utilization

Measure the proportion of overall network capacity used, overall storage capacity utilized, and processor utilization. This measure can be grouped either globally or by service/location. A basic level of reporting can be very informative and should take energy consumption into account. The definition of utilization may vary depending on factors such as available capacity and what is considered used.

Implementing this best practice in a data centre can lead to optimized server, network, and storage utilization, thus ensuring that resources are used efficiently and reducing unnecessary energy consumption. This results in cost savings, improved performance, and lower environmental impacts.

Select or Develop Efficient Software

Consider making energy efficiency a priority in the software selection process or in the software development process, as described earlier. This practice is primarily aimed at reducing energy consumption to save money and reduce environmental impacts.

Eliminate Traditional 2N Hardware Clusters

Eliminating traditional 2N hardware clusters involves replacing active/passive server hardware clusters with faster recovery approaches as described in Subsection 1.6 .

By adopting this strategy, organizations streamline their infrastructure, thus reducing redundancy and improving resource utilization.

Reduce Total Data and Storage Volume

Organizations should implement robust data identification and management policies aimed at minimizing both the total volume of stored data and the number of data copies, as described in Subsection 1.6 .

This practice serves to minimize storage overhead and improve resource utilization, ultimately contributing to a more sustainable and cost-effective storage environment.

2.5Hardware

2.5.1Importance

In a data centre, hardware refers to the physical components that make up the infrastructure necessary for storing, processing, and transmitting data. Colocation facility customers who have the freedom to choose certain equipment such as servers and storage devices should ensure they respect best practices.

In the choice of equipment as well as in its operation, it is important to prioritize energy consumption reductions. In addition, the entire equipment life cycle must be taken into consideration. The materials used in the design, the end-of-life of the equipment, the methods of disposal are all elements that impact the total energy consumption of equipment in addition to greenhouse gas emissions. Emissions emitted outside the operation, i.e. the life cycle stages before and after equipment is installed in a data centre, are known as Scope 3 emissions. Scope 3 emissions can often exceed those of the use phase itself, so they should not be neglected.

An informed choice of equipment is important to reduce not only energy consumption when using equipment, but also Scope 3 emissions. Specific actions are outlined in Figure 22 below.

Jump to the Control Operating Temperature and Humidity Ranges section Jump to the Manage Power Management and Control of Equipment Energy Use section Jump to the Remove Unused Equipment and Consolidate Existing Services section Jump to the Manage End-of-Life Equipment section Jump to the Select Lower Power Storage Devices section Jump to the Select ENERGY STAR^® Compliant Equipment section Jump to the Install Energy and Temperature Reporting Equipment section

2.5.2Specific Actions and Potential Impacts

Control Operating Temperature and Humidity Ranges

Procure equipment capable of operating within ASHRAE Class A2 ranges. If not possible, then equipment supporting at least ASHRAE Class A1 may be procured. Additionally, consider equipment that operates within a broader range of intake temperature and humidity levels as defined in ASHRAE Class A4 (which is broadly equivalent to ETSI EN 300 019-1-3 Class 3.1).

By respecting the operating temperature and humidity ranges, optimal equipment performance can be achieved. In addition, equipment that operates over a wider range of inlet temperature and humidity levels as defined in ASHRAE Class A4 allows greater flexibility under varying environmental conditions.

Manage Power Management and Control of Equipment Energy Use

As hardware is deployed, change the deployment process to include enabling power management features, as described in Subsection 1.5 .

Selecting equipment with built-in energy control features facilitates external management of energy consumption.

Remove Unused Equipment and Consolidate Existing Services

Consider removing and decommissioning all unnecessary equipment while regularly reviewing installed equipment to ensure ongoing necessity, as described in Subsection 1.5 .

This practice yields significant benefits including cost savings from reduced equipment purchasing, maintenance, and energy consumption.

Manage End-of-Life Equipment

Removed equipment should be disposed of in line with the principles of the circular economy to minimize environmental impacts and maximize utility. Prioritize evaluating unused equipment to determine if it can be repurposed, sold, or recycled as part of the decision-making process. Explore opportunities for internal or external equipment reuse. If equipment cannot be reused, ensure proper recycling. Look for certified recycling centres with the necessary facilities to safely process electronic components.

Responsible end-of-life equipment management promotes a circular economy by enabling the reuse and recycling of hardware resources. This approach also helps reduce Scope 3 emissions by minimizing electronic waste and promoting sustainable equipment use.

Select Lower Power Storage Devices

It is crucial to prioritize lower power storage devices that encompass factors like performance achieved, storage volume per watt, levels of data protection, availability, and recovery capacity, as described in Subsection 1.5 .

Prioritizing lower power storage hardware reduces operational costs, minimizes environmental impacts, and ensures alignment with performance and data protection requirements, thereby enhancing overall sustainability and resource optimization.

Select ENERGY STAR^® Compliant Equipment

The choice of IT equipment in data centres should be as energy efficient as possible using ENERGY STAR^® programs as a reference, as described in Subsection 1.10 .

By prioritizing energy efficiency in the selection of all data centre equipment, operators aim to minimize energy consumption and maximize resource utilization.

Install Energy and Temperature Reporting Equipment

Choose equipment with direct local measurement capabilities for energy consumption as well as temperature indication, as described in Subsection 1.10 .

The aim of this practice is to ensure data centre energy and environmental monitoring across the entire infrastructure, with increasing levels of granularity.

2.6Cooling

2.6.1Importance

It is essential to remove the heat produced by data centre IT equipment and transfer that heat elsewhere as high temperatures and humidity levels can damage IT equipment, causing it to fail.

Generally, heat removal for cooling or data centre cooling collectively refers to the equipment, tools, systems, techniques, and processes that work together to maintain ideal temperatures and humidity levels within a facility. For colocation users, cooling systems are normally controlled by the data centre operator, so only certain practices regarding IT equipment can optimize cooling. Specific actions are outlined in Figure 23 below.

Jump to the Cabinet Air Flow Management – Blanking Plates and Other Openings section Jump to the Ensure Adequate Free Area on Cabinet Doors section

2.6.2Specific Actions and Potential Impacts

Cabinet Air Flow Management – Blanking Plates and Other Openings

Installing blanking plates or blanking panels to cover openings on racks where there is no equipment as well as other air gaps reduces the volume of hot air recirculating through gaps in server cabinets, as described in Subsection 1.8 .

Ensure Adequate Free Area on Cabinet Doors

ITE cabinets with solid doors should be considered for replacement with perforated doors to ensure adequate cooling airflow, as described earlier.

2.7Summary and Recommendations

As previously stated, owners, operators, and users of data centre services can adopt many best practices presented herein to increase their overall efficiency and help manage costs. In doing so, they will be more cost-competitive and prepared for uncertainties while reducing future risks, and prepare for potentially new regulations.

While there are no indications as yet about further governmental actions on data centres in Canada, it is clear that energy and costs for services are increasing with economic progress. Therefore, users of data centre services can manage costs, and prepare for uncertainties by adopting many of the best practices outline herein, which will also encourage their providers to increase their competitiveness, optimize their operations, and thus reduce future risks for users and operators alike.

As of 2024, many governments are refining their approach to meet the challenges of emissions reduction commitments and address increasing energy demands from data centres. Of note, data centres located in the EU with an installed IT power demand of at least 500 kW are required to report to the EU database by 15 September 2024, as required by the EU Energy Efficiency Directive (EED). The Summary and Recommendations section for DC operators ( Subsection 1.11 ) and Appendix III contain further information on these requirements and measurement metrics used. It is recommended that colocation users also take actions to measure and examine various aspects of their operations using the tools suggested herein using the appropriate indicators as applicable to their needs and available capacity. They can also work with their service providers to understand their practices and impacts. Doing so can help point to areas of excellence and others that require attention.

Another key recommended step for colocation users is to involve all relevant decision makers and consult experts to help identify, select, and implement those practices that can deliver cost-effective impacts in the short and long terms. Stakeholders should also consider collaboration with other organizations to learn, promote, and share successes and lessons learned.