An Empirical Analysis of Cloud Carbon Models

• Data centers are a major contributor of greenhouse gas emissions. Google and Amazon have recently released dashboards that attribute cloud-related carbon emissions to specific customer-facing services. It is your task to evaluate the accuracy and behavior of these models.
• Related Work
  • Google Cloud: https://cloud.google.com/carbon-footprint
    • They also have detailed information on their methology.
  • AWS: https://aws.amazon.com/blogs/aws/new-customer-carbon-footprint-tool/
  • Microsoft Azure:
    • https://www.microsoft.com/en-us/sustainability/emissions-impact-dashboard
    • https://learn.microsoft.com/en-us/azure/batch/
      • This does carbon-aware scheduling according to https://news.microsoft.com/wp-content/uploads/prod/sites/418/2023/01/carbon_aware_computing_whitepaper.pdf
  • WattScope: Non-intrusive Application-level Power Disaggregation in Datacenters (https://dl.acm.org/doi/10.1145/3649477.3649491)
    • AWS/Google apparently don’t do that. What are the consequences?
  • Carbon Explorer: A Holistic Framework for Designing Carbon Aware Datacenters (https://dl.acm.org/doi/10.1145/3575693.3575754)
    • https://if.greensoftware.foundation/
  • Going Green for Less Green: Optimizing the Cost of Reducing Cloud Carbon Emissions (https://dl.acm.org/doi/10.1145/3620666.3651374)
    • They compare different approaches including the following:
      • “Let’s wait awhile: how temporal workload shifting can reduce carbon emissions in the cloud” https://dl.acm.org/doi/10.1145/3464298.3493399
  • https://arxiv.org/abs/2406.16250
• Research Questions
  • Do we need a carbon tax, or are the existing prices good enough already?
    • Analyze pricing incentives for low-carbon computing in the cloud. Find examples where the price is low, but the carbon footprint is high (carbon bugs).
    • For example, compare VMs in cheap/expensive locations
    • In some cases, the prices may already align with the emissions (because renewables are cheaper). It is your task to determine the extent by analyzing the emissions per dollar spent for different tasks.
  • Can customers effectively optimize carbon using this model?
    • Try the different techniques they propose to reduce the footprint in https://cloud.google.com/architecture/reduce-carbon-footprint - which are the most impactful in practice? Do they work at all? (I would be quite suprised if they actually have numbers on all of those.)
      • VMs vs. serverless
        • But serverless is more expensive which correlates with carbon footprint per Google’s model.
      • Apache Hadoop/Spark vs. BigQuery/Dataproc/Spanner
      • Batch workloads: Do tight/loose deadlines have any effect?
    • As an extension, you can compare the carbon footprint of a VM with the carbon footprint of a local machine.
      • The data includes separate scope 1/2/3 emissions: https://cloud.google.com/carbon-footprint/docs/data-schema
  • Does the model actually account for carbon-aware scheduling?
    • Their documentation on the topic is not really clear.
    • If yes, what’s the effect according to their model?
  • Is the model accurate?
    • Not sure if this can be answered.
    • Compare the reported emissions from the different clouds.
    • Are the results reproducible between different VM instances / services?
    • Maybe check using electricity map, life cycle analysis (embodied), and RAPL?
  • Which accounting metrics do the models use?
    • SCI-based Active Job Carbon Intensity: operational_job + job_usage * active_server_embodied
    • Job Carbon Intensity Proportional: operational_job + job_usage * active_server_embodied + operational_idle_servers + embodied_idle_servers
    • Active Job Operational Intensity: operational_job + operational_idle_servers
• Challenges
  • Cloud pricing
    • 300USD for 90 days are free. Maybe we can apply for more.
  • Emission data may only be shown on the dashboard with a delay (21 days in Google Cloud).
    • Carefully choose a set of orthogonal experiments that allows you to draw conclusions.
    • Also, we should maybe publish the resulting dataset.
  • There is limited information on model behavior and accuracy.
    • AWS: None I believe.
    • Google Cloud: https://cloud.google.com/carbon-footprint/docs/methodology
      • Maybe, they actually do power monitoring: “Carbon Footprint builds its calculations from bottom-to-top, relying heavily on machine-level power and activity monitoring inside Google data centers.”
      • Maybe, they actually take time-of-use into account: “The increased granularity of the emissions calculation is more suitable for optimizing workload location and timing to reduce your operational greenhouse gas emissions.”
      • “The carbon emissions from electricity per customer and product are then augmented with proportional allocations of emissions arising from the non-electricity sources”
        • This implies a customer’s share of embodied emissions is determined by their share of operational emissions. But shouldn’t their share be determined by the economic incentive they gave to Google to buy the hardware (i.e., the price they paid)?
      • “Each machine’s hourly dynamic power is allocated to the internal services it supported that hour, based on relative internal service CPU usage.”
        • This implies they do only estimate app. power consumption based on CPU usage. This might benefit IO-intensive tasks.
      • “Where Electricity Maps data is unavailable, Google uses country-specific annual average carbon emission intensity factors published by the International Energy Agency.”
        • This implies attribution does not account for time-of-use in those regions.
      • Prices have a direct impact on reported emissions: “An internal service’s carbon footprint is divided amongst its SKUs proportional to their usage (quantity purchased) and list prices (all in US dollars)”
      • They apparently do not account for the embodied carbon of renewables
        • “Fixed an issue that omitted Scope 1 and Scope 3 emissions for a given region in a given month that has zero Scope 2 emissions due to 100% carbon free energy. Specifically, in July 2022, northamerica-northeast1 had 100% of its electricity generated from hydro power for the month, resulting in no Scope 2 emissions for the month despite positive energy consumption to run the data center.” (https://cloud.google.com/carbon-footprint/docs/release-notes)
• Design / Implementation
  • The main practical work will be in selecting / running the tests and analyzing the data. You will learn to work with system benchmarks and cloud services / APIs.
  • Actually implementing an improved ’carbon footprint’ tool for something like OpenStack is not required, but we may decide to change the focus in case there is some serious blocker regarding the data collection.
    • In case we do that: For Kubernetes, idle CPU carbon footprint should be attributed to the container that reserved the resource.