{ "cells": [ { "cell_type": "markdown", "id": "4cec4544-df31-4e16-9733-db53ed8e40ac", "metadata": {}, "source": [ "# Grouping Generators by Fuel Type\n", "\n", "This script is an example from my article about AEMO's MMS data:\n", "\n", "https://www.mdavis.xyz/mms-guide\n", "\n", "Here we download the list of generators. Unlike all other AEMO data, this one is on their aemo.com.au domain, not Nemweb, and it's an excel file, not CSV. \n", "\n", "This list contains a lot of very nuanced and varied fuel type descriptions, as well as several typos. So it is a bit fiddly to coerce this into something nice like solar/wind/hydro/coal/gas/battery. The purpose of this example is to show you how." ] }, { "cell_type": "code", "execution_count": null, "id": "f15df104-98e2-4c25-80ef-a6e6acad2197", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "import polars as pl\n", "import requests\n", "from nemosis import static_table" ] }, { "cell_type": "code", "execution_count": null, "id": "5f75779e-369d-46ca-8815-1c29b40271b0", "metadata": {}, "outputs": [], "source": [ "# Point to a folder on your machine\n", "nemosis_data_cache = \"/home/matthew/Data/nemosis/\"" ] }, { "cell_type": "markdown", "id": "b5b7c077-1745-4d43-8744-e63605aacac7", "metadata": {}, "source": [ "# Download The Data" ] }, { "cell_type": "code", "execution_count": null, "id": "dab7d7d0-2adb-42a0-ab3b-d708b54aa3f7", "metadata": {}, "outputs": [], "source": [ "# This normally works\n", "# But AEMO's recent firewall change broke it\n", "# https://github.com/UNSW-CEEM/NEMOSIS/issues/60\n", "def get_with_nemosis():\n", " os.makedirs(nemosis_data_cache, exist_ok=True)\n", " df_pd = static_table(\"Generators and Scheduled Loads\", nemosis_data_cache)\n", " # from Pandas to Polars\n", " df_pl = pl.from_pandas(df_pd)\n", " return df_pl\n", "\n", "\n", "# A fallback, which is a bit more fiddly\n", "def get_manually():\n", " url = \"https://www.aemo.com.au/-/media/files/electricity/nem/participant_information/nem-registration-and-exemption-list.xlsx\"\n", "\n", " # AEMO recently started discriminating based on User-Agent, blocking us\n", " # AEMO, if you are reading this and do not like it, then\n", " # please just serve this file through your existing channels for automated downloads.\n", " # i.e. the MMS Participant Data Replicator channels (nemweb and FTP)\n", " headers = {\n", " \"Referer\": \"https://www.aemo.com.au/energy-systems/electricity/national-electricity-market-nem/participate-in-the-market/registration\",\n", " \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36\",\n", " }\n", "\n", " response = requests.get(url, headers=headers)\n", " response.raise_for_status()\n", "\n", " return pl.read_excel(response.content, sheet_name=\"PU and Scheduled Loads\")\n", "\n", "\n", "try:\n", " df = get_with_nemosis()\n", "except ValueError as e:\n", " if (\n", " str(e)\n", " == \"Excel file format cannot be determined, you must specify an engine manually.\"\n", " ):\n", " print(\"Nemosis failed, switching to fallback method\")\n", " df = get_manually()\n", " else:\n", " raise\n", "\n", "df.head()" ] }, { "cell_type": "markdown", "id": "5687d961-9d59-4efa-a8f9-229935655cfd", "metadata": {}, "source": [ "# Simplify Fuel Categories" ] }, { "cell_type": "code", "execution_count": null, "id": "42e514a9-58f8-49f2-93ba-2e2875f577cb", "metadata": {}, "outputs": [], "source": [ "static: pl.LazyFrame = (\n", " df\n", " # simplify fuel categories\n", " .with_columns(\n", " pl.concat_str(\n", " [\n", " \"Fuel Source - Primary\",\n", " \"Technology Type - Descriptor\",\n", " \"Fuel Source - Descriptor\",\n", " ],\n", " separator=\" - \",\n", " )\n", " .str.to_lowercase()\n", " .alias(\"Fuel Detail\"),\n", " )\n", " .filter(~pl.col(\"Participant\").str.contains(\"Basslink\"))\n", " .with_columns(\n", " pl.when(\n", " (pl.col(\"DUID\") == \"-\")\n", " & (pl.col(\"Station Name\").str.contains(\"Jindabyne Pump At Guthega\"))\n", " )\n", " .then(pl.lit(\"hydro\")) # exception for this one\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"battery\"))\n", " .then(pl.lit(\"battery\"))\n", " .when(\n", " pl.col(\"Fuel Detail\").str.contains(\"hydro\")\n", " & (pl.col(\"Dispatch Type\") == \"Load\")\n", " )\n", " .then(pl.lit(\"pumps\"))\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"hydro\"))\n", " .then(pl.lit(\"hydro\"))\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"solar\"))\n", " .then(pl.lit(\"solar_gridscale\"))\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"wind\"))\n", " .then(pl.lit(\"wind\"))\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"waste coal mine gas\"))\n", " .then(pl.lit(\"coal\"))\n", " .when(\n", " pl.any_horizontal(\n", " pl.col(\"Fuel Detail\").str.contains(s)\n", " for s in [\n", " \"natural gas\",\n", " \"ocgt\",\n", " \"coal seam gas\",\n", " \"coal seam methane\",\n", " ]\n", " )\n", " )\n", " .then(pl.lit(\"gas\"))\n", " .when(\n", " pl.any_horizontal(\n", " (pl.col(\"Fuel Detail\").str.contains(c))\n", " # don't blindly search for the word \"coal\"\n", " # because coal seam gas should count as gas not coal\n", " for c in [\"black coal\", \"brown coal\"]\n", " )\n", " )\n", " .then(pl.lit(\"coal\"))\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"diesel\"))\n", " .then(pl.lit(\"distillate\"))\n", " .when(pl.col(\"Fuel Detail\").str.contains(\"biomass\"))\n", " .then(pl.lit(\"biomass\"))\n", " .when(pl.col(\"DUID\").str.contains(\"PUMP\") & (pl.col(\"Dispatch Type\") == \"Load\"))\n", " .then(pl.lit(\"pumps\"))\n", " .alias(\"FUEL_TYPE\")\n", " )\n", " .select(\"DUID\", pl.col(\"Region\").alias(\"REGIONID\"), \"FUEL_TYPE\")\n", ")\n", "\n", "static.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "f47ea18b-404f-475e-96ec-428e3020c1d3", "metadata": {}, "outputs": [], "source": [ "# check we've categorised everything\n", "uncategorised = static.filter(pl.col(\"FUEL_TYPE\").is_null())\n", "if not uncategorised.is_empty():\n", " display(uncategorised)\n", "\n", "assert uncategorised.height == 0, \"Some generators were not categorised\"" ] }, { "cell_type": "markdown", "id": "fd58e9c3-1d10-4732-8b02-409bd1f08084", "metadata": {}, "source": [ "# Conclusion\n", "\n", "Now we have a dataframe (`static`) which has columns:\n", "\n", "- `DUID`: generator identifier\n", "- `REGIONID`: where is it\n", "- `FUEL_TYPE`: wind, battery, distillate, hydro, biomass, pumps, coal, solar_gridscale, gas\n", "\n", "Note that rooftop solar is not included here." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }