Skip to content

I Feel Your Pain

As you delve into the world of AEMO's data, you will come across countless inconsistencies which will drive you mad. You need to be prepared to roll with the punches. Just remember to be thankful that we have this data at all. I do not know of any other grid or industry with this much public data.

Examples of inconsistencies:

  • AEMO have standard file formats for most of their data. However for some data (e.g. price setter data, or 4 second SCADA data) they choose something different, like XML.
  • AEMO publishes new schemas about twice a year. They have a well defined system, with consultation and clear major and minor version numbers. They often add new columns, but rarely delete old ones or change column order. The DISPATCHREGIONSUM table has several columns at the end about aggregate renewables. Despite sounding promising, these are always empty. When I asked why, I was told that it is something they plan to add in the future. So they are effectively adding this columns to this one table in a way which is outside their normal schema versioning process for all other tables.
  • AEMO have the MMS dataset on Nemweb and the FTP servers, which contains hundreds of different tables. However for some data (e.g. the Participant Registration List) they publish it as an excel file on their aemo.com.au web page. For most of AEMO's data, they keep all version history (e.g. each iteration of a price forecast). However when generators retire, they are deleted from this registration form, instead of just adding a column to mark them as retired. They changed the firewall rules for this other domain while I was writing this blog post, which broke Nemosis. This breakage applies even to market participants with FTP access, because they do not publish this file through the usual channels.
  • Additionally, in this same list of generators:
    • There are spelling mistakes, such as "Photovoltalic" and "Natrual Gas".
    • All batteries are listed with a "Fuel Source - Primary" of "Battery Storage", except for the Hornsdale battery, which is listed as "Wind". (Probably because it was the first battery, and the registration rules have changed since then.)
    • The interconnector between Tasmania and Victoria (Basslink) is in the registration list. Other HVDC interconnectors are not listed.
  • Some batteries are registered as one DUID, some are two (one for charge, one for discharge). Even when there are two, the power values can be both positive and negative for both of them.
  • Some non-scheduled generators have no DUID, but some do.
  • Typically each monthly file ends with the 5 minute period before the 5 minute period that the next month starts with. Except for one month where they overlapped.
  • For some months, some tables are missing. They are present for earlier and later months.
  • Up to July 2024 the filenames on Nemweb were like PUBLIC_DVD_ANCILLARY_RECOVERY_SPLIT_202309010000.zip. Then the month after that they changed to PUBLIC_ARCHIVE#ANCILLARY_RECOVERY_SPLIT#FILE01#202409010000.zip. This broke webscraping scripts. When writing scripts today you must handle both cases.
  • For a week or so, the URLs were double-URL encoded, which broke scraping scripts.
  • The schema documentation groups similar tables together. When the data for tables is combined into multi-table CSVs for the realtime data, they are partitioned into different groups.
  • There are tables with CDEII in the name which are about emissions in the daily data, but not the monthly data. The monthly data contains tables with CO2 in the name, which are not in the daily data. Neither is in the documentation. Unlike all the other daily data, CDEII is published as uncompressed CSV files, instead of zips of CSVs.
  • The INTERVENTION column is always a 1 or 0, although the schema documentation says they can take a range from 0 to 99. Except for table GENCONSETINVOKE, where it is a VARCHAR2(1) with values Y or N (even though the documentation comment says it is 0 or 1).
  • Data types in the documentation are normally like NUMBER(2,0), but sometimes they are Number(2,0) or Number (2,0)
  • The MAXCAPACITY and REGISTEREDCAPACITY columns in DUDETAIL are described as integers in the schema documentation. However in the actual CSV data they are floats, with non-zero decimal parts. If you use AEMO's PDR Loader, you will lose the decimal part. (I notified AEMO about this. They said it is not a bug.)
  • The list of tables in the documentation sometimes has spaces in the middle of table names. (e.g. SET_APC_COMPENSATION used to appear as SET_ APC_COMPENSATION, thus making it difficult to find the table you want with ctrl-F)
  • Sometimes the column names in the documentation are incorrect. e.g. for table DISPATCHCONSTRAINT, the documentation says DUID, but the actual CSV has column CONFIDENTIAL_TO. The documentation for table MTPASA_REGIONRESULT says there is a column TOTALSEMISCHEDULEGEN10, but the actual CSV contains TOTALSEMISCHEDULEDGEN10 (with an additional D).
  • Table SET_APC_COMPENSATION (in the SETTLEMENT_DATA package is different to SETAPCCOMPENSATION (in the HISTORICAL TABLES package).
  • The HISTORICAL TABLES package is the only one with a space in the name. All the others have underscores.
  • The list of table update triggers includes several tables not published anywhere nor documented anywhere else (as far as I can tell).

Whilst these things are frustrating, I am still very grateful that AEMO publish this data at all. This is far better than any other grid operator I have encountered.