Skip to content

How to download the data?

Data Size

The biggest two tables add up to as the rest combined. So only download what you need.

For the full decade or more of historical data, most of the tables are hundreds of megabytes (as compressed CSVs). Price forecasts are tens of gigabytes. Bidding data is hundreds of gigabytes.

Nemosis

There are a few open source tools out there for downloading and parsing AEMO's data. Nick Gorman from UNSW has written a great library called Nemosis.

Here is an example of how to obtain the DISPATCHPRICE data as a Pandas dataframe:

from nemosis import dynamic_data_compiler

start_time = '2017/01/01 00:00:00'
end_time = '2017/01/01 00:05:00'
table = 'DISPATCHPRICE'
raw_data_cache = 'C:/Users/your_data_storage'

price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache)

Nemosis has some limitations. I am working with the maintainer to make some improvements. In particular, it loads each file into memory as a Pandas dataframe (even when using cache_compiler). This means that you cannot process the very large files (e.g. bid data) on a normal laptop.

If Nemosis works for you, skip the next sections and jump to How to Query and Understand the Data. If not, the next few sections describe the details of how to download and parse AEMO's data files yourself.

Polars

Personally I prefer Polars to Pandas. If you have not heard of Polars, it is like the new Pandas. It is about 30 times faster, and I find the syntax easier to read and write. (No more dreaded errors about "A value is trying to be set on a copy of a slice from a DataFrame", nor clunky df[df[col]], df.loc['col', df['col'] == df['col']].)

On each table's documentation page there is a code sample showing how to query CSVs on disk with Polars. That is applicable if you downloaded the files yourself. (e.g. for tables which are not supported by Nemosis.) Those code samples use deliberately narrow data types (e.g. pl.Categorical() or pl.Enum()), as an optimisation. This is only really useful for the largest tables. If you run into errors, just delete the schema inference and use a simpler approach.

Nemosis combined with Polars (or other tools)

Nemosis' dynamic_data_compiler returns a Pandas dataframe. You can use the cache_compiler to just save the data as Parquet with Nemosis, then query those files on disk with Polars. (Use the same approach for DuckDB, Arrow, R etc.) For example:

from nemosis import cache_compiler
import polars as pl
import os

start_time = '2024/01/01 00:00:00'
end_time = '2024/01/01 00:05:00'
table = 'DISPATCHPRICE'
raw_data_cache = '/home/matthew/Data/nemosis'

 download zips of CSV files 
 extract them, conver them to parquet with nemosis
cache_compiler(start_time, end_time, table, raw_data_cache, fformat='parquet')

def parse_datetimes(
    lf, cols=["SETTLEMENTDATE"], format="%Y/%m/%d %H:%M:%S"
) -> pl.LazyFrame:
    for col in cols:
        lf = lf.with_columns(
            pl.col(col).str.strptime(pl.Datetime(time_unit="ms"), format=format)
        )
    return lf


 query the data with Polars
lf = (
    pl.scan_parquet(os.path.join(raw_data_cache, "*DISPATCHPRICE*.parquet"))
    .pipe(parse_datetimes)
)

print(lf.collect())

Note that I am passing a wildcard filename *DISPATCHPRICE*.parquet to polars. This is because Nemosis downloads all files into the same directory. If you are processing multiple tables (e.g. P5MIN_REGIONSOLUTION and DISPATCHPRICE) you need this string to tell Polars which files to read and which to ignore. Sometimes Polars struggles to read in multiple files in one scan_parquet() call if the schema differs. (e.g. if older files are missing a column.) If this happens, scan each file separately, then combine them (pl.concat(lfs, how='diagonal_relaxed')).

Nemosis currently saves datetimes as strings in the Parquet file. If using cache_compiler, you need to parse them into datetimes yourself. In the example above, I factored that into a simple function with .pipe(parse_datetimes).

Sometimes Nemosis excludes columns from the data. (It includes only the most important columns, as an optimisation.) If this happens (e.g. 1 second FCAS data), you can fix that with this workaround. This workaround is included with the specific column names on each table's documentation page.

PDR Batcher and PDR Loader

AEMO created this dataset and the Data Interchange for market participants (generators, retailers etc) in the 1990s, prior to REST APIs, TLS, parquet files, distributed systems, clouds etc. They expect that anyone who wants to read the data will download a pair of applications called "PDR Batcher" and "PDR Loader". ("PDR" stands for Participant Data Replicator.) Batcher downloads the files over FTP, inside a private VPN, then Loader loads them one row at a time into an Oracle SQL Database. The documentation is here.

There are several challenges with this approach. The first is that AEMO removed this application from their website. They took it down when the log4j vulnerability become public, because this software was affected. Then when they released a patched version months later, they did so only through the private FTP server. So if you are not a participant (e.g. you are an academic researcher), you will probably not be able to even obtain the binary.

Even if you have a copy, it is extremely difficult to get working unless you know someone with experience. If you are a researcher, you do not have the time to spend weeks learning how this bespoke system works, and debugging it when it does not. The system is designed as a thick client, instead of a clear abstract API. There are many obscure mapping tables and configuration options. (e.g. How do you tell PDR Batcher which tables should be append-only, and which ones should overwrite existing rows based on certain partition columns? The prod configuration is missing some tables. So you need to carefully merge it with the preprod configuration.) The latest release did add a lot of great functionality (e.g. natively connecting to cloud storage such as S3), but my view is that if you are just a researcher (and even for some market participants) it is not worth using. e.g. I know people who tried to backfill a fresh database with all publicly available data. They ran into many issues. They asked AEMO for help, and were told that there is no simple, generalised way to backfill all the public data.

PDR Loader operates one row at a time. If you want to backfill all bid or price prediction data for the last decade, that takes weeks, even if you use large, expensive servers. This was probably because the system is optimised for operational use cases, where generators insert a few rows at a time into a row-based database, and query mostly the last few rows in each table. However if you are a researcher doing analytic queries about historical data, you will probably do infrequent, batched insertions, and your queries will scan most of the rows in a table. So a column-based approach (e.g. parquet files) is probably more suitable. Running an Oracle or Microsoft SQL Server database with 1 TB of data in the cloud is expensive. On-premise options have their challenges too. (e.g. will your laptop have enough memory even if you run the queries you want on a SQL server installed locally, connected to a slow external hard drive?) Using a more modern, column-based approach (e.g. parquet files locally or even in the cloud) will be far cheaper and gives faster query results.

The main benefit of PDR Loader over a DIY approach is that it will figure out which tables are append-only, and which are update-insert. (Or rather, you need to somehow find the configuration file that tells PDR Loader how to do this.) That is, AEMO publishes new data all the time. Sometimes the new rows should be added above/below the old rows. Sometimes the old rows should be updated. It depends on the table. (Remember, there are hundreds of tables.) If you use a DIY approach (including Nemosis), you may need to deduplicate the data when you query it. This is described later in Deduplication.

If you do want to know more about how to run PDR Batcher and Loader, and the pros and cons, check out Hachiko's guide. As they put it: "It’s not broken. However it’s also not easy."

Connectivity

As mentioned earlier, AEMO expects market participants to download these files and private files over FTP inside a private VPN. I think the public website is intended for researchers, which is fantastic. (AEMO if you are reading this, thank you so much!) Although I have never seen AEMO state this goal, or enumerate conditions of use. Please do not hammer their website (e.g. downloading data you do not need, or doing an unreasonable number of parallel downloads). I live in fear that one day they will simply turn off Nemweb.

Nemweb is hosted in AWS's Sydney region (ap-southeast-2). So if you are connecting from the cloud, choose something in Sydney. (For example, I am living in France at the moment. Downloading the data to my laptop is slow. If I spin up a server in Sydney the download is far faster, and then I connect to that server with Jupyter over SSH.)

Nemweb has both an HTTP and HTTPS interface. Note that if you are crawling the HTML file pages, sometimes the encrypted HTTPS pages contain links to unencrypted HTTP pages. I think they have fixed this now. However I am mentioning it because if you are doing this in a network environment where outbound HTTP (port 80) is blocked and HTTPS (port 443) is allowed, then you will get unexpected timeouts which cannot be resolved through retries.

File Formats

Quick File Format Explanation For Monthly MMSDM Data

One benefit of Nemosis is that it handles unzipping and dropping metadata rows and columns for you. This section contains details of how to do this yourself.

For the monthly MMSDM data that this post focuses on, the files are zips of CSVs, with one CSV per zip file. (Pandas can do the unzipping for us. Polars cannot do this yet.)

A CSV file might look like:

C,SETP.WORLD,DVD_DISPATCHPRICE,AEMO,PUBLIC,2025/10/08,09:02:27,001759878147550,MONTHLY_ARCHIVE,001759878147550
I,DISPATCH,PRICE,5,SETTLEMENTDATE,RRP
D,DISPATCH,PRICE,5,"2025/09/01 00:05:00",1.1
D,DISPATCH,PRICE,5,"2025/09/01 00:10:00",2.2
C,"END OF REPORT",4

The content you care about is really:

SETTLEMENTDATE,RRP
"2025/09/01 00:05:00",1.1
"2025/09/01 00:10:00",2.2

The CSVs have a header and a footer. (These have a different number of columns to the main data, which some CSV parsing libraries do not like.) Most CSV parsers can skip the header easily. You are unlikely to need that information. The timestamp in the header tells you the exact time when AEMO generated the data, which might be useful for some niche queries. Skipping the footer can be harder. Few libraries have a skip_footer option. With some libraries you may need to read the file as text first to count the rows, then tell it to only read the first N - 1 rows. Alternatively you can tell it that the "comment" character is C, but only if that comment option only applies from the start of the line. I tend to parse these with Polars, in which case you can tell it that the comment prefix is C,"END OF REPORT",.

After stripping the header and footer, you will also need to drop the first 4 columns. These are metadata columns. (e.g. in this example, DISPATCH and PRICE tell you that this data is for the DISPATCHPRICE table. However most such mappings are less obvious. You should figure out which file goes where from the filename if you can.) The names of these metadata columns vary between tables. Note that there may be overlapping column names. Some tables have a name for the 2nd or 3rd column (which is metadata) which is the same as the name of another column later on (which is data). Some libraries struggle to cope with this.

Here is an example of how to read these files with Pandas. This is the same approach which Nemosis uses under the hood. For the footer, it is parsed as an incomplete data row, with mostly NaNs, and then that last row dropped after reading. Since the footer is 3 columns, and we drop the first 4 columns anyway because they are metadata columns, this works.

import pandas as pd

df = pd.read_csv(
    'SOMEFILE.ZIP',  # pandas can unzip if there is only one CSV inside
    header=1, # skip first metadata row
).iloc[:-1, 4:] # drop 4 metadata columns and footer

A safer way is to use skipfooter=1, but this is far slower because it requires engine='python'.

df = pd.read_csv(
    'SOMEFILE.ZIP', # pandas can unzip if there is only one CSV inside
    header=1, # skip first metadata row
    skipfooter=1, # skip footer
    engine='python', # far slower, but required for skipfooter
).iloc[:, 4:] # drop metadata columns

You could also read the whole file once (with Pandas or something else) to count the rows, then exclude the footer with nrows.

If you prefer Polars to Pandas:

import polars as pl
import polars.selectors as cs

lf = (
    pl.scan_csv(
        'SOMEFILE.CSV', # must be unzipped beforehand
        skip_rows=1, # skip the header
        comment_prefix='C,"END OF REPORT",' # skip the footer
    )
    .select(~cs.by_index(range(4))) # drop metadata columns
)

If you get error polars.exceptions.ComputeError: CSV malformed see this workaround.

In base R we can use read.csv with the same trick as for Pandas (reading the footer row as data, and then dropping it).

df <- read.csv(
  "/tmp/example.CSV",
  skip=1, # skip header metadata
  header=TRUE, # next row has column names
)[df[[1]] != "C", -c(1:4)] # drop footer and metadata columns

To use read_csv from Tidyverse in R:

library(tidyverse)

read_csv("example.CSV", skip=1) |> 
  filter(I != "C") |> # filter out metadata footer
  select(-c(1:4)) # drop metadata columns

Note that in all of these examples the column schema is inferred from the data by the library you use, based on the first thousand rows or so. For large data this might not work. In some AEMO CSVs a particular column may have only integers for the first million rows, and then in row 1,000,001 the number has a decimal component so cannot be parsed as an integer. Many libraries (e.g. Pandas, Polars) will throw an error when they encounter this row, after spending a long time parsing all the prior rows. An easy, but slow way to resolve this is to tell your library to scan the entire file to infer data types. Otherwise you will need to hard-code the schema for that particular column or all columns.

File Format Details And Other Data Files

For other AEMO data (e.g. the daily data) the files are zips of many zips of many CSVs. (I have never seen AEMO publish zips of zips of zips. If you are programmatically unzipping nested zips, watch out for zip bombs.)

The daily files have a more complicated file structure. You should avoid parsing these unless you really want live data (within 5 minutes). In case you do, here is an example of such a file.

C,SETP.WORLD,DVD_DISPATCHPRICE,AEMO,PUBLIC,2025/10/08,09:02:27,001759878147550,MONTHLY_ARCHIVE,001759878147550
I,DISPATCH,PRICE,5,SETTLEMENTDATE,RRP
D,DISPATCH,PRICE,5,"2025/09/01 00:05:00",1.1
D,DISPATCH,PRICE,5,"2025/09/01 00:10:00",2.2
I,DISPATCH,REGIONSUM,3,REGIONID,SETTLEMENTDATE,TOTALDEMAND
D,DISPATCH,REGIONSUM,3,NSW1,"2025/09/01 00:10:00",5.6
D,DISPATCH,REGIONSUM,3,QLD1,"2025/09/01 00:10:00",7.8
D,DISPATCH,REGIONSUM,3,VIC1,"2025/09/01 00:10:00",9.0
C,"END OF REPORT",9

The interesting thing here is that the number of columns changes throughout the file, even after removing the header and footer. This is because each CSV file is actually several CSV files concatenated together. Some libraries will refuse to read this CSV at all. There are two chunks of data you want to extract:

DISPATCHPRICE:

SETTLEMENTDATE,RRP
"2025/09/01 00:05:00",1.1
"2025/09/01 00:10:00",2.2

and DISPATCHREGIONSUM:

REGIONID,SETTLEMENTDATE,TOTALDEMAND
NSW1,"2025/09/01 00:10:00",5.6
QLD1,"2025/09/01 00:10:00",7.8
VIC1,"2025/09/01 00:10:00",9.0

To process this, you must read line by line, and look at the first character, which will be C, I or D.

  • C means this is a 'control' row. Typically just the header or footer, although I have seen multiline headers in some obscure files. You should probably ignore all such lines.
  • I means this is an 'information' row. These rows contain the column headers of a new table.
  • D means this is a 'data' row. These rows contain the actual data, corresponding to column names given by the most recent I row.

Again it is worth pointing out that mapping DISPATCH and REGIONSUM to DISPATCHREGIONSUM sounds straightforward, but it is often not. Sometimes it would be something like DISPATCH_REGIONSUM, sometimes it is something completely different (especially for bids). These metadata fields are known as the REPORT_NAME and REPORT_SUB_TYPE.

The 4th column is an integer. It is somehow related to the versioning of the schemas of each table. I do not understand it exactly, and have only seen one situation in all my years of working with this data where it mattered.

The final row is used as a checksum. The integer is the number of rows in the overall file (including the header and footer). I am unsure how rows with escaped newlines are counted. I normally just ignore this. The only time I have seen a checksum mismatch was with data in this format from someone other than AEMO, who calculated the checksum excluding the footer. This checksum was put in because this system was designed back in the 1990s, when formats such as FTP could result in partial files being written to disk, and file processing commencing prior to the download finishing. With modern protocols such as HTTP over TCP, and especially with object storage such as AWS S3, getting only half a file is very unlikely. Even more so if you are splitting up your analysis script to first download all the files you want, then parse them, in the same process.

File Names

Many files have a # in the name. Note that if you are using a terminal (e.g. Bash on Linux), this will be treated as a comment. So instead of cat my#file.CSV do cat "my#file.CSV", otherwise you will get "File my not found". AEMO uses uppercase for file extensions. (.CSV and .ZIP, not .csv, .zip.)

Reusing the HTTP Connection

f you are downloading files with Requests in Python you may write something like this:

import requests

data = []
for year in range(2024, 2026):
    for month in range(12):
        url = f"https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/{year}/MMSDM_{year}_{month}/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23{year}{month}010000.zip"
        resp = requests.get(url)
        resp.raise_for_status()
        data.append(resp.json())

There's a small trick you can use to improve this: requests.Session! This looks like:

import requests

session = requests.Session()
data = []
for year in range(2024, 2026):
    for month in range(12):
        url = f"https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/{year}/MMSDM_{year}_{month}/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23{year}{month}010000.zip"
        resp = session.get(url)
        resp.raise_for_status()
        data.append(resp.json())

Reusing the HTTP connection speeds up the download and reduces AEMO's server load (so they are less likely to take down the data one day). (You should to this for other webscraping and APIs too.) I explained this in more detail here.

Retries

Nemweb's servers can be slow and unreliable. Try to add sleeps between requests, and have aggressive retries with delays and backoff. (You do not want to leave a script running overnight to download data, only to find that it gave up after 5 minutes because of a timeout.)

If you are using a requests.Session, this is simple:

from urllib3.util.retry import Retry
import requests

session = requests.session()

retries = Retry(
    total=4,
    backoff_factor=2,
    status_forcelist=[403, 429, 500, 502, 503, 504], 
    allowed_methods=["HEAD", "GET", "OPTIONS"]
)
adapter = requests.adapters.HTTPAdapter(max_retries=retries)
for protocol in ['http://', 'https://']:
    session.mount(protocol, adapter)

Note that including 403 in a retry configuration is unusual. I added this because Nemweb often returns this incorrectly as a throttling signal, instead of using the more standard 429. So sleeping and retrying is the correct response.