Skip to content

How to Preprocess the Data?

Parsing Timestamps

Most timestamps are in the format %Y/%m/%D %H:%M:%S, e.g. "2025/11/10 19:30:00". AEMO data never contains just a date. Where something is logically a date not a datetime, it will appear as a datetime at midnight at the start of that day.

Special characters

DUID is the identifier for a generator. Some DUIDs contain funny characters, such as W/HOE#2 for Wivenhoe Power Station. This slash and hash can cause errors with misleading error messages. (e.g. if you are saving files to disk with hive partitioning on the DUID key, you may accidentally create an additional nested folder called HOE#2 within parent folder W.)

There are a few obscure tables which contain newline characters within the cell values. These are escaped with double quotes around the whole value. These are in tables which researchers do not normally look at, such as the Market Suspension Notices in MARKETNOTICEDATA. Most CSV parsing libraries can handle this. I am mentioning this just in case you are trying to read the data with something simple like:

with open('data.CSV', 'r') as f:
    headers = f.readline().strip().split(',')
    for data_row in f:
        cells = data_row.strip().split(',')

If you do that, it will work for the vast majority of tables, but not MARKETNOTICEDATA.

Although AEMO has started to base64-encode these strings, to avoid CSV parsing challenges.

Constraint identifiers also have many funny characters (e.g. #, >)

Running Out of Memory With Polars

Sometimes when you query data with Polars you may run out of memory, even when the query seems small. To prevent this:

  • Pass low_memory=True to scan_parquet()
  • Even if the resulting dataframe is small enough to just call .collect(), stream to a Parquet file on disk, then read it back. For some reason this may use far less memory. This is particularly useful for intermediate queries where you just use .head() instead of an aggregation. (Polars is still a new library. They made some changes to their streaming engine since I discovered this trick.)
  • Add swap. (On Linux you can even create a file in a normal folder and register it as swap memory.)

Warning

Vertical scaling (switching to a larger computer) might not prevent you from running out of memory.

Polars uses all CPU cores by default. If the bigger computer has more cores, each core consumes memory, so the total memory usage will go up and you may still run out of memory.

You can tell Polars to use fewer cores with the POLARS_MAX_THREADS environment variable.

:::