Skip to content

TRADINGREGIONSUM

TRADINGREGIONSUM sets out the half-hourly average regional demand and frequency control services. TRADINGREGIONSUM includes fields for the Raise Regulation and Lower Regulation Ancillary Services plus improvements to demand calculations.

Deprecation Warning

This table is in the HISTORICAL TABLES package, so is likely deprecated.

TRADINGREGIONSUM is public data, and is available to all participants.

Source

TRADINGREGIONSUM is updated every 30 minutes.

Tip

1 second, 6 second, 5 minute, 60 second and regulation refer to types of FCAS markets.

Column Schema

PERIODID

Trading interval identifier within settlement day.

SQL Type: NUMBER(3,0) Polars Type: pl.Int16

This column is mandatory, and a primary key.

Examples: 1, 48

REGIONID

Region Identifier

For more information, see the dedicated documentation page.

The SQL type is VARCHAR2(10). This is actually a factor/enum, with values:

  • QLD1

  • VIC1

  • TAS1

  • NSW1

  • SA1

  • SNOWY1

This column is mandatory, and a primary key.

RUNNO

Dispatch run no.

SQL Type: NUMBER(3,0) Polars Type: pl.UInt16

This column is mandatory, and a primary key.

Example: 1

SETTLEMENTDATE

Date that this data applies to

Warning

Despite what the name suggests, this is a datetime, not a date.

SQL Type: DATE Polars Type: pl.Datetime()

This column is mandatory, and a primary key.

Example: 2021/09/01 00:30:00

AVAILABLEGENERATION

The available generation in the Region for the interval

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 2010.81, 2445, 13496.75, 12941

CLEAREDSUPPLY

Sum of cleared generation and import for region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 698.2, 991, 9080.9, 8142

DEMAND_AND_NONSCHEDGEN

Sum of Cleared Scheduled generation, imported generation (at the region boundary) and allowances made for non-scheduled generation (MW).

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 710.82, 1253, 9130.59, 8490

DEMANDFORECAST

Forecast demand for region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: -0.04, 0.01, -2, 18

DISPATCHABLEGENERATION

Averaged generation dispatched in region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 584.21, 932, 8847.84, 8141

DISPATCHABLELOAD

Averaged load dispatched in region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.02, 0, 1, 311

EXCESSGENERATION

Average excess generation in region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Example: 0

INITIALSUPPLY

Sum of initial generation and import for region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 713.9, 1042, 9019.49, 6858

LASTCHANGED

Last date and time record changed

For more information, see the dedicated documentation page.

SQL Type: DATE Polars Type: pl.Datetime()

This column is not mandatory.

Example: 2021/09/01 00:25:04

LOWER5MINLOCALDISPATCH

Lower 5 min local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.5, 0, 1, 102

LOWER6SECLOCALDISPATCH

Lower 6 sec local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 1.33, 0, 5, 129

LOWER60SECLOCALDISPATCH

Lower 60 sec local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.53, 0, 5, 170

LOWERREGLOCALDISPATCH

Lower Regulation local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.83, 0, 1, 91

NETINTERCHANGE

Average energy transferred over interconnector

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: -0.23, 2.14, -40, 178

RAISE5MINLOCALDISPATCH

Raise 5 min local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.21, 0, 6, 203

RAISE6SECLOCALDISPATCH

Raise 6 sec local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 7.47, 18, 312.28, 289

RAISE60SECLOCALDISPATCH

Raise 60 sec local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 2.04, 1, 276.82, 258

RAISEREGLOCALDISPATCH

Raise Regulation local dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.33, 0, 3, 112

TOTALDEMAND

Total demand for region

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 657.05, 636, 9068.73, 7042

TOTALINTERMITTENTGENERATION

Allowance made for non-scheduled generation in the demand forecast (MW).

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.86, 3, 426.37, 379

UIGF

Regional aggregated Unconstrained Intermittent Generation Forecast of Semi-scheduled generation (MW).

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 16.83, 267, 2762.96, 1574

AVAILABLELOAD

Not used

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

Examples: 8.33, 0, 5, 311

LOWER5MINDISPATCH

Not used since Dec 2003. Lower 5 min MW dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINIMPORT

Not used since Dec 2003. Lower 5 min MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINLOCALPRICE

Not used since Dec 2003. Local price of lower 5 min

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINLOCALREQ

Not used since Dec 2003. Lower 5 min local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Lower 5 min local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINPRICE

Not used since Dec 2003. Regional price of lower 5 min

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINREQ

Not used since Dec 2003. Lower 5 min total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINSUPPLYPRICE

Not used since Dec 2003. Supply price of lower 5 min

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER5MINVIOLATION

Not used since Dec 2003. Violation (MW) of Lower 5 min requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECDISPATCH

Not used since Dec 2003. Lower 6 sec MW dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECIMPORT

Not used since Dec 2003. Lower 6 sec MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECLOCALPRICE

Not used since Dec 2003. Local price of lower 6 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECLOCALREQ

Not used since Dec 2003. Lower 6 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Lower 6 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECPRICE

Not used since Dec 2003. Regional price of lower 6 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECREQ

Not used since Dec 2003. Lower 6 sec total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECSUPPLYPRICE

Not used since Dec 2003. Supply price of lower 6 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER6SECVIOLATION

Not used since Dec 2003. Violation (MW) of Lower 6 seconds requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECDISPATCH

Not used since Dec 2003. Lower 60 sec MW dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECIMPORT

Not used since Dec 2003. Lower 60 sec MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECLOCALPRICE

Not used since Dec 2003. Local price of lower 60 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECLOCALREQ

Not used since Dec 2003. Lower 60 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Lower 60 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECPRICE

Not used since Dec 2003. Regional price of lower 60 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECREQ

Not used since Dec 2003. Lower 60 sec total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECSUPPLYPRICE

Not used since Dec 2003. Supply price of lower 60 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWER60SECVIOLATION

Not used since Dec 2003. Violation (MW) of Lower 60 seconds requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWERREGIMPORT

Not used since Dec 2003. Lower Regulation MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWERREGLOCALREQ

Not used since Dec 2003. Lower Regulation local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWERREGLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Lower Reg local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWERREGREQ

Not used since Dec 2003. Lower Regulation total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

LOWERREGVIOLATION

Not used since Dec 2003. Violation (MW) of Lower Reg requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINDISPATCH

Not used since Dec 2003. Raise 5 min MW dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINIMPORT

Not used since Dec 2003. Raise 5 min MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINLOCALPRICE

Not used since Dec 2003. Local price of raise 5 min

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINLOCALREQ

Not used since Dec 2003. Raise 5 min local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Raise 5 min local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINPRICE

Not used since Dec 2003. Regional price of raise 5 min

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINREQ

Not used since Dec 2003. Raise 5 min total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINSUPPLYPRICE

Not used since Dec 2003. Supply price of raise 5 min

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE5MINVIOLATION

Not used since Dec 2003. Violation (MW) of Raise 5 min requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECDISPATCH

Not used since Dec 2003. Raise 6 sec MW dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECIMPORT

Not used since Dec 2003. Raise 6 sec MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECLOCALPRICE

Not used since Dec 2003. Local price of raise 6 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECLOCALREQ

Not used since Dec 2003. Raise 6 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Raise 6 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECPRICE

Not used since Dec 2003. Regional price of raise 6 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECREQ

Not used since Dec 2003. Raise 6 sec total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECSUPPLYPRICE

Not used since Dec 2003. Supply price of raise 6 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE6SECVIOLATION

Not used since Dec 2003. Violation (MW) of Raise 6 seconds requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECDISPATCH

Not used since Dec 2003. Raise 60 sec MW dispatch

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECIMPORT

Not used since Dec 2003. Raise 60 sec MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECLOCALPRICE

Not used since Dec 2003. Local price of raise 60 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECLOCALREQ

Not used since Dec 2003. Raise 60 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Raise 60 sec local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECPRICE

Not used since Dec 2003. Regional price of raise 60 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECREQ

Not used since Dec 2003. Raise 60 sec total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECSUPPLYPRICE

Not used since Dec 2003. Supply price of raise 60 sec

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISE60SECVIOLATION

Not used since Dec 2003. Violation (MW) of Raise 60 seconds requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISEREGIMPORT

Not used since Dec 2003. Raise Regulation MW imported

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISEREGLOCALREQ

Not used since Dec 2003. Raise Regulation local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISEREGLOCALVIOLATION

Not used since Dec 2003. Violation (MW) of Raise Reg local requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISEREGREQ

Not used since Dec 2003. Raise Regulation total requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

RAISEREGVIOLATION

Not used since Dec 2003. Violation (MW) of Raise Reg requirement

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is unused.

New Data Timing

Every 30 minutes

Code Examples

Polars

Casting is enums, categoricals etc is not essential, but may speed up parsing. Swapping int for pl.UInt16, pl.Int32 would also speed up parsing. However some of the datatypes are inferred, so they may be wrong.

This code deduplicates for you. Deduplication is normally only required across multiple months.

This code assumes that all columns are present, which may not be true for older files.

import polars as pl
import polars.selectors as cs

# e.g. "2029/03/15 00:05:00"
dt_s_fmt = "%Y/%m/%d %H:%M:%S"

csv_paths = ["SOMEFILE.CSV", "OTHERFILE.CSV"]

lf = (
    pl.concat(
        [
            pl.scan_csv(
                csv_path,  # must be unzipped beforehand
                skip_rows=1,  # skip the header
                infer_schema=False,  # much faster
            )
            .filter(cs.by_index(0) == "D")  # drop the footer
            .select(~cs.by_index(range(4)))  # drop metadata columns
            for csv_path in csv_paths
        ],
        how="diagonal_relaxed",
    )
    .with_columns(
        pl.col("SETTLEMENTDATE", "LASTCHANGED").str.strptime(pl.Datetime, dt_s_fmt),
    )
    .cast(
        {
            "PERIODID": int,  # or pl.Int16
            "REGIONID": pl.Enum(["QLD1", "VIC1", "TAS1", "NSW1", "SA1", "SNOWY1"]),
            "RUNNO": int,  # or pl.UInt16
            "AVAILABLEGENERATION": pl.Float64,
            "CLEAREDSUPPLY": pl.Float64,
            "DEMAND_AND_NONSCHEDGEN": pl.Float64,
            "DEMANDFORECAST": pl.Float64,
            "DISPATCHABLEGENERATION": pl.Float64,
            "DISPATCHABLELOAD": pl.Float64,
            "EXCESSGENERATION": pl.Float64,
            "INITIALSUPPLY": pl.Float64,
            "LOWER5MINLOCALDISPATCH": pl.Float64,
            "LOWER6SECLOCALDISPATCH": pl.Float64,
            "LOWER60SECLOCALDISPATCH": pl.Float64,
            "LOWERREGLOCALDISPATCH": pl.Float64,
            "NETINTERCHANGE": pl.Float64,
            "RAISE5MINLOCALDISPATCH": pl.Float64,
            "RAISE6SECLOCALDISPATCH": pl.Float64,
            "RAISE60SECLOCALDISPATCH": pl.Float64,
            "RAISEREGLOCALDISPATCH": pl.Float64,
            "TOTALDEMAND": pl.Float64,
            "TOTALINTERMITTENTGENERATION": pl.Float64,
            "UIGF": pl.Float64,
            "AVAILABLELOAD": pl.Float64,
            "LOWER5MINDISPATCH": pl.Float64,
            "LOWER5MINIMPORT": pl.Float64,
            "LOWER5MINLOCALPRICE": pl.Float64,
            "LOWER5MINLOCALREQ": pl.Float64,
            "LOWER5MINLOCALVIOLATION": pl.Float64,
            "LOWER5MINPRICE": pl.Float64,
            "LOWER5MINREQ": pl.Float64,
            "LOWER5MINSUPPLYPRICE": pl.Float64,
            "LOWER5MINVIOLATION": pl.Float64,
            "LOWER6SECDISPATCH": pl.Float64,
            "LOWER6SECIMPORT": pl.Float64,
            "LOWER6SECLOCALPRICE": pl.Float64,
            "LOWER6SECLOCALREQ": pl.Float64,
            "LOWER6SECLOCALVIOLATION": pl.Float64,
            "LOWER6SECPRICE": pl.Float64,
            "LOWER6SECREQ": pl.Float64,
            "LOWER6SECSUPPLYPRICE": pl.Float64,
            "LOWER6SECVIOLATION": pl.Float64,
            "LOWER60SECDISPATCH": pl.Float64,
            "LOWER60SECIMPORT": pl.Float64,
            "LOWER60SECLOCALPRICE": pl.Float64,
            "LOWER60SECLOCALREQ": pl.Float64,
            "LOWER60SECLOCALVIOLATION": pl.Float64,
            "LOWER60SECPRICE": pl.Float64,
            "LOWER60SECREQ": pl.Float64,
            "LOWER60SECSUPPLYPRICE": pl.Float64,
            "LOWER60SECVIOLATION": pl.Float64,
            "LOWERREGIMPORT": pl.Float64,
            "LOWERREGLOCALREQ": pl.Float64,
            "LOWERREGLOCALVIOLATION": pl.Float64,
            "LOWERREGREQ": pl.Float64,
            "LOWERREGVIOLATION": pl.Float64,
            "RAISE5MINDISPATCH": pl.Float64,
            "RAISE5MINIMPORT": pl.Float64,
            "RAISE5MINLOCALPRICE": pl.Float64,
            "RAISE5MINLOCALREQ": pl.Float64,
            "RAISE5MINLOCALVIOLATION": pl.Float64,
            "RAISE5MINPRICE": pl.Float64,
            "RAISE5MINREQ": pl.Float64,
            "RAISE5MINSUPPLYPRICE": pl.Float64,
            "RAISE5MINVIOLATION": pl.Float64,
            "RAISE6SECDISPATCH": pl.Float64,
            "RAISE6SECIMPORT": pl.Float64,
            "RAISE6SECLOCALPRICE": pl.Float64,
            "RAISE6SECLOCALREQ": pl.Float64,
            "RAISE6SECLOCALVIOLATION": pl.Float64,
            "RAISE6SECPRICE": pl.Float64,
            "RAISE6SECREQ": pl.Float64,
            "RAISE6SECSUPPLYPRICE": pl.Float64,
            "RAISE6SECVIOLATION": pl.Float64,
            "RAISE60SECDISPATCH": pl.Float64,
            "RAISE60SECIMPORT": pl.Float64,
            "RAISE60SECLOCALPRICE": pl.Float64,
            "RAISE60SECLOCALREQ": pl.Float64,
            "RAISE60SECLOCALVIOLATION": pl.Float64,
            "RAISE60SECPRICE": pl.Float64,
            "RAISE60SECREQ": pl.Float64,
            "RAISE60SECSUPPLYPRICE": pl.Float64,
            "RAISE60SECVIOLATION": pl.Float64,
            "RAISEREGIMPORT": pl.Float64,
            "RAISEREGLOCALREQ": pl.Float64,
            "RAISEREGLOCALVIOLATION": pl.Float64,
            "RAISEREGREQ": pl.Float64,
            "RAISEREGVIOLATION": pl.Float64,
        }
    )
    # Drop deprecated/unused columns
    .drop(
        "AVAILABLELOAD",
        "LOWER5MINDISPATCH",
        "LOWER5MINIMPORT",
        "LOWER5MINLOCALPRICE",
        "LOWER5MINLOCALREQ",
        "LOWER5MINLOCALVIOLATION",
        "LOWER5MINPRICE",
        "LOWER5MINREQ",
        "LOWER5MINSUPPLYPRICE",
        "LOWER5MINVIOLATION",
        "LOWER6SECDISPATCH",
        "LOWER6SECIMPORT",
        "LOWER6SECLOCALPRICE",
        "LOWER6SECLOCALREQ",
        "LOWER6SECLOCALVIOLATION",
        "LOWER6SECPRICE",
        "LOWER6SECREQ",
        "LOWER6SECSUPPLYPRICE",
        "LOWER6SECVIOLATION",
        "LOWER60SECDISPATCH",
        "LOWER60SECIMPORT",
        "LOWER60SECLOCALPRICE",
        "LOWER60SECLOCALREQ",
        "LOWER60SECLOCALVIOLATION",
        "LOWER60SECPRICE",
        "LOWER60SECREQ",
        "LOWER60SECSUPPLYPRICE",
        "LOWER60SECVIOLATION",
        "LOWERREGIMPORT",
        "LOWERREGLOCALREQ",
        "LOWERREGLOCALVIOLATION",
        "LOWERREGREQ",
        "LOWERREGVIOLATION",
        "RAISE5MINDISPATCH",
        "RAISE5MINIMPORT",
        "RAISE5MINLOCALPRICE",
        "RAISE5MINLOCALREQ",
        "RAISE5MINLOCALVIOLATION",
        "RAISE5MINPRICE",
        "RAISE5MINREQ",
        "RAISE5MINSUPPLYPRICE",
        "RAISE5MINVIOLATION",
        "RAISE6SECDISPATCH",
        "RAISE6SECIMPORT",
        "RAISE6SECLOCALPRICE",
        "RAISE6SECLOCALREQ",
        "RAISE6SECLOCALVIOLATION",
        "RAISE6SECPRICE",
        "RAISE6SECREQ",
        "RAISE6SECSUPPLYPRICE",
        "RAISE6SECVIOLATION",
        "RAISE60SECDISPATCH",
        "RAISE60SECIMPORT",
        "RAISE60SECLOCALPRICE",
        "RAISE60SECLOCALREQ",
        "RAISE60SECLOCALVIOLATION",
        "RAISE60SECPRICE",
        "RAISE60SECREQ",
        "RAISE60SECSUPPLYPRICE",
        "RAISE60SECVIOLATION",
        "RAISEREGIMPORT",
        "RAISEREGLOCALREQ",
        "RAISEREGLOCALVIOLATION",
        "RAISEREGREQ",
        "RAISEREGVIOLATION",
    )
    # Deduplicate
    # We want one row per ['PERIODID', 'REGIONID', 'RUNNO', 'SETTLEMENTDATE']
    # Tie-breaker: greatest value of ['LASTCHANGED']
    .sort(["LASTCHANGED"], descending=False)
    .unique(subset=["PERIODID", "REGIONID", "RUNNO", "SETTLEMENTDATE"], keep="last")
)

Nemosis

from nemosis import dynamic_data_compiler

df = dynamic_data_compiler(
    '2024/01/01 00:00:00',
    '2024/01/31 00:00:00',
    "TRADINGREGIONSUM",
    "data/"
)

Nemosis only includes a subset of columns by default, as an optimisation. (i.e. the commonly used columns.) To include others:

from nemosis import defaults

defaults.table_columns["TRADINGREGIONSUM"] += ["PERIODID", "RUNNO", "LASTCHANGED", "LOWER5MINDISPATCH", "LOWER5MINIMPORT", "LOWER5MINLOCALPRICE", "LOWER5MINLOCALREQ", "LOWER5MINLOCALVIOLATION", "LOWER5MINPRICE", "LOWER5MINREQ", "LOWER5MINSUPPLYPRICE", "LOWER5MINVIOLATION", "LOWER6SECDISPATCH", "LOWER6SECIMPORT", "LOWER6SECLOCALPRICE", "LOWER6SECLOCALREQ", "LOWER6SECLOCALVIOLATION", "LOWER6SECPRICE", "LOWER6SECREQ", "LOWER6SECSUPPLYPRICE", "LOWER6SECVIOLATION", "LOWER60SECDISPATCH", "LOWER60SECIMPORT", "LOWER60SECLOCALPRICE", "LOWER60SECLOCALREQ", "LOWER60SECLOCALVIOLATION", "LOWER60SECPRICE", "LOWER60SECREQ", "LOWER60SECSUPPLYPRICE", "LOWER60SECVIOLATION", "LOWERREGIMPORT", "LOWERREGLOCALREQ", "LOWERREGLOCALVIOLATION", "LOWERREGREQ", "LOWERREGVIOLATION", "RAISE5MINDISPATCH", "RAISE5MINIMPORT", "RAISE5MINLOCALPRICE", "RAISE5MINLOCALREQ", "RAISE5MINLOCALVIOLATION", "RAISE5MINPRICE", "RAISE5MINREQ", "RAISE5MINSUPPLYPRICE", "RAISE5MINVIOLATION", "RAISE6SECDISPATCH", "RAISE6SECIMPORT", "RAISE6SECLOCALPRICE", "RAISE6SECLOCALREQ", "RAISE6SECLOCALVIOLATION", "RAISE6SECPRICE", "RAISE6SECREQ", "RAISE6SECSUPPLYPRICE", "RAISE6SECVIOLATION", "RAISE60SECDISPATCH", "RAISE60SECIMPORT", "RAISE60SECLOCALPRICE", "RAISE60SECLOCALREQ", "RAISE60SECLOCALVIOLATION", "RAISE60SECPRICE", "RAISE60SECREQ", "RAISE60SECSUPPLYPRICE", "RAISE60SECVIOLATION", "RAISEREGIMPORT", "RAISEREGLOCALREQ", "RAISEREGLOCALVIOLATION", "RAISEREGREQ", "RAISEREGVIOLATION"]
df = dynamic_data_compiler(...)

Example File

From PUBLIC_DVD_TRADINGREGIONSUM_202109010000.zip

SETTLEMENTDATERUNNOREGIONIDPERIODIDTOTALDEMANDAVAILABLEGENERATIONAVAILABLELOADDEMANDFORECASTDISPATCHABLEGENERATIONDISPATCHABLELOADNETINTERCHANGEEXCESSGENERATIONLOWER5MINDISPATCHLOWER5MINIMPORTLOWER5MINLOCALDISPATCHLOWER5MINLOCALPRICELOWER5MINLOCALREQLOWER5MINPRICELOWER5MINREQLOWER5MINSUPPLYPRICELOWER60SECDISPATCHLOWER60SECIMPORTLOWER60SECLOCALDISPATCHLOWER60SECLOCALPRICELOWER60SECLOCALREQLOWER60SECPRICELOWER60SECREQLOWER60SECSUPPLYPRICELOWER6SECDISPATCHLOWER6SECIMPORTLOWER6SECLOCALDISPATCHLOWER6SECLOCALPRICELOWER6SECLOCALREQLOWER6SECPRICELOWER6SECREQLOWER6SECSUPPLYPRICERAISE5MINDISPATCHRAISE5MINIMPORTRAISE5MINLOCALDISPATCHRAISE5MINLOCALPRICERAISE5MINLOCALREQRAISE5MINPRICERAISE5MINREQRAISE5MINSUPPLYPRICERAISE60SECDISPATCHRAISE60SECIMPORTRAISE60SECLOCALDISPATCHRAISE60SECLOCALPRICERAISE60SECLOCALREQRAISE60SECPRICERAISE60SECREQRAISE60SECSUPPLYPRICERAISE6SECDISPATCHRAISE6SECIMPORTRAISE6SECLOCALDISPATCHRAISE6SECLOCALPRICERAISE6SECLOCALREQRAISE6SECPRICERAISE6SECREQRAISE6SECSUPPLYPRICELASTCHANGEDINITIALSUPPLYCLEAREDSUPPLYLOWERREGIMPORTLOWERREGLOCALDISPATCHLOWERREGLOCALREQLOWERREGREQRAISEREGIMPORTRAISEREGLOCALDISPATCHRAISEREGLOCALREQRAISEREGREQRAISE5MINLOCALVIOLATIONRAISEREGLOCALVIOLATIONRAISE60SECLOCALVIOLATIONRAISE6SECLOCALVIOLATIONLOWER5MINLOCALVIOLATIONLOWERREGLOCALVIOLATIONLOWER60SECLOCALVIOLATIONLOWER6SECLOCALVIOLATIONRAISE5MINVIOLATIONRAISEREGVIOLATIONRAISE60SECVIOLATIONRAISE6SECVIOLATIONLOWER5MINVIOLATIONLOWERREGVIOLATIONLOWER60SECVIOLATIONLOWER6SECVIOLATIONTOTALINTERMITTENTGENERATIONDEMAND_AND_NONSCHEDGENUIGF
2021/09/01 00:30:001NSW117747.8611367.520-48.627779.11031.2409.1171.733.81153.46211.09249.462021/09/01 00:25:047812.147758.529.8351.5266.077824.57612.52
2021/09/01 00:30:001QLD115475.868723.090-32.175705.360229.50085.9885.9822.4481.9281.922021/09/01 00:25:045524.395479.931.4246.26665545.90169.09
2021/09/01 00:30:001SA111489.092270.4105.33-9.871093.2115.67-411.54091.5104.67114.34129.67150.76172.42021/09/01 00:25:041518.91515.793.2796.0610.421526.12156.4
2021/09/01 00:30:001TAS111099.452464.70-3.211578.550479.1052.8934.6621.431.5118.992021/09/01 00:25:041112.531119.5538.2317.5160.091279.64389.7
2021/09/01 00:30:001VIC114669.697199.70-40.044423.650-246.04066.6565.2747.5162.56178.67100.672021/09/01 00:25:044743.454706.2317.2510.3366.374772.60396.7
2021/09/01 01:00:001NSW127634.7211644.170-34.057737.80103.08010.8872.9248.41167243.77272.452021/09/01 00:55:057684.517643.4828.3566.4157.867701.34569.17
2021/09/01 01:00:001QLD125348.338726.670-20.435540.180191.850084.3484.3418.6775.4275.422021/09/01 00:55:055375.755349.3652.4251.6766.145415.50166.67
2021/09/01 01:00:001SA121494.862243.9105.33-12.851106.117.67-406.43088.36105115.33125142.13159.222021/09/01 00:55:051523.821523.1670.2290.677.561530.72129.9
2021/09/01 01:00:001TAS121081.342460.760-2.341560.440479.1054.456.326.7661192021/09/01 00:55:051095.851101.4440.6813149.541250.98385.76
2021/09/01 01:00:001VIC124545.497082.150-24.474255.150-290.34065.534430135.96156.4592.672021/09/01 00:55:054615.494582.2118.331151.124633.33304.15
2021/09/01 01:30:001NSW137427.5511596.180-45.657563.780136.22016.9275.3356.04164244.31271.582021/09/01 01:25:047484.817436.0838.564.8630.517466.59521.18
...

CSETP.WORLDDVD_TRADINGREGIONSUMAEMOPUBLIC2021/10/0703:45:0600000003503539490000000350353744
ITRADINGREGIONSUM4SETTLEMENTDATERUNNOREGIONIDPERIODIDTOTALDEMANDAVAILABLEGENERATIONAVAILABLELOADDEMANDFORECASTDISPATCHABLEGENERATIONDISPATCHABLELOADNETINTERCHANGEEXCESSGENERATIONLOWER5MINDISPATCHLOWER5MINIMPORTLOWER5MINLOCALDISPATCHLOWER5MINLOCALPRICELOWER5MINLOCALREQLOWER5MINPRICELOWER5MINREQLOWER5MINSUPPLYPRICELOWER60SECDISPATCHLOWER60SECIMPORTLOWER60SECLOCALDISPATCHLOWER60SECLOCALPRICELOWER60SECLOCALREQLOWER60SECPRICELOWER60SECREQLOWER60SECSUPPLYPRICELOWER6SECDISPATCHLOWER6SECIMPORTLOWER6SECLOCALDISPATCHLOWER6SECLOCALPRICELOWER6SECLOCALREQLOWER6SECPRICELOWER6SECREQLOWER6SECSUPPLYPRICERAISE5MINDISPATCHRAISE5MINIMPORTRAISE5MINLOCALDISPATCHRAISE5MINLOCALPRICERAISE5MINLOCALREQRAISE5MINPRICERAISE5MINREQRAISE5MINSUPPLYPRICERAISE60SECDISPATCHRAISE60SECIMPORTRAISE60SECLOCALDISPATCHRAISE60SECLOCALPRICERAISE60SECLOCALREQRAISE60SECPRICERAISE60SECREQRAISE60SECSUPPLYPRICERAISE6SECDISPATCHRAISE6SECIMPORTRAISE6SECLOCALDISPATCHRAISE6SECLOCALPRICERAISE6SECLOCALREQRAISE6SECPRICERAISE6SECREQRAISE6SECSUPPLYPRICELASTCHANGEDINITIALSUPPLYCLEAREDSUPPLYLOWERREGIMPORTLOWERREGLOCALDISPATCHLOWERREGLOCALREQLOWERREGREQRAISEREGIMPORTRAISEREGLOCALDISPATCHRAISEREGLOCALREQRAISEREGREQRAISE5MINLOCALVIOLATIONRAISEREGLOCALVIOLATIONRAISE60SECLOCALVIOLATIONRAISE6SECLOCALVIOLATIONLOWER5MINLOCALVIOLATIONLOWERREGLOCALVIOLATIONLOWER60SECLOCALVIOLATIONLOWER6SECLOCALVIOLATIONRAISE5MINVIOLATIONRAISEREGVIOLATIONRAISE60SECVIOLATIONRAISE6SECVIOLATIONLOWER5MINVIOLATIONLOWERREGVIOLATIONLOWER60SECVIOLATIONLOWER6SECVIOLATIONTOTALINTERMITTENTGENERATIONDEMAND_AND_NONSCHEDGENUIGF
DTRADINGREGIONSUM42021/09/01 00:30:001NSW117747.8611367.520-48.627779.11031.2409.1171.733.81153.46211.09249.462021/09/01 00:25:047812.147758.529.8351.5266.077824.57612.52
DTRADINGREGIONSUM42021/09/01 00:30:001QLD115475.868723.090-32.175705.360229.50085.9885.9822.4481.9281.922021/09/01 00:25:045524.395479.931.4246.26665545.90169.09
DTRADINGREGIONSUM42021/09/01 00:30:001SA111489.092270.4105.33-9.871093.2115.67-411.54091.5104.67114.34129.67150.76172.42021/09/01 00:25:041518.91515.793.2796.0610.421526.12156.4
DTRADINGREGIONSUM42021/09/01 00:30:001TAS111099.452464.70-3.211578.550479.1052.8934.6621.431.5118.992021/09/01 00:25:041112.531119.5538.2317.5160.091279.64389.7
DTRADINGREGIONSUM42021/09/01 00:30:001VIC114669.697199.70-40.044423.650-246.04066.6565.2747.5162.56178.67100.672021/09/01 00:25:044743.454706.2317.2510.3366.374772.60396.7
DTRADINGREGIONSUM42021/09/01 01:00:001NSW127634.7211644.170-34.057737.80103.08010.8872.9248.41167243.77272.452021/09/01 00:55:057684.517643.4828.3566.4157.867701.34569.17
DTRADINGREGIONSUM42021/09/01 01:00:001QLD125348.338726.670-20.435540.180191.850084.3484.3418.6775.4275.422021/09/01 00:55:055375.755349.3652.4251.6766.145415.50166.67
DTRADINGREGIONSUM42021/09/01 01:00:001SA121494.862243.9105.33-12.851106.117.67-406.43088.36105115.33125142.13159.222021/09/01 00:55:051523.821523.1670.2290.677.561530.72129.9
DTRADINGREGIONSUM42021/09/01 01:00:001TAS121081.342460.760-2.341560.440479.1054.456.326.7661192021/09/01 00:55:051095.851101.4440.6813149.541250.98385.76
DTRADINGREGIONSUM42021/09/01 01:00:001VIC124545.497082.150-24.474255.150-290.34065.534430135.96156.4592.672021/09/01 00:55:054615.494582.2118.331151.124633.33304.15
DTRADINGREGIONSUM42021/09/01 01:30:001NSW137427.5511596.180-45.657563.780136.22016.9275.3356.04164244.31271.582021/09/01 01:25:047484.817436.0838.564.8630.517466.59521.18
...
CEND OF REPORT1003

SETTLEMENTDATE,RUNNO,REGIONID,PERIODID,TOTALDEMAND,AVAILABLEGENERATION,AVAILABLELOAD,DEMANDFORECAST,DISPATCHABLEGENERATION,DISPATCHABLELOAD,NETINTERCHANGE,EXCESSGENERATION,LOWER5MINDISPATCH,LOWER5MINIMPORT,LOWER5MINLOCALDISPATCH,LOWER5MINLOCALPRICE,LOWER5MINLOCALREQ,LOWER5MINPRICE,LOWER5MINREQ,LOWER5MINSUPPLYPRICE,LOWER60SECDISPATCH,LOWER60SECIMPORT,LOWER60SECLOCALDISPATCH,LOWER60SECLOCALPRICE,LOWER60SECLOCALREQ,LOWER60SECPRICE,LOWER60SECREQ,LOWER60SECSUPPLYPRICE,LOWER6SECDISPATCH,LOWER6SECIMPORT,LOWER6SECLOCALDISPATCH,LOWER6SECLOCALPRICE,LOWER6SECLOCALREQ,LOWER6SECPRICE,LOWER6SECREQ,LOWER6SECSUPPLYPRICE,RAISE5MINDISPATCH,RAISE5MINIMPORT,RAISE5MINLOCALDISPATCH,RAISE5MINLOCALPRICE,RAISE5MINLOCALREQ,RAISE5MINPRICE,RAISE5MINREQ,RAISE5MINSUPPLYPRICE,RAISE60SECDISPATCH,RAISE60SECIMPORT,RAISE60SECLOCALDISPATCH,RAISE60SECLOCALPRICE,RAISE60SECLOCALREQ,RAISE60SECPRICE,RAISE60SECREQ,RAISE60SECSUPPLYPRICE,RAISE6SECDISPATCH,RAISE6SECIMPORT,RAISE6SECLOCALDISPATCH,RAISE6SECLOCALPRICE,RAISE6SECLOCALREQ,RAISE6SECPRICE,RAISE6SECREQ,RAISE6SECSUPPLYPRICE,LASTCHANGED,INITIALSUPPLY,CLEAREDSUPPLY,LOWERREGIMPORT,LOWERREGLOCALDISPATCH,LOWERREGLOCALREQ,LOWERREGREQ,RAISEREGIMPORT,RAISEREGLOCALDISPATCH,RAISEREGLOCALREQ,RAISEREGREQ,RAISE5MINLOCALVIOLATION,RAISEREGLOCALVIOLATION,RAISE60SECLOCALVIOLATION,RAISE6SECLOCALVIOLATION,LOWER5MINLOCALVIOLATION,LOWERREGLOCALVIOLATION,LOWER60SECLOCALVIOLATION,LOWER6SECLOCALVIOLATION,RAISE5MINVIOLATION,RAISEREGVIOLATION,RAISE60SECVIOLATION,RAISE6SECVIOLATION,LOWER5MINVIOLATION,LOWERREGVIOLATION,LOWER60SECVIOLATION,LOWER6SECVIOLATION,TOTALINTERMITTENTGENERATION,DEMAND_AND_NONSCHEDGEN,UIGF
"2021/09/01 00:30:00",1,NSW1,1,7747.86,11367.52,0,-48.62,7779.11,0,31.24,0,,,9.11,,,,,,,,71.7,,,,,,,,33.81,,,,,,,,153.46,,,,,,,,211.09,,,,,,,,249.46,,,,,,"2021/09/01 00:25:04",7812.14,7758.5,,29.83,,,,51.52,,,,,,,,,,,,,,,,,,,66.07,7824.57,612.52
"2021/09/01 00:30:00",1,QLD1,1,5475.86,8723.09,0,-32.17,5705.36,0,229.5,0,,,0,,,,,,,,85.98,,,,,,,,85.98,,,,,,,,22.44,,,,,,,,81.92,,,,,,,,81.92,,,,,,"2021/09/01 00:25:04",5524.39,5479.9,,31.42,,,,46.26,,,,,,,,,,,,,,,,,,,66,5545.90,169.09
"2021/09/01 00:30:00",1,SA1,1,1489.09,2270.4,105.33,-9.87,1093.21,15.67,-411.54,0,,,91.5,,,,,,,,104.67,,,,,,,,114.34,,,,,,,,129.67,,,,,,,,150.76,,,,,,,,172.4,,,,,,"2021/09/01 00:25:04",1518.9,1515.7,,93.27,,,,96.06,,,,,,,,,,,,,,,,,,,10.42,1526.12,156.4
"2021/09/01 00:30:00",1,TAS1,1,1099.45,2464.7,0,-3.21,1578.55,0,479.1,0,,,52.89,,,,,,,,34.66,,,,,,,,21.43,,,,,,,,1.5,,,,,,,,1,,,,,,,,18.99,,,,,,"2021/09/01 00:25:04",1112.53,1119.55,,38.23,,,,17.5,,,,,,,,,,,,,,,,,,,160.09,1279.64,389.7
"2021/09/01 00:30:00",1,VIC1,1,4669.69,7199.7,0,-40.04,4423.65,0,-246.04,0,,,66.65,,,,,,,,65.27,,,,,,,,47.5,,,,,,,,162.56,,,,,,,,178.67,,,,,,,,100.67,,,,,,"2021/09/01 00:25:04",4743.45,4706.23,,17.25,,,,10.33,,,,,,,,,,,,,,,,,,,66.37,4772.60,396.7
"2021/09/01 01:00:00",1,NSW1,2,7634.72,11644.17,0,-34.05,7737.8,0,103.08,0,,,10.88,,,,,,,,72.92,,,,,,,,48.41,,,,,,,,167,,,,,,,,243.77,,,,,,,,272.45,,,,,,"2021/09/01 00:55:05",7684.51,7643.48,,28.35,,,,66.41,,,,,,,,,,,,,,,,,,,57.86,7701.34,569.17
"2021/09/01 01:00:00",1,QLD1,2,5348.33,8726.67,0,-20.43,5540.18,0,191.85,0,,,0,,,,,,,,84.34,,,,,,,,84.34,,,,,,,,18.67,,,,,,,,75.42,,,,,,,,75.42,,,,,,"2021/09/01 00:55:05",5375.75,5349.36,,52.42,,,,51.67,,,,,,,,,,,,,,,,,,,66.14,5415.50,166.67
"2021/09/01 01:00:00",1,SA1,2,1494.86,2243.9,105.33,-12.85,1106.1,17.67,-406.43,0,,,88.36,,,,,,,,105,,,,,,,,115.33,,,,,,,,125,,,,,,,,142.13,,,,,,,,159.22,,,,,,"2021/09/01 00:55:05",1523.82,1523.16,,70.22,,,,90.67,,,,,,,,,,,,,,,,,,,7.56,1530.72,129.9
"2021/09/01 01:00:00",1,TAS1,2,1081.34,2460.76,0,-2.34,1560.44,0,479.1,0,,,54.4,,,,,,,,56.3,,,,,,,,26.76,,,,,,,,6,,,,,,,,1,,,,,,,,19,,,,,,"2021/09/01 00:55:05",1095.85,1101.44,,40.68,,,,13,,,,,,,,,,,,,,,,,,,149.54,1250.98,385.76
"2021/09/01 01:00:00",1,VIC1,2,4545.49,7082.15,0,-24.47,4255.15,0,-290.34,0,,,65.53,,,,,,,,44,,,,,,,,30,,,,,,,,135.96,,,,,,,,156.45,,,,,,,,92.67,,,,,,"2021/09/01 00:55:05",4615.49,4582.21,,18.33,,,,11,,,,,,,,,,,,,,,,,,,51.12,4633.33,304.15
"2021/09/01 01:30:00",1,NSW1,3,7427.55,11596.18,0,-45.65,7563.78,0,136.22,0,,,16.92,,,,,,,,75.33,,,,,,,,56.04,,,,,,,,164,,,,,,,,244.31,,,,,,,,271.58,,,,,,"2021/09/01 01:25:04",7484.81,7436.08,,38.5,,,,64.86,,,,,,,,,,,,,,,,,,,30.51,7466.59,521.18
...

C,SETP.WORLD,DVD_TRADINGREGIONSUM,AEMO,PUBLIC,2021/10/07,03:45:06,0000000350353949,,0000000350353744
I,TRADING,REGIONSUM,4,SETTLEMENTDATE,RUNNO,REGIONID,PERIODID,TOTALDEMAND,AVAILABLEGENERATION,AVAILABLELOAD,DEMANDFORECAST,DISPATCHABLEGENERATION,DISPATCHABLELOAD,NETINTERCHANGE,EXCESSGENERATION,LOWER5MINDISPATCH,LOWER5MINIMPORT,LOWER5MINLOCALDISPATCH,LOWER5MINLOCALPRICE,LOWER5MINLOCALREQ,LOWER5MINPRICE,LOWER5MINREQ,LOWER5MINSUPPLYPRICE,LOWER60SECDISPATCH,LOWER60SECIMPORT,LOWER60SECLOCALDISPATCH,LOWER60SECLOCALPRICE,LOWER60SECLOCALREQ,LOWER60SECPRICE,LOWER60SECREQ,LOWER60SECSUPPLYPRICE,LOWER6SECDISPATCH,LOWER6SECIMPORT,LOWER6SECLOCALDISPATCH,LOWER6SECLOCALPRICE,LOWER6SECLOCALREQ,LOWER6SECPRICE,LOWER6SECREQ,LOWER6SECSUPPLYPRICE,RAISE5MINDISPATCH,RAISE5MINIMPORT,RAISE5MINLOCALDISPATCH,RAISE5MINLOCALPRICE,RAISE5MINLOCALREQ,RAISE5MINPRICE,RAISE5MINREQ,RAISE5MINSUPPLYPRICE,RAISE60SECDISPATCH,RAISE60SECIMPORT,RAISE60SECLOCALDISPATCH,RAISE60SECLOCALPRICE,RAISE60SECLOCALREQ,RAISE60SECPRICE,RAISE60SECREQ,RAISE60SECSUPPLYPRICE,RAISE6SECDISPATCH,RAISE6SECIMPORT,RAISE6SECLOCALDISPATCH,RAISE6SECLOCALPRICE,RAISE6SECLOCALREQ,RAISE6SECPRICE,RAISE6SECREQ,RAISE6SECSUPPLYPRICE,LASTCHANGED,INITIALSUPPLY,CLEAREDSUPPLY,LOWERREGIMPORT,LOWERREGLOCALDISPATCH,LOWERREGLOCALREQ,LOWERREGREQ,RAISEREGIMPORT,RAISEREGLOCALDISPATCH,RAISEREGLOCALREQ,RAISEREGREQ,RAISE5MINLOCALVIOLATION,RAISEREGLOCALVIOLATION,RAISE60SECLOCALVIOLATION,RAISE6SECLOCALVIOLATION,LOWER5MINLOCALVIOLATION,LOWERREGLOCALVIOLATION,LOWER60SECLOCALVIOLATION,LOWER6SECLOCALVIOLATION,RAISE5MINVIOLATION,RAISEREGVIOLATION,RAISE60SECVIOLATION,RAISE6SECVIOLATION,LOWER5MINVIOLATION,LOWERREGVIOLATION,LOWER60SECVIOLATION,LOWER6SECVIOLATION,TOTALINTERMITTENTGENERATION,DEMAND_AND_NONSCHEDGEN,UIGF
D,TRADING,REGIONSUM,4,"2021/09/01 00:30:00",1,NSW1,1,7747.86,11367.52,0,-48.62,7779.11,0,31.24,0,,,9.11,,,,,,,,71.7,,,,,,,,33.81,,,,,,,,153.46,,,,,,,,211.09,,,,,,,,249.46,,,,,,"2021/09/01 00:25:04",7812.14,7758.5,,29.83,,,,51.52,,,,,,,,,,,,,,,,,,,66.07,7824.57,612.52
D,TRADING,REGIONSUM,4,"2021/09/01 00:30:00",1,QLD1,1,5475.86,8723.09,0,-32.17,5705.36,0,229.5,0,,,0,,,,,,,,85.98,,,,,,,,85.98,,,,,,,,22.44,,,,,,,,81.92,,,,,,,,81.92,,,,,,"2021/09/01 00:25:04",5524.39,5479.9,,31.42,,,,46.26,,,,,,,,,,,,,,,,,,,66,5545.90,169.09
D,TRADING,REGIONSUM,4,"2021/09/01 00:30:00",1,SA1,1,1489.09,2270.4,105.33,-9.87,1093.21,15.67,-411.54,0,,,91.5,,,,,,,,104.67,,,,,,,,114.34,,,,,,,,129.67,,,,,,,,150.76,,,,,,,,172.4,,,,,,"2021/09/01 00:25:04",1518.9,1515.7,,93.27,,,,96.06,,,,,,,,,,,,,,,,,,,10.42,1526.12,156.4
D,TRADING,REGIONSUM,4,"2021/09/01 00:30:00",1,TAS1,1,1099.45,2464.7,0,-3.21,1578.55,0,479.1,0,,,52.89,,,,,,,,34.66,,,,,,,,21.43,,,,,,,,1.5,,,,,,,,1,,,,,,,,18.99,,,,,,"2021/09/01 00:25:04",1112.53,1119.55,,38.23,,,,17.5,,,,,,,,,,,,,,,,,,,160.09,1279.64,389.7
D,TRADING,REGIONSUM,4,"2021/09/01 00:30:00",1,VIC1,1,4669.69,7199.7,0,-40.04,4423.65,0,-246.04,0,,,66.65,,,,,,,,65.27,,,,,,,,47.5,,,,,,,,162.56,,,,,,,,178.67,,,,,,,,100.67,,,,,,"2021/09/01 00:25:04",4743.45,4706.23,,17.25,,,,10.33,,,,,,,,,,,,,,,,,,,66.37,4772.60,396.7
D,TRADING,REGIONSUM,4,"2021/09/01 01:00:00",1,NSW1,2,7634.72,11644.17,0,-34.05,7737.8,0,103.08,0,,,10.88,,,,,,,,72.92,,,,,,,,48.41,,,,,,,,167,,,,,,,,243.77,,,,,,,,272.45,,,,,,"2021/09/01 00:55:05",7684.51,7643.48,,28.35,,,,66.41,,,,,,,,,,,,,,,,,,,57.86,7701.34,569.17
D,TRADING,REGIONSUM,4,"2021/09/01 01:00:00",1,QLD1,2,5348.33,8726.67,0,-20.43,5540.18,0,191.85,0,,,0,,,,,,,,84.34,,,,,,,,84.34,,,,,,,,18.67,,,,,,,,75.42,,,,,,,,75.42,,,,,,"2021/09/01 00:55:05",5375.75,5349.36,,52.42,,,,51.67,,,,,,,,,,,,,,,,,,,66.14,5415.50,166.67
D,TRADING,REGIONSUM,4,"2021/09/01 01:00:00",1,SA1,2,1494.86,2243.9,105.33,-12.85,1106.1,17.67,-406.43,0,,,88.36,,,,,,,,105,,,,,,,,115.33,,,,,,,,125,,,,,,,,142.13,,,,,,,,159.22,,,,,,"2021/09/01 00:55:05",1523.82,1523.16,,70.22,,,,90.67,,,,,,,,,,,,,,,,,,,7.56,1530.72,129.9
D,TRADING,REGIONSUM,4,"2021/09/01 01:00:00",1,TAS1,2,1081.34,2460.76,0,-2.34,1560.44,0,479.1,0,,,54.4,,,,,,,,56.3,,,,,,,,26.76,,,,,,,,6,,,,,,,,1,,,,,,,,19,,,,,,"2021/09/01 00:55:05",1095.85,1101.44,,40.68,,,,13,,,,,,,,,,,,,,,,,,,149.54,1250.98,385.76
D,TRADING,REGIONSUM,4,"2021/09/01 01:00:00",1,VIC1,2,4545.49,7082.15,0,-24.47,4255.15,0,-290.34,0,,,65.53,,,,,,,,44,,,,,,,,30,,,,,,,,135.96,,,,,,,,156.45,,,,,,,,92.67,,,,,,"2021/09/01 00:55:05",4615.49,4582.21,,18.33,,,,11,,,,,,,,,,,,,,,,,,,51.12,4633.33,304.15
D,TRADING,REGIONSUM,4,"2021/09/01 01:30:00",1,NSW1,3,7427.55,11596.18,0,-45.65,7563.78,0,136.22,0,,,16.92,,,,,,,,75.33,,,,,,,,56.04,,,,,,,,164,,,,,,,,244.31,,,,,,,,271.58,,,,,,"2021/09/01 01:25:04",7484.81,7436.08,,38.5,,,,64.86,,,,,,,,,,,,,,,,,,,30.51,7466.59,521.18
...
C,"END OF REPORT",1003

Availability

Visibility: Public

This table is in the public monthly MMSDM archive.

I have collated the list of all those files into one place, in this CSV metadata file. (Columns year, month, url, compressed_size in bytes).

The earliest available data is month 1 of 2015. The most recent available data is month 9 of 2021.

Every month in between has a file. (No gaps.)

Data Size

Each year of data is approximately 6 MB when zipped, 63 MB as unzipped CSV.

Deduplication and PDR Loader Configuration

The primary keys are:

The transaction type is "insert", which means that you will usually not see duplicate data. If you do, you can deduplicate based on the primary keys, without sorting first. Rows from different files with the same primary keys should have the same values.

PDR Loader configuration is:

Report Name Report Subtype Version
TRADING REGIONSUM 4

This means that the CSV files start with metadata columns:

I,TRADING,REGIONSUM,4,...
D,TRADING,REGIONSUM,4,...

Indexes

This page as JSON (schema documented here, but may change)

Download a JSON Schema specification for this table.

Official AEMO schema documentation (multi-page):

Other Information

Package: HISTORICAL TABLES