INTERMITTENT_GEN_SCADA

INTERMITTENT_GEN_SCADA provides the SCADA Availability for every intermittent generating unit, including Elements Available (wind turbines/solar inverters) and Local Limit

Column Schema

DUID

Dispatchable Unit Identifier

Warning

This column may contain funny characters such as / (slash) or # (hash). This may break hive-style partitioning.

For more information, see the dedicated documentation page.

SQL Type: VARCHAR2(20) Polars Type: pl.Categorical

This column is mandatory, and a primary key.

Examples:

ADPPV1
ALDGASF1
ARWF1
AVLSF1
BALDHWF1
BANGOWF1
BANGOWF2
BANN1
BERYLSF1
BLUEGSF1

RUN_DATETIME

Date Time of the dispatch interval (interval ending)

SQL Type: DATE Polars Type: pl.Datetime()

This column is mandatory, and a primary key.

Example: 2025/04/11 10:35:00

SCADA_TYPE

SCADA snapshot for intermittent generating unit at start of interval for a specified SCADA signal type. ELAV = Total Elements Available (# turbines for wind farms, # inverters for solar farms); LOCL = Local Limit (MW).

SQL Type: VARCHAR2(20) Polars Type: pl.Categorical

This column is mandatory, and a primary key.

Examples:

ELAV
LOCL

LASTCHANGED

Last date and time record changed

For more information, see the dedicated documentation page.

SQL Type: DATE Polars Type: pl.Datetime()

This column is not mandatory.

Example: 2025/04/12 04:11:34

SCADA_QUALITY

SCADA quality snapshot for intermittent generating unit at start of interval for a specified SCADA signal type.

SQL Type: VARCHAR2(20) Polars Type: str

This column is not mandatory.

Examples:

Good
ManuallyReplaced
NotInService
Suspect
Uninitialised
Unreasonable

SCADA_VALUE

SCADA value snapshot for intermittent generating unit at start of interval for a specified SCADA signal type.

SQL Type: NUMBER(15,5) Polars Type: pl.Float64

This column is not mandatory.

Examples: 0.41305, 0, 2, 1076

New Data Timing

Shows data for every 5 minutes for all Intermittent Generating Units

Code Examples

Polars

This table is so large that you should explicitly enumerate and cast every column, which will speed things up and prevents some errors. Although some of the datatypes are inferred, so they may be wrong. In particular, UInt16 vs UInt8 vs Int16 etc. If you get errors, you can swap these for just int.

This code deduplicates for you. Deduplication is normally only required across multiple months.

This code assumes that all columns are present, which may not be true for older files.

import polars as pl
import polars.selectors as cs

# e.g. "2029/03/15 00:05:00"
dt_s_fmt = "%Y/%m/%d %H:%M:%S"

csv_paths = ["SOMEFILE.CSV", "OTHERFILE.CSV"]

lf = (
    pl.concat(
        [
            pl.scan_csv(
                csv_path,  # must be unzipped beforehand
                skip_rows=1,  # skip the header
                infer_schema=False,  # much faster
            )
            .filter(cs.by_index(0) == "D")  # drop the footer
            .select(~cs.by_index(range(4)))  # drop metadata columns
            for csv_path in csv_paths
        ],
        how="diagonal_relaxed",
    )
    .with_columns(
        pl.col("RUN_DATETIME", "LASTCHANGED").str.strptime(pl.Datetime, dt_s_fmt),
    )
    .cast(
        {
            "DUID": pl.Categorical,
            "SCADA_TYPE": pl.Categorical,
            "SCADA_QUALITY": str,
            "SCADA_VALUE": pl.Float64,
        }
    )
    # Deduplicate
    # These files are so large that this step may cause your laptop to run out of memory.
    # We want one row per ['DUID', 'RUN_DATETIME', 'SCADA_TYPE']
    # Tie-breaker: greatest value of LASTCHANGED
    .sort(LASTCHANGED, descending=False)
    .unique(subset=["DUID", "RUN_DATETIME", "SCADA_TYPE"], keep="last")
)

Nemosis

from nemosis import dynamic_data_compiler

df = dynamic_data_compiler(
    '2024/01/01 00:00:00',
    '2024/01/31 00:00:00',
    "INTERMITTENT_GEN_SCADA",
    "data/"
)

Nemosis only includes a subset of columns by default, as an optimisation. (i.e. the commonly used columns.) To include others:

from nemosis import defaults

defaults.table_columns["INTERMITTENT_GEN_SCADA"] += ["LASTCHANGED"]
df = dynamic_data_compiler(...)

Example File

From PUBLIC_ARCHIVE_INTERMITTENT_GEN_SCADA_FILE02_202609010000.zip

Table without MetadataTable with MetadataRaw Text without Metadata

RUN_DATETIME	DUID	SCADA_TYPE	SCADA_VALUE	SCADA_QUALITY	LASTCHANGED
2025/04/11 10:35:00	BNGSF1	ELAV	54	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BNGSF1	LOCL	110	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BNGSF2	ELAV	54	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BNGSF2	LOCL	110	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BOCORWF1	ELAV	66	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BOCORWF1	LOCL	110.88	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BODWF1	ELAV	27.99969	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BODWF1	LOCL	119.99547	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BOMENSF1	ELAV	44	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BOMENSF1	LOCL	100.00061	Good	2025/04/12 04:11:34
2025/04/11 10:35:00	BOWWPV1	ELAV	3	Good	2025/04/12 04:11:34
...

C	SETP.WORLD	DVD_INTERMITTENT_GEN_SCADA	AEMO	PUBLIC	2026/10/09	18:32:01	001791534721978	MONTHLY_ARCHIVE	001791534721978
I	DEMAND	INTERMITTENT_GEN_SCADA	1	RUN_DATETIME	DUID	SCADA_TYPE	SCADA_VALUE	SCADA_QUALITY	LASTCHANGED
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BNGSF1	ELAV	54	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BNGSF1	LOCL	110	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BNGSF2	ELAV	54	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BNGSF2	LOCL	110	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BOCORWF1	ELAV	66	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BOCORWF1	LOCL	110.88	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BODWF1	ELAV	27.99969	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BODWF1	LOCL	119.99547	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BOMENSF1	ELAV	44	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BOMENSF1	LOCL	100.00061	Good	2025/04/12 04:11:34
D	DEMAND	INTERMITTENT_GEN_SCADA	1	2025/04/11 10:35:00	BOWWPV1	ELAV	3	Good	2025/04/12 04:11:34
...
C	END OF REPORT	1003

RUN_DATETIME,DUID,SCADA_TYPE,SCADA_VALUE,SCADA_QUALITY,LASTCHANGED
"2025/04/11 10:35:00",BNGSF1,ELAV,54,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BNGSF1,LOCL,110,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BNGSF2,ELAV,54,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BNGSF2,LOCL,110,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BOCORWF1,ELAV,66,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BOCORWF1,LOCL,110.88,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BODWF1,ELAV,27.99969,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BODWF1,LOCL,119.99547,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BOMENSF1,ELAV,44,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BOMENSF1,LOCL,100.00061,Good,"2025/04/12 04:11:34"
"2025/04/11 10:35:00",BOWWPV1,ELAV,3,Good,"2025/04/12 04:11:34"
...

Raw Text with Metadata

C,SETP.WORLD,DVD_INTERMITTENT_GEN_SCADA,AEMO,PUBLIC,2026/10/09,18:32:01,001791534721978,MONTHLY_ARCHIVE,001791534721978
I,DEMAND,INTERMITTENT_GEN_SCADA,1,RUN_DATETIME,DUID,SCADA_TYPE,SCADA_VALUE,SCADA_QUALITY,LASTCHANGED
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BNGSF1,ELAV,54,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BNGSF1,LOCL,110,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BNGSF2,ELAV,54,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BNGSF2,LOCL,110,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BOCORWF1,ELAV,66,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BOCORWF1,LOCL,110.88,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BODWF1,ELAV,27.99969,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BODWF1,LOCL,119.99547,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BOMENSF1,ELAV,44,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BOMENSF1,LOCL,100.00061,Good,"2025/04/12 04:11:34"
D,DEMAND,INTERMITTENT_GEN_SCADA,1,"2025/04/11 10:35:00",BOWWPV1,ELAV,3,Good,"2025/04/12 04:11:34"
...
C,"END OF REPORT",1003

Availability

Visibility: Private & Public Next-Day

This table is in the public monthly MMSDM archive.

I have collated the list of all those files into one place, in this CSV metadata file. (Columns year, month, url, compressed_size in bytes).

Tip

There are multiple files per month, because this is such a large table.

The earliest available data is month 12 of 2024. The most recent available data is month 9 of 2026.

Every month in between has at least one file.

Data Size

Each year of data is approximately 6 GB when zipped, 60 GB as unzipped CSV.

Tip

This table is large. Simply calling pd.read_csv() might fill up all the memory on your laptop. (As a rule of thumb, you need double the amount of memory as the size of the CSV.) You will need to think hard about memory optimisation. e.g. consider using Polars instead of Pandas, and selecting only the columns you care about.

Deduplication and PDR Loader Configuration

The primary keys are:

DUID
RUN_DATETIME
SCADA_TYPE

The transaction type is "insert", which means that you will usually not see duplicate data. If you do, you can deduplicate based on the primary keys, without sorting first. Rows from different files with the same primary keys should have the same values.

PDR Loader configuration is:

Report Name	Report Subtype	Version
DEMAND	INTERMITTENT_GEN_SCADA	1

This means that the CSV files start with metadata columns:

I,DEMAND,INTERMITTENT_GEN_SCADA,1,...
D,DEMAND,INTERMITTENT_GEN_SCADA,1,...

Links

This page as JSON (schema documented here, but may change)

Download a JSON Schema specification for this table.

Official AEMO schema documentation

Other Information

Package: DEMAND_FORECASTS