Test Trial 3

Will Holmgren, Oct 29, 2020

Test Trial 3 will test day ahead and hour ahead forecasts for AC power, mean SURFRAD GHI, and probabilistic forecasts of GHI. This trial will run on Solar Forecast Arbiter version 1.0rc4.

We ask that users not upload real forecasts for this test of the trial system. Rather, users should upload non-proprietary, random, constant, or other types of forecast that can be made public. This will allow all of parties to the testing to trace and debug errors without security concerns. To the extent that it’s feasible to program within your systems, we recommend running your systems as if you were making a real forecast but then replacing the final data with fake data before posting it to the Arbiter’s API.

Please see the Trials Documentation for a complete description of the Solar Forecast Arbiter’s trials feature. The documentation also contains an example Python script to help you get started programming against the API, but we encourage you to write your own scripts that mimic how you would handle real forecast data from your systems.

The following forecast start, end, and timing parameters are the same for all forecasts:

• Start time: 2020-11-10 19:00Z
• End time: 2020-11-12 19:00Z
• Forecast parameters:
  • 60 minutes ahead of the hour, 60 minute intervals:
    • Issue time: 00:00
    • Lead time to start: 60 minutes
    • Interval length: 60 minutes
    • Interval value type: mean
    • Run length: 60 minutes
    • Interval label: ending
  • Day ahead:
    • Issue time: 19Z
    • Lead time to start: 24 hours
    • Interval length: 1 hour
    • Interval value type: mean
    • Interval label: ending
    • Run length: 24 hours

The rules for metadata and data accessibility for each participant are as follows.

• Daily reports during the trial:
  • Accessible: Raw and processed observation data, reference forecast data, participant forecast data, metrics of all participants
  • Not Accessible: Time series data of other participants, identities of participants, association of anonymous user names and identities
• The final report:
  • Accessible: Raw and processed observation data, reference forecast data, participant forecast data, metrics of all participants, time series data of all participants
  • Not Accessible: Identities of participants, association of anonymous user names and identities

The following 3 sets of metadata describe the observations, metrics, categories, quality flags, missing forecast policy, and reference forecasts.

Deterministic:

• Observation:
  • UO SRML Bend OR PV AC Power
• Metrics:
  • MAE, MBE, RMSE, Skill
• Categories:
  • Total, daily, hour of day
• Quality filters to exclude:
  • flag = (ISNAN, USER FLAGGED, DAYTIME STALE VALUES, DAYTIME INTERPOLATED VALUES), discard before resample = True, resample threshold = 10%
• Missing forecasts:
  • Filled with 0
• Reference forecast:
  • Hour ahead: Persistence of clear sky index
  • Day ahead: GFS

Aggregate:

• Observation:
  • Average of all SURFRAD GHI
• Metrics:
  • MAE, MBE, RMSE
• Categories:
  • Total, daily, hour of day
• Quality filters to exclude:
  • flag = (ISNAN, USER FLAGGED), discard before resample = True, resample threshold = 20%
• Missing forecasts:
  • Filled with 0
• Reference forecast:
  • None

Probabilistic:

• Observation:
  • Table Mountain Boulder CO ghi
• Probabilistic Forecast Thresholds:
  • Prob(obs <= 50 W/m^2)
  • Prob(obs <= 500 W/m^2)
• Metrics:
  • BS, REL, RES, UNC
• Categories:
  • Total, daily, hour of day
• Quality filters to exclude:
  • flag = (ISNAN, USER FLAGGED), discard before resample = True, resample threshold = 10%
  • flag = NIGHTTIME, discard before resample = False, resample threshold = 10%
• Missing forecasts:
  • Filled with 0
• Reference forecast:
  • None

Please contact Will Holmgren at holmgren@email.arizona.edu with comments, questions, or to sign up for the trials testing.