Trials Testing

Will Holmgren, Feb 12, 2020

This post may be updated based on stakeholder feedback.

The Solar Forecast Arbiter now supports anonymous operational forecast trials. The team has run a series of internal tests of this feature. Because of the importance of this feature, we’re requesting that forecast users and forecast vendors help us test the feature to ensure that it operates smoothly and meets their needs.

We propose a series of short forecast trials described below. We expect that trials may need to be rerun to accommodate errors on our side or unexpected difficulties for users.

We ask that users not upload real forecasts for these initial beta-tests of the trial system. Rather, users should upload non-proprietary, random, constant, or other types of forecast that can be made public. This will allow all of parties to the testing to trace and debug errors without security concerns. To the extent that it’s feasible to program within your systems, we recommend running your systems as if you were making a real forecast but then replacing the final data with fake data before posting it to the Arbiter’s API.

Before starting a trial, users should be familiar with posting forecast data to the Arbiter’s API. We expect the API to be relatively straightforward to program against. The API documentation is available at https://api.solarforecastarbiter.org/. The development team has also created a script and accompanying blog post to help testers spin up as quickly as possible.

It is worth noting that work done in this testing phase can be reused in the future for a real trial and thus has benefit to stakeholders too. In recognition of your help in this critical beta-testing, we will provide any support needed.

Trial 1:

Given the forecast parameters, one forecast would be uploaded by 2020-06-03 19:00Z with an hourly timerange of [2020-06-04 19:00Z, 2020-06-05 19:00Z). The second (and final) forecast would be uploaded at 2020-06-04 19:00Z covering the time range [2020-06-04 19:00Z, 2020-06-06 19:00Z). The API will restrict forecast uploads to be valid forecasts given the parameters, meaning it will not allow forecasts to be POSTed after each issue time. Please make sure to take this into account by, for example, scheduling the forecast job to run at 18:50Z.

Trial 2:

Given the forecast parameters, the first forecast would be uploaded by 2020-07-29 18:00Z with an hourly timerange of [2020-07-29 19:00Z, 2020-07-29 20:00Z). The second forecast would be uploaded by 2020-07-29 19:00Z covering the time range [2020-07-29 20:00Z, 2020-07-29 21:00Z). And so on through the final forecast issue time of 2020-07-31 17:00Z. The API will restrict forecast uploads to be valid forecasts given the parameters, meaning it will not allow forecasts to be POSTed after each issue time. Please make sure to take this into account by, for example, scheduling the forecast job to run at 18:50Z. Analysis reports will be recomputed at 3Z each day of the trial and may not reflect the latest forecasts until the latest observation data is available in the Arbiter.

Trial 3:

Test trial 3 is described in a separate post