Running and configuring Scientific Benchmark Tests

The Sci Benchmark testing interface for CREDO is still being developed, but essentially requires the user to write a Python script to configure and run a particular benchmark. This interface was chosen since benchmarks generally require more detailed specification and configuration than standard system tests.

Setup

An example science benchmark test is that a Rayleigh Taylor model can perform as required by the Van Keken benchmark:

As the code shows, once you set up a SciBenchmarkTest, you need to then add TestComponents that check that the model to be run actually passes some benchmark conditions. In this case, we’re checking that the Vrms output into the FrequentOutput.txt each timestep has a maximum value within a specified range, within a specified time range.

Outputs

Running the above script will take some time on most PCs, as we’ve asked for a significant number of steps at a reasonable resolution.

It should show something like the following at the terminal:

Running System test 0, with name 'RayleighTaylorBenchmark-sciBenchmarkTest':
Writing pre-test info to XML
Running the 1 modelRuns specified in the suite
Doing run 1/1 (index 0), of name 'RayleighTaylorBenchmark-sciBenchmarkTest':
ModelRun description: "Run the model needed for the benchmark."
Generating analysis XML:
Running the Model (saving results in output/RayleighTaylor-VanKekenBenchmark):
Running model 'RayleighTaylorBenchmark-sciBenchmarkTest' with command 'mpirun -np 1 /home/psunter/AuScopeCodes/stgUnderworldE-credoDev-work/build/bin/StGermain RayleighTaylorBenchmark.xml credo-analysis.xml  --elementResI=128 --elementResJ=128 --maxTimeSteps=-1 --stopTime=250 > logFile.txt' ...
Model ran successfully.
Doing post-run tidyup:
Checking test result:
Model output 'Vrms' value 0.00312004 within required range (0.0028,0.0032) for all runs.
Test result was Pass
Saved test result to output/RayleighTaylor-VanKekenBenchmark/SysTest-RayleighTaylorBenchmark-sciBenchmarkTest.xml
--------------------------------------------------------------------------------
CREDO System Tests results summary:
Ran 1 system tests, with 1 passes, 0 fails, and 0 errors
--------------------------------------------------------------------------------

and will save outputs from the run in the directory output/RayleighTaylor-VanKekenBenchmark/.

These include the following XML description of the SysTest as a record of the result:

<StgSysTest name="RayleighTaylorBenchmark-sciBenchmarkTest" status="Pass" type="SciBenchmark">
  <description>Runs a user-defined science benchmark.</description>
  <testSpecification>
    <inputFiles>
      <inputFile>RayleighTaylorBenchmark.xml</inputFile>
    </inputFiles>
    <outputPathBase>output/RayleighTaylor-VanKekenBenchmark</outputPathBase>
    <nproc>1</nproc>
    <paramOverrides>
      <param modelPath="maxTimeSteps" paramVal="-1" />
      <param modelPath="stopTime" paramVal="250" />
      <param modelPath="elementResI" paramVal="128" />
      <param modelPath="elementResJ" paramVal="128" />
    </paramOverrides>
  </testSpecification>
  <testComponents>
    <testComponent name="VRMS of first diapir" status="Pass" type="outputWithinRange">
      <specification>
        <outputName value="Vrms" />
        <reductionOp value="&lt;built-in function max&gt;" />
        <allowedRange-min value="0.0028" />
        <allowedRange-max value="0.0032" />
        <tRange-min value="200" />
        <tRange-max value="220" />
      </specification>
      <result status="Pass">
        <statusMsg>Model output 'Vrms' value  within required range
        (0.0028,0.0032) for all runs.</statusMsg>
        <actualValue>0.00312004</actualValue>
        <actualTime>213.728</actualTime>
        <withinRange>True</withinRange>
      </result>
    </testComponent>
  </testComponents>
  <testResult status="Pass">
    <statusMsg>All aspects of the benchmark passed.</statusMsg>
  </testResult>
</StgSysTest>

In future we will add capability to automatically save images of important aspects of the System test, such as a graph of VRMS against time like the one below, and compare them to expected results: