Add support to the testing framework to test for expected failures
Add support to the Tester
tool to enable testing for expected error conditions.
For example, if a required input value is not provided, verify that this case is handled and the code fails with a reasonable error message that highlights the problem.
The suggestion for addressing this is to add an additional "metric" to the Tester
tool, e.g. MetricError
, enabling the developer to specify, when a given executable is run with a given input, that a particular error message is generated. For example:
<test>
<description>...</description>
<executable>...</executable>
<parameters>...</parameters>
<files>
...
</files>
<metrics>
<metric type="error" id="1">
<value type="fatal">RequiredMethod must be specified in the input file.</value>
</metric>
...
</metrics>
</test>
This will require the tester to be able to accept a non-zero return code from the executable in cases where an error is expected.