Add support to the testing framework to test for expected failures
Add support to the
Tester tool to enable testing for expected error conditions.
For example, if a required input value is not provided, verify that this case is handled and the code fails with a reasonable error message that highlights the problem.
The suggestion for addressing this is to add an additional "metric" to the
Tester tool, e.g.
MetricError, enabling the developer to specify, when a given executable is run with a given input, that a particular error message is generated. For example:
<test> <description>...</description> <executable>...</executable> <parameters>...</parameters> <files> ... </files> <metrics> <metric type="error" id="1"> <value type="fatal">RequiredMethod must be specified in the input file.</value> </metric> ... </metrics> </test>
This will require the tester to be able to accept a non-zero return code from the executable in cases where an error is expected.