1. OVERVIEW
This document describes an International Standard for
interfaces between programming contest control systems and "validator"
programs intended to be used in conjunction with such systems.
The standard is intended as a guide for people implementing
validator programs used for "automated judging" in programming contests,
as well as for implementors of contest control systems which support
the use of such validator programs.
The standard specifies two interfaces: the invocation interface
from a contest control system to a validator program, and the result
interface from the validator back to the contest control system.
2. INVOCATION INTERFACE
A validator program must be an application (executable or
interpreted) capable of being invoked with a command line call.
The contest control system is responsible for invoking the validator
and passing it at least four command line parameters, as follows:
parameter1: a string specifying the name of the input data file which
was used to test the program whose results are being
validated. In the event that contest problem has no
input data file, the contest environment must pass an
empty string ("") as the value for this parameter.
parameter2: a string specifying the name of the output file which
was produced by the program being validated when it was
run using the data file named in parameter1 (that is,
this is the name of the file containing the output to be
'validated').
parameter3: a string specifying the name of an arbitrary "answer
file" which acts as input to the validator program.
The answer file may, but is not necessarily required
to, contain the "correct answer" for the problem. For
example, it might contain the output which was produced
by a "Judge's Solution" for the problem when run with the
data file named in parameter1 as input. Alternatively,
the "answer file" might contain information, in arbitrary
format, which instructs the validator in some way about
how to accomplish its task. The meaning of the contents
of the answer file is not defined by this standard.
If there is no "answer file" to be input to the validator,
the contest control system must pass an empty string
("") as the value of this parameter.
parameter4: a string which specifies the name of the "result file"
which the validator must produce.
To comply with this standard, a
validator MUST report its result in a file whose name
exactly matches this parameter value.
(The content of the result file produced by the
validator is defined in the following section.)
The contest control system is responsible for insuring that the data file,
the program output file, and the answer file are either found in the current
directory in which the validator program runs, or are accessible via the
environment using the name string passed to the
validator.
3. RESULT INTERFACE
The "result file" generated by a validator
must be a valid "XML Document". This means that
it must start with a valid XML Declaration, such as
<?xml version="1.0"?>
The root element of the XML document must be of the form
<result outcome = "string1"> string2 </result>
The tag name "result" is fixed and required by this standard.
The attribute name "outcome" is fixed and required by this
standard.
"string1" is an "outcome string" defining the result (outcome) which the
validator is reporting to the contest control system. If the value of
"string1" is "accepted" (or any case-variation of that word), the contest
control system is expected to assume that the validator is indicating that
the program output file "passed" the validation test(s). If "string1"
contains any other value, the contest control system is expected to
assume that the validator is indicating that the program output file
"failed" the validation test(s).
"string2" is an arbitrary message string being returned from the validator
to the contest control system. The interpretation of this string is up
to the contest control system.
The XML <result> element produced by the validator may include
other attributes in addition to the "outcome" attribute, and may also
include additional (nested) elements. The interpretation of any such
additional attributes and/or elements is up to the contest control
system. The contest control system must be able to ignore any additional
attributes and/or elements it does not wish to process while at the same
time properly processing the remainder of the <result> element.
4. OPTIONAL VALUES
The contest control system may pass additional
command line parameters to a validator, as long as the first four
command line parameters are specified as listed above. The meaning of
any additional command line parameters is not defined by this standard;
the interpretation of such parameters is up to the validator. A validator
must be capable of accepting an arbitrary number of additional parameters
even if it chooses to ignore them (that is, the mere presence of
additional parameters may not cause a validator to function incorrectly).
Likewise, a validator can return additional information to the
contest control system, in the form of additional attributes and/or
nested elements in the XML <result> element in the result file.
The meaning of any additional attributes or elements is not defined by
this standard; the interpretation of such additional attributes/elements
is up to the contest control system. As long as the XML document in
the result file is well-formed XML, a contest control system must be
capable of accepting an arbitrary number of additional attributes and/or
elements in the <result> element, even if it chooses to ignore them
(that is, the mere presence of additional attributes and/or elements may
not cause a contest control system to be unable to correctly interpret
the <result> element.
The optional command parameters and return attributes/elements can
be used to implement a variety of powerful features in a system. For
example, to insure that the "result file" processed by the contest control
system was generated by the most recent invocation of a validator, the
contest control system can pass a special "key" value as an extra command
line parameter, and the validator can be written to return this key back
in its result file. In this case the validator would accept the key
value from the command line parameter and return an XML <result>
element that contains a special "security" attribute whose value was
the received key. In this case, the XML result file might look like:
<xml version="1.0"?>
<result outcome="accepted" security="inputKey"> MessageString2 </result>
The contest control system could then compare the security attribute
value "inputKey" with the value which it passed to the validator when the
validator was invoked, verifying that they matched. Again, this example
represents the use of optional features not required by the standard.
*end standard*
Mail comments or questions to pc2@ecs.csus.edu
Revised: Sun Aug 17 19:06:32 PDT 2008