The tests assume that the implementation-defined margin for output lines is at least 72 characters long and contains at least 5 print zones. This should not be confused with the length of a line in the source code itself. The standard requires implementations to accept source lines up to 72 characters long If the margin is smaller than 72, the tests should still run (according to the standard), but the output will be aesthetically less pleasing.

Finally, the standard does not specify how the tests are' to be submitted to the processor for execution. Therefore, the machine-readable part of the test system consists only of source code, i.e., there are no system control commands. It is your responsibility to submit the programs to the implementation in natural way which does not violate the integrity of the tests.

4.4

Operating And Interpreting The Tests

This section will attempt to guide you through the practical aspects of using the test

programs as a tool to measure implementation conformance. The more general issues of conformance are covered in section 3, and of course in the standard itself, especially sections 1 and 2 of the ANSI document.

All of the test programs require interpretation of their behavior by the user.

As mentioned earlier, the user is an active component in the test system; the source code of the test programs is another component, subordinate to the test user. An important goal in the design of the programs was the minimization of the need for sophisticated interpretation of the test results ; but minimization is not elimination. In the best case,

the program will print out a conspicuous message indicating that the test passed or failed, and you need only interpret this message correctly. In other cases, you have to examine rather carefully the results and behavior of the program, and must apply the rules of the standard your self. This interpretation is necessary in:

2.

correct

Programs which test that termination occurs at the time (this arises in many of the exception tests)

3.

Programs for which conformance depends on the existence of adequate documentation of

implementation-defined features (both those defined in Appendix C of the standard and for any of the error tests that are accepted).

The test programs are an only partially automated solution to the

problem of determining processor conformance. Naive reliance on the test programs alone can very well lead

to

an incorrect judgment about whether

an

implementation meets the standard.

There are four types of test programs: 1) standard, 2) exception, 3)

error, and 4) informative. Within each of the functional groups (described above, section 4.2) the tests Occur in that order, although not all groups have all four types. The rules that pertain to each type follow. It

is

quite important that you

be aware of which type of test you are running and use the rules which apply to that type.

they

These tests are the ones whose title does not begin with " EXCEPTION" or "ERROR" and which generate a message about passing or failing at the end of each section. The

paragraph below on documentation describes the concept of sections of a test. Since these programs are syntactically standard and raise no exception conditions,

must be accepted and executed to completion by the implementation. If the implementation fails to do

it has failed the test. For example, if the implementation fails to recognize the key word OPTION, or if it does not accept one the numeric constants, then the test has failed. Quite obviously, it is you who must apply this rule, since the program itself won't execute at all.

this,

of

Assuming that the implementation does process the program, the next question is whether it has done so correctly. The program may be able to determine this itself, or you may have to do some active interpretation of the results. See the section below on documentation for more detail.

These tests have titles that begin with the word "EXCEPTION" and examine the behavior of the implementation when exception conditions occur during execution. Nonetheless, these programs are also standard conforming (i.e., syntactically valid) and thus the implementation must accept and process them.

There are two special considerations. The first is the distinction between So-called fatal and non-fatal exceptions. Some exceptions in the standard specify a recovery procedure which allows continued execution of the program, while others is

(the fatal exceptions) do not. If no recovery procedure is specified, the implementation must report the exception and then terminate the program. Programs testing fatal exceptions will print out a message that they are about to attempt the instruction causing the exception. If execution proceeds beyond that point, the test fails and prints a message so stating. With the non-fatal exceptions, the test program attempts to discover whether the recovery procedure has been applied or not and in this instance, the test is much like the standard tests, where the question

whether the implementation has followed the semantic rules correctly. For instance, the semantic meaning of division by zero is to report the exception, supply machine infinity, and continue. The standard, however, allows implementations

to terminate execution after even a non-fatal exception "if restrictions imposed by the hardware or operating environment make it impossible to follow the given procedures." Because it would be redundant to keep noting this allowance, the test programs do not print such a message for each non-fatal exception. Therefore when

running a test for a non-fatal exception,

note that the implementation may, under the stated circumstances, terminate the program, rather than apply the recovery procedure.

The second special consideration is that in the case of INPUT and numeric and string overflow, the precise conditions for the exception can be implementation-defined. It

is

possible, therefore, that a standard program, executing on two different standard-conforming processors, using the same data, could cause an exception in

one implementation and not in the other. The tests attempt to force the exception to occur, but it could happen, especially

in the case of string overflow, that a syntactically standard program cannot force such an exception in a given processor. The documentation accompanying the implementation under test must describe correctly those implementation-defined features upon which

the

occurrence of exceptions depends. That is, it must be possible to

find

out from the documentation whether and when overflow and INPUT exceptions will occur in the test programs.

There is a summary of the requirements for exception handling in the form of pseudo-code in section 3.2.2 (Figure 1).

These tests have titles that begin with the word "ERROR" and examine how a processor handles a non-standard program. Each of these programs contains a syntactic construction explicitly ruled out by the standard, either in the various syntax sections, or in the semantics sections. Given a

program with a syntactically non-standard construction the processor must either reject the program with a message to the user noting the reason

for rejection, or if it accepts the program, it must be accompanied by documentation which describes the interpretation

the

of

construction. Testing this requirement involves the submission of deliberately illegal programs to the processor to see if it will produce an appropriate message,

or

if it contains an enhancement of the language such as to assign a semantic meaning to the error. Thus we are faced with an interesting selection problem: out of the infinity of non-standard programs, which are worth

submitting to the processor? Three criteria seem reasonable to apply:

Test errors which we might expect would be most difficult for a processor to detect, e.g., violations of context-sensitive constraints. These are the ones ruled out by the semantics rather than syntax sections of the standard.

2.

Test errors likely to be made by beginners, for
of a two character array name.

3.

Test errors for which there may very well exist a

language enhancement, e.g., comparing strings with "<" and ">".

.

Based on these criteria, the test system contains programs for the errors in the two lists which follow. The first list is for constructions ruled out

by the semantics sections alone (these usually are instances of context-sensitive syntax constraints) and the second for plausible syntax errors ruled out by the BNF productions.

9.

number

reference to numeric-supplied-function with incorrect
of arguments

other

elements

or

spaces within keywords and
number

16.

17.

18.

conflict on number of dimensions among references: A, A(1), A(1,1)

conflict on number of dimensions between DIM and reference,
e.g.,
DIM A(20) and either A or A(2,2)

reference to subscripted variable
thereof

followed by DIMensioning

19.

multiple OPTION statements

20.

OPTION follows reference to subscripted variable

21. OPTION follows DIM

22. OPTION BASE 1 followed by DIM A(0)