Page images
PDF
EPUB

feature in a simple way unlikely to interact with the feature under test so as to mask errors in its own implementation.

4.2

we

Hierarchical Organization Of The Tests

rule, This

Within the constraints imposed by the test-before-use tried to group together functionally related tests. grouping should also help you interpret the tests better since you can usually concentrate on one part of the standard at a time, even if the parts themselves are not in order. Section 6.1 of this manual contains a summary of the hierarchical group structure. It relates a functional subject to a sequential range of tests and also to the corresponding sections of the standard. We strongly recommend that you read the relevant sections of the standard carefully before running the tests in a particular group. The documentation contained herein explains the rationale for the tests in each group, but it is not a substitute for a detailed understanding of the standard itself.

broken

into

Many of the individual test programs are themselves further down so-called sections. Thus the overall hierarchical subdivision scheme is given by, from largest to smallest: system, groups, sub-groups, programs, sections. Program sections are further discussed below under: 4.4.3

[blocks in formation]

The test programs are oriented towards executing in an interactive environment, but generally can be but generally can be run in batch mode as well. Some of the programs do require input, however, and these present more of a problem, since the input needed often depends on the immediately preceding output of the program. See the sample output in Volume 2 for help in setting up data files if you plan to run all the programs non-interactively. The programs which use the INPUT statement are 73, 81, 84, 107-113, and 203.

We have tried to keep the storage required for execution within reasonable bounds. Array sizes are as small as small as possible, consistent with adequate testing. No program exceeds 300 lines in length. The programs print many informative messages which may be changed without affecting the outcome of the tests. If your implementation cannot handle a program because of its size, you should set up a temporary copy of the the program with the informative messages cut down to a minimum and use that version. Be careful not to omit printing which is a substantive part of the test itself.

the

The tests assume that the implementation-defined margin for output lines is at least 72 characters long and contains at least 5 print zones. This should not be confused with the length of line in source code itself. The standard requires implementations to accept source lines up to 72 characters long. If the margin is smaller than 72, the tests should still run (according to the standard), but the output will be aesthetically less pleasing.

be

Finally, the standard does not specify how the tests are to submitted to the processor for execution. Therefore, the machine-readable part of the test system consists only of source code, i.e., there are no system control commands. It is your responsibility to submit the programs to the implementation in natural way which does not violate the integrity of the tests.

a

[blocks in formation]

interpretation

of their

All of the test programs require behavior by the user. As mentioned earlier, the user is an active component in the test system; the source code of the test programs is another component, subordinate to the test user. An important goal in the design of the programs was the minimization of the need for sophisticated interpretation of the test results; but minimization is not elimination. In the best case, the program will print out a conspicuous message indicating that the test passed or failed, and you need only interpret this message correctly. In other cases, you have to examine rather carefully the results and behavior of the program, and must apply the rules of the standard yourself. This interpretation is necessary in:

1.

2.

3.

Programs which test that PRINTed output is produced in a certain format

Programs which test that termination occurs at the correct time (this arises in many of the exception tests)

Programs for which conformance depends on the existence of adequate documentation of implementation-defined features (both those defined in Appendix C of the standard and for any of the error tests that are accepted).

[ocr errors]

automated solution conformance. Naive well lead to an implementation meets the

The test programs are an an only partially to the problem of determining processor reliance on the test programs alone can very incorrect judgment about whether an standard.

4.4.2 Types Of Tests

There are four types of

test programs:

1) standard, 2) exception, 3) error, and 4) informative. Within each of the functional groups (described above, section 4.2) the tests Occur in that order, although not all groups have all four types. The rules that pertain to each type follow. It is quite important that you be aware of which type of test you are running and use the rules which apply to that type.

[blocks in formation]

These tests are the ones whose title does not begin with "EXCEPTION" or "ERROR" and which generate a message about passing or failing at the end of each section. section. The paragraph below on documentation describes the concept of sections of a test. Since these programs are syntactically standard and raise no exception conditions, they must be accepted and executed to completion by the implementation. If the implementation fails to do this, it has failed the test. For example, if the implementation fails to recognize the key word OPTION, or if it does not accept one of numeric constants, then the test has failed.

the

obviously, it is you who must apply this rule, since the itself won't execute at all.

Quite program

to

Assuming that the implementation does process the program, the next question is whether it has done so correctly. The program may be able to determine this itself, or you may have do some active interpretation of the results. See the section below on documentation for more detail.

4.4.2.2 Exception Tests

These tests have titles that begin with the word "EXCEPTION" and examine the behavior of the implementation when exception conditions occur during execution. Nonetheless, these programs are also standard conforming (i.e., syntactically valid) and thus the implementation must accept and process them.

There are two special considerations. The first is the distinction between so-called fatal and non-fatal exceptions. Some exceptions in the standard specify a recovery procedure which allows continued execution of the program, while others

(the fatal exceptions) do not.

If no recovery procedure is specified, the implementation must report the exception and then terminate the program. Programs testing fatal exceptions will print out a message that they are about to attempt the instruction causing the exception. If execution proceeds beyond that point, the test fails and prints a message so stating. With the non-fatal exceptions, the test program attempts to discover whether the recovery procedure has been applied or not and in this instance, the test is much like the standard tests, where the question is whether the implementation has followed the semantic rules correctly. For instance, the semantic meaning of division by zero is to report the exception, supply machine infinity, and The standard, however, implementations to terminate execution after even a non-fatal exception "if restrictions imposed by the hardware or operating environment make it impossible to follow the given procedures." Because it would be redundant to keep noting this allowance, the test programs do not print such a message for each non-fatal exception. Therefore, when running a test for a non-fatal exception, note that the implementation may, under the stated circumstances, terminate the program, rather than apply the recovery procedure.

continue.

allows

The second special consideration is that in the case of INPUT and numeric and string overflow, the precise conditions for the exception can be implementation-defined. It is possible, therefore, that a standard program, executing on two different standard-conforming processors, using the same data, could cause an exception in one implementation and not in the other. The tests attempt to force the exception to occur, but it could happen, especially in the case of string overflow, that a syntactically standard program cannot force such an exception in a given processor. The documentation accompanying the implementation under test must describe correctly those features upon which the occurrence of exceptions depends. That is, it must be possible to find out from the documentation whether and when overflow and INPUT exceptions will occur in the test programs.

implementation-defined

exception

There is a summary of the requirements for handling in the form of pseudo-code in section 3.2.2 (Figure 1).

4.4.2.3 Error Tests

These tests have titles that begin with the word "ERROR" and examine how a processor handles a non-standard program. Each of these programs contains a syntactic construction explicitly ruled out by the standard, either in the various syntax sections, or in the semantics sections. Given a program with a syntactically construction the processor must either reject the a message to the user noting the reason for rejection, or, if it accepts the program, it must be accompanied by documentation which describes the interpretation of the

non-standard

program with

construction. Testing this requirement involves the submission of deliberately illegal programs to the processor to see if it will produce an appropriate message, or if it contains an enhancement of the language such as to assign a semantic meaning to the error. Thus we are faced with an interesting selection problem: out of the infinity of non-standard programs, which are worth submitting to the processor? Three criteria seem reasonable to apply:

1.

2.

Test errors which we might expect would be most difficult for a processor to detect, e.g., violations of context-sensitive constraints. These are the ones ruled out by the semantics rather than syntax sections of the standard.

Test errors likely to be made by beginners, for example use of a two character array name.

3.

Test errors for which there may very well
enhancement, e.g., comparing strings with

exist a language "<" and ">".

Based on these criteria, the test system contains programs for the errors in the two lists which follow. The first list is for constructions ruled out by the semantics sections alone (the se usually are instances of context-sensitive syntax constraints) and the second for plausible syntax errors ruled out by the BNF productions.

[blocks in formation]

8.

9.

number of arguments in function invocation <> number of parameters in function definition

reference to numeric-supplied-function with incorrect number of arguments

10. no spaces around keywords

11.

spaces within keywords and other elements or before line number

« PreviousContinue »