Page images
PDF
EPUB

The tests assume that the implementation-defined margin for output lines is at least 72 characters long and contains at least 5 print zones. This should not be confused with the length of a line in the source code itself. The standard requires implementations to accept source lines up to 72 characters long. If the margin is smaller than 72, the tests should still run (according to the standard), but the output will be aesthetically less pleasing.

Finally, the standard does not specify how the tests are to be submitted to the processor for execution. Therefore, the machine-readable part of the test system consists only of code, i.e., there are no system control commands. It is your responsibility to submit the programs to the implementation in a natural way which does not violate the integrity of the tests.

4.4 Operating And Interpreting The Tests

1

programs as a tool to measure
The more general issues of
section 3, and of course in the
sections
and 2 of the ANSI

This section will attempt to guide you through the practical aspects of using the test implementation conformance. conformance are covered in standard itself, especially document.

4.4.1 User Checking Vs. Self Checking

All of the test programs require interpretation of their behavior by the user. As mentioned earlier, the user is an active component in the test system; the source code of the test programs is another component, subordinate to the test user. An important goal in the design of the programs was the minimization of the need for sophisticated interpretation of the test results; but minimization is not elimination. In the best case, the program will print out a conspicuous message indicating that the test passed or failed, and you need only interpret this message correctly. In other cases, you have to examine rather carefully the results and behavior of the program, and must apply the rules of the standard yourself. This interpretation is necessary in:

1.

2.

3.

Programs which test that PRINTed output is produced in certain format

Programs which test that termination occurs at the correct time (this arises in many of the exception tests)

Programs for which conformance depends on the existence of adequate documentation of implementation-defined features (both those defined in Appendix C of the standard and for any of the error tests that are accepted).

automated solution conformance. Naive well lead to an implementation meets the

The test programs are an only partially to the problem of determining processor reliance on the test programs alone can very incorrect judgment about whether an standard.

4.4.2 Types Of Tests

test programs:

There are four types of 1) standard, 2) exception, 3) error, and 4) informative. Within each of the functional groups (described above, section 4.2) the tests occur that order, although not all groups have all four types. The rules that pertain to each type follow. It is quite important that you be aware of which type of test you are running and use the rules which apply to that type.

[blocks in formation]

These tests are the ones whose title does not begin with "EXCEPTION" or "ERROR" and which generate a message about passing or failing at the end of each section. The paragraph below on documentation describes the concept of sections of a test. Since these programs are syntactically standard and raise no exception conditions, they must be accepted and executed to completion by the implementation. If the implementation fails to do this, it has failed the test. For example, if the implementation fails to recognize the key word OPTION, or if it does not accept one of numeric constants, then the test has failed. Quite program

the

obviously, it is you who must apply this rule, since the itself won't execute at all.

Assuming that the implementation does process the program, the next question is whether it has done so correctly. The program may be able to determine this itself, or you may have to do some active interpretation of the results. See the section below on documentation for more detail.

4.4.2.2 Exception Tests

These tests have titles that begin with the word "EXCEPTION" and examine the behavior of the implementation when exception conditions occur during execution. Nonetheless, these programs are also standard conforming (i.e., syntactically valid) the implementation must accept and process them.

considerations.

There are two special The first is the distinction between so-called fatal and non-fatal exceptions. Some exceptions in the standard specify a recovery procedure which allows continued execution of the program, while others

(the fatal exceptions) do not.

If no recovery procedure is

specified, the implementation must report the exception and then terminate the program. Programs Programs testing fatal exceptions will print out a message that they are about to attempt the instruction causing the exception. If execution proceeds beyond that point, the test fails and prints a message so stating. With the non-fatal exceptions, the test program attempts to discover whether the recovery procedure has been applied or not and in this instance, the test is much like the standard tests, where

semantic rules correctly.

the question is whether the implementation has followed the For instance, the semantic meaning of division by zero is to report the exception, supply machine infinity, and continue. The standard, however,

allows

implementations to terminate execution after even a non-fatal exception "if restrictions imposed by the hardware or operating environment make it impossible to follow the given procedures." Because it would be redundant to keep noting this allowance,

test programs do not print such a message for each non-fatal exception. Therefore, when running a test for a non-fatal exception, note that the implementation may, under the stated circumstances, terminate the program, rather than apply the recovery procedure.

The second special consideration is that in the case of INPUT and numeric and string overflow, the precise conditions for the exception can be implementation-defined. It is possible, therefore, that a standard program, executing on two different standard-conforming processors, using the same data, could cause an exception in one implementation and not in the other. The tests attempt to force the exception to occur, but it could happen, especially in the case of string overflow, that a syntactically standard program cannot force such an exception in a given processor. The documentation accompanying the implementation under test must describe correctly those features upon which the Occurrence of exceptions depends. That is, it must be possible to find out from the documentation whether and when overflow and INPUT exceptions will occur in the test programs.

implementation-defined

There is a summary of the requirements for exception handling in the form of pseudo-code in section 3.2.2 (Figure 1).

4.4.2.3 Error Tests

These tests have titles that begin with the word "ERROR" and examine how a processor handles a non-standard program. Each of these programs contains a syntactic construction explicitly ruled out by the standard, either in the various syntax sections, or in the semantics sections. Given a program with a syntactically non-standard processor must either reject the program with message a user noting the rejection or if it accepts the program, it must be accompanied by documentation which describes the interpretation of

construction the

to the

reason

for

the

construction. Testing this requirement involves the submission of deliberately illegal programs to the processor to

see if it

will produce an appropriate message, or if it contains an enhancement of the language such as to assign a semantic meaning to the error. Thus we are faced with an interesting selection problem: out of the infinity of non-standard programs, which are worth submitting to the processor? Three criteria seem reasonable to apply:

1.

2.

Test errors which we might expect would be most difficult for a processor to detect, e.g., violations of context-sensitive constraints. These are the ones ruled out by the semantics rather than syntax sections of the standard.

Test errors likely to be made by beginners, for example
of a two character array name.

exist

use

a language

3. Test errors for which there may very well
enhancement, e.g., comparing strings with "<" and ">".

contains programs

Based on these criteria, the test system for the errors in the two lists which follow. The first list is for constructions ruled out by the semantics sections alone (these usually are instances context-sensitive syntax constraints) and the second for plausible syntax errors ruled out by the BNF productions.

[blocks in formation]

9.

10.

number of arguments

in function invocation <> number of

parameters in function definition

reference to numeric-supplied-function with incorrect number of arguments

no spaces around keywords

11. spaces within keywords and other elements or before line number

[blocks in formation]

line number for GOTO, GOSUB, IF...THEN,

[blocks in formation]

17.

conflict on number of dimensions between DIM and reference, e.g., DIM A(20) and either A or A(2,2)

[blocks in formation]

22. OPTION BASE 1 followed by DIM A(0)

[blocks in formation]

4.

comparison of two strings for < or >

5. comparison of a string with a number unmatched parenthesis in expression

6.

[blocks in formation]
« PreviousContinue »