Your browser does not appear to support Javascript, please update your browser or contact your system administrator to enable Javascript on your Internet browser. Thank you. Chapter 4: Documentation and Design Reviews — U.S. Election Assistance Commission
Skip to content

U.S. Election Assistance Commission

Personal tools
You are here: Home TGDC Recommended Guidelines Part 3: Testing Requirements Chapter 4: Documentation and Design Reviews
TGDC Recommended
Guidelines

VVSG Navigation
 

Chapter 4: Documentation and Design Reviews (Inspections)

An inspection or review is logically reported as one or more tests with a verdict of Pass or Fail. The number of tests reported corresponds to how the test lab chooses to structure the inspection.

To the extent possible, these VVSG provide guidance on the criteria to be applied. However, the nature of some of these inspections is to rely on the professional judgment of an expert reviewer to assess conformity with general guidelines.

4.1 Initial Review of Documentation

The accredited test lab reviews the documentation submitted by the manufacturer for its completeness and satisfaction of requirements.

4.1-A Initial review of documentation

At the beginning of inspection, the test lab SHALL verify that the documentation submitted by the manufacturer in the TDP meets all requirements applicable to the TDP, is sufficient to enable the inspections specified in this chapter, and is sufficient to enable the tests specified in Part 3: Chapter 5: “Test Methods”.

Applies to: Voting system

DISCUSSION

This includes verifying that source code has been supplied compliant with Requirement Part 2: 3.4.7.2-E.

Source: [VSS2002]/[VVSG2005] II.5.3, generalized

4.1-B Review of COTS suppliers' specifications

For COTS components, such as printers and touchscreens, that were integrated into a voting device by the manufacturer, the test lab SHALL review the COTS manufacturers' specifications to verify that those manufacturers approve of their products' use under the conditions specified by these VVSG for voting systems.

Applies to: Voting system

DISCUSSION

For example, if the operating and/or storage environmental conditions specified by the manufacturer of a printer do not meet or exceed the requirements of these VVSG, a system that includes that printer cannot be found conforming.

Source: New requirement

4.2 Physical Configuration Audit

The Physical Configuration Audit (PCA) is the formal examination of the as-built version of a voting system against its design documentation in order to establish the product baseline. After successful completion of the audit, subsequent changes are subject to test lab review and reexamination.

4.2-A As-built configuration reflected by records

The test lab SHALL audit the system's documentation and quality assurance records to verify that the as-built configuration is reflected by the documentation and records.

Applies to: Voting system

DISCUSSION

This includes both hardware and logic (e.g., software, firmware, etc.).

Source: [MIL85] 80.1, [VVSG2005] II.6.6

4.2-B Check identity of previously tested devices

If a limited scope of testing is planned for a system containing previously tested devices or subsystems, the test lab SHALL verify that the affected devices or subsystems are identical to those previously tested.

Applies to: Voting system

Source: [VSS2002] II.6.3.a / [VVSG2005] II.6.3

4.2-C Accuracy of system and device classification

The test lab SHALL verify that the classes claimed in the implementation statement accurately characterize the system and devices submitted for testing.

Applies to: Voting system

DISCUSSION

Any electronic device that includes software or firmware installed or commissioned by the voting system manufacturer is a programmed device. Manufacturers claiming that an electronic device is not programmed must demonstrate to the satisfaction of the test lab and any authorities approving the test plan that the device contains no software or firmware that should be subject to the requirements indicated for programmed devices.

Source: New requirement

4.2-D Validate configuration

The test lab SHALL confirm the propriety and correctness of the configuration choices described in Part 2: 3.8 “Configuration for Testing”.

Applies to: Voting system

Source: [VSS2002] I.4.1.1

4.3 Verification of Design Requirements

Many design requirements state simply that the system SHALL have some physical feature without any additional constraints. Such requirements are easily verified by inspection. Other requirements that state that the system SHALL prevent something from occurring are not verifiable through operational testing, so inspection (with expert judgment) is the only effective testing strategy.

4.3-A Verify design requirements

For each requirement of Part 1 that is not amenable to operational testing, the test lab SHALL review the application logic, border logic, third-party logic, configuration data, and/or design of the voting system as needed to verify that the requirement is satisfied.

4.3-B Identification of security control inconsistencies

The test lab SHALL determine if all security controls properly implemented have no obvious inconsistencies with the voting system’s functional requirements, the overall objectives of the voting device’s security strategy, and no obvious internal errors.

Applies to: Voting system

Source: [NIST05]

4.4 Manufacturer Practices for Quality Assurance and Configuration Management

4.4.1 Examination of quality assurance and configuration management data package

4.4.1-A Quality and Configuration Management Manual

The Quality and Configuration Management Manual SHALL be reviewed for its fulfillment of Requirement Part 1: 6.4.2.1-A, and the requirements specified in Part 2: 2.1 “Quality and Configuration Management Manual”.

Source: New requirement

4.4.2 Examination of voting systems submitted for testing

These requirements deal with the quality assurance and configuration examination of voting systems submitted for testing to a test lab.

4.4.2.1 Configuration management

4.4.2.1-A Identification of systems

The test lab SHALL verify that the voting system has an identification tag attached to the main body as described in Requirements Part 1: 6.4.2.2-A.1 and Part 1: 6.4.2.2-A.2

Applies to: Voting system

Source: New requirement

4.4.2.1-B Configuration log

The test lab SHALL verify that the voting system has associated with it a Configuration Log, as described in Requirements Part 1: 6.4.2.2-B.1 and Part 1: 6.4.2.2-B.2

Applies to: Voting system

Source: New requirement

4.5 Source Code Review

In the source code review, the accredited test lab will look at programming completeness, consistency, correctness, modifiability, structure, modularity and construction.

4.5.1 Workmanship

Although these requirements are scoped to application logic, in some cases the test lab may need to inspect border logic and third-party logic to assess conformity. Per Requirement Part 2: 3.4.7.2-E, the source code for all of these must be provided.

4.5.1-A Review source versus manufacturer specifications

The test lab SHALL assess the extent to which the application logic adheres to the specifications made in its design documentation.

Applies to: Voting system

DISCUSSION

Since the nature of the requirements specified by the manufacturer is unknown, conformity may be subject to interpretation. Nevertheless, egregious disagreements between the application logic and its design documentation should lead to a defensible adverse finding.

Source: [VSS2002] II.5.4

4.5.1-B Review source versus coding conventions

The test lab SHALL assess the extent to which the application logic adheres to the published, credible coding conventions chosen by the manufacturer.

Applies to: Voting system

DISCUSSION

See Requirement Part 1: 6.4.1.3-A.

Since the nature of the requirements specified by the coding conventions is unknown, conformity may be subject to interpretation. Nevertheless, egregious disagreements between the application logic and the coding conventions should lead to a defensible adverse finding.

Source: [VSS2002] II.5.4, II.5.4.2

4.5.1-C Review source versus workmanship requirements

The test lab SHALL assess the extent to which the application logic adheres to the requirements of Part 1: 6.4.1 “Software engineering practices”.

Applies to: Voting system

DISCUSSION

With respect to Requirement Part 1: 6.4.1.4-B, see Requirement Part 2: 3.4.7.2-I. The reviewer should consider the functional organization of each module or callable unit and the use of formatting, such as blocking into readable units, that supports the intent of Requirement Part 1: 6.4.1.4-B.

Source: [VSS2002] II.5.4

4.5.1-D Efficacy of built-in self-tests

The test lab SHALL verify the efficacy of built-in measurement, self-test, and diagnostic capabilities described in Part 1: 7.3.1 “Logic and accuracy testing”.

Applies to: Voting system

Source: [VSS2002] I.2.3.4.1.a2 (the second a)

4.5.2 Security

4.5.2-A Security control source code review

The test lab SHALL analyze the source code of the security controls to assess whether they function correctly and cannot be bypassed.

Applies to: Voting system

4.6 Logic Verification

This inspection is to assess conformity with Requirement Part 1: 6.3.2-A and related requirements.

Because of its high complexity, the scope of logic verification is pragmatically limited to core logic. Software modules that are solely devoted to interacting with election officials or voters or formatting reports are not subject to logic verification. However, they are required to conform with Requirement Part 1: 6.1-A, the testing of which is described in Part 3: 4.3 “Verification of Design Requirements” and Part 3: 4.5.2 “Security”.

Although these requirements are scoped to core logic, in some cases the test lab may need to inspect other application logic, border logic and third-party logic to assess conformity. Per Requirement Part 2: 3.4.7.2-E, the source code for all of these must be provided.

[Redmill88] provides the following description of logic verification, therein known as "program proving:"

Assertions are made at various locations in the program, which are used as pre-, and post-conditions to various paths through the program. The proof consists of two parts. The first involves showing that the program transfers the pre-conditions into the post-conditions according to a set of logical rules defining the semantics of the programming language, provided that the program actually terminates (i.e., reaches its proper conclusion). The second part is to demonstrate that the program does indeed terminate (e.g., does not go into an infinite loop). Both parts may need inductive arguments.

The inspection specified here does not assume that the programming language has formally specified semantics. Consequently, a formal proof at any level cannot be mandated. Instead, a combination of informal arguments (see Requirement Part 2: 3.4.7.2-F.b) and limitations on complexity (see Requirement Part 1: 6.4.1.4-B.1) seeks to make the correctness of callable units at the lowest level intuitively obvious and to enable the verification of higher level units using the correctness of invoked units as theorems. The resulting inspection is not as rigorous as a formal proof, but still provides greater assurance than is provided by operational testing alone.

Inasmuch as the following behaviors would almost certainly preclude a demonstration of the correctness of the logic, logic verification will almost certainly involve a demonstration that they cannot occur:

  • Numeric errors such as overflow and divide-by-zero;
  • Buffer overruns / out-of-bounds accesses of arrays or strings;
  • Null pointer dereferences;
  • Stack overflows;
  • Invocations of undefined or implementation-dependent behaviors;
  • Race conditions or other nondeterministic execution;
  • Abrupt termination.

It is acceptable, even expected, that logic verification will show that some or most exception handlers in the source code cannot logically be invoked. These exception handlers are not redundant—they provide defense-in-depth against faults that escape detection during logic verification and unpredictable failures that compromise the system.

4.6-A Check inductive assertions

For each callable unit (function, method, operation, subroutine, procedure, etc.) in core logic, the test lab SHALL check that the preconditions and postconditions correctly describe the behavior of the unit in all cases.

Applies to: Voting system

DISCUSSION

See Requirement Part 2: 3.4.7.2-F. For a callable unit at the lowest level, this should be achievable through code reading. For a higher level unit, the correctness of the pre- and postconditions of the units that it invokes is assumed as a premise in the argument that the pre- and postconditions of the higher level unit are correct.

4.6-B Check limits

The test lab SHALL check that the assumptions about capacities and limits that appear in the preconditions, postconditions, and proofs are consistent with the capacities and limits that the devices are claimed in the implementation statement to be capable of processing correctly.

Applies to: Voting system

DISCUSSION

See Requirement Part 2: 3.4.7.2-F.a and Requirement Part 1: 2.4-A.e.

4.6-C Check constraints

For the core logic as a whole, and for each constraint indicated in Part 1: 8.3 “Logic Model (normative)”, the test lab SHALL check that the constraint is satisfied in all cases within the aforementioned capacities and limits.

Applies to: Voting system

DISCUSSION

See Requirement Part 2: 3.4.7.2-G.

4.6-D Burden of proof

If the test lab finds that the preconditions, postconditions, and proofs provided by the manufacturer are insufficient or incorrect, the responsibility for completing or correcting them SHALL be the manufacturer's.

Applies to: Voting system

DISCUSSION

Although test labs will doubtless provide advice and assistance to their clients, they are not required to fill in gaps in the manufacturer's submission.