Your browser does not appear to support Javascript, please update your browser or contact your system administrator to enable Javascript on your Internet browser. Thank you. Chapter 6: Test Report — U.S. Election Assistance Commission
Skip to content

U.S. Election Assistance Commission

Personal tools
You are here: Home TGDC Recommended Guidelines Part 2: Documentation Requirements Chapter 6: Test Report
TGDC Recommended
Guidelines

VVSG Navigation
 

Chapter 6: Test Report (test lab)

6.1 Test report contents

Reporting performance test results for usability is covered under Part 1: 3.2.1.1 “Overall performance metrics”.

6.1-A Test report, include revision history

For modifications to previously tested systems, the test lab SHALL include the test reports that are precedential to the current evaluation.

Applies To: Voting system

DISCUSSION

It is anticipated that the test report will be delivered in electronic form, so the volume of data should not be a problem.

Source: New requirement

6.1-B Test report, include test plan as amended

The test lab SHALL include a copy of the test plan, amended to reflect any changes that were allowed during the course of the testing campaign.

Applies To: Voting system

DISCUSSION

Test plans must be updated whenever a change to a voting system requires deviation from the original test plan.

6.1-C Test report, implementation statement as amended

The test lab SHALL include the implementation statement submitted by the manufacturer, amended to reflect any changes that were allowed during the course of the testing campaign.

Applies To: Voting system

DISCUSSION

Because minor defects in a system may be corrected during the course of the testing campaign, the system that completes the conformity assessment process might not be identical to the one for which an implementation statement was submitted. The product identification for the revised system must be different. Also, if a system fails a test for a particular voting variation, the manufacturer and test lab may agree to eliminate that voting variation from the list of classes to which conformity assessment is desired rather than correct the system.

6.1-D Test report, witness build

The test lab SHALL include a copy of the record of the final (witnessed) build and sufficient description of the build process to reproduce it.

Applies To: Voting system

DISCUSSION

See Part 3: 2.6.1 “Voting system software version recommended for certification”.

Source: New requirement

6.1-E Test report, setup validation info

The test lab SHALL identify the repository for software reference information and include the unique identifier assigned to the software reference information by the repository.

Applies To: Voting system

Source: New requirement

6.1-F Test report, summary finding

The test lab SHALL include a summary finding of whether or not the implementation under test satisfies all applicable, mandatory ("SHALL") requirements of the Voluntary Voting System Guidelines.

Applies To: Voting system

6.1-G Test report, reasons for adverse opinion

If the test lab finds that the implementation under test does not satisfy all applicable, mandatory ("SHALL") requirements of the Voluntary Voting System Guidelines, the test lab SHALL identify each of the specific requirements that is not satisfied.

Applies To: Voting system

6.1-H Test report, evidence supporting adverse opinion

For each unsatisfied mandatory requirement, the test lab SHALL describe the inspections or tests that detected the nonconformities and include applicable evidence (e.g., vote data report, citation of logic error in source code).

Applies To: Voting system

6.1-I Test report, anomalies

The test lab SHALL summarize all failures, errors, nonconformities and anomalies that were observed during conformity assessment, no matter how minor.

Applies To: Voting system

6.1-I.1 Test report, deficiencies corrected during test campaign

The test lab SHALL identify those deficiencies that were corrected during the course of the testing campaign and identify the inspections or tests that confirm that the deficiencies were corrected.

Applies To: Voting system

DISCUSSION

For minor defects of a localized nature, the test lab may permit the manufacturer to correct the fault without incurring a complete regression test of the system. However, a certifying authority may require that revised documents be submitted whenever changes are made.

6.1-J Test report, benchmarks

For requirements that specify benchmarks, the test lab SHALL report the result of the measurement for the implementation under test.

Applies To: Voting system

6.1-J.1 Test report, failure rate

The test lab SHALL report the observed cumulative failure rate and the failure rate that was demonstrated with 90 % confidence for each type of device, for each applicable failure type in Part 1: Table 6-3 (Part 1: 6.3.1.5 “Requirements”).

Applies To: Voting device

DISCUSSION

See also Part 3: 5.3.2 “Critical values”. "Type of device" refers to the different models produced by the manufacturer. These are not the same as device classes. The system may include several different models of the same class, and a given model may belong to more than one class.

6.1-J.2 Test report, error rate

The test lab SHALL report the observed cumulative report total error rate and the report total error rate that was demonstrated with 90 % confidence for the system as a whole.

Applies To: Voting system

DISCUSSION

See Part 3: 5.3.4 “Accuracy”.

6.1-J.3 Test report, misfeed rate

For paper-based tabulators and EBMs, this SHALL include the observed cumulative misfeed rate and the misfeed rate that was demonstrated with 90 % confidence for each type of device.

Applies To: Paper-based device Λ Tabulator, EBM

DISCUSSION

See Part 3: 5.3.5 “Misfeed rate”.

6.1-K Test report, ballot tabulation rate

For paper-based tabulators, the test lab SHALL report the ballot tabulation rate used in tests.

Applies To: Paper-based device Λ Tabulator

DISCUSSION

Stress tests might use a higher rate than other tests.

6.1-L Test report, shoulds that were not done

The test lab SHALL identify each applicable, non-mandatory ("SHOULD") requirement to which nonconformity was demonstrated.

Applies To: Voting system

DISCUSSION

Test labs are not required to test every "should" requirement; however, if they do, they must report the results.

6.1-M Test report, waived tests

The test lab SHALL identify all tests that were waived.

Applies To: Voting system

DISCUSSION

A test is waived if the documented assumptions of an applicable test are not met by the implementation under test. A test that pertains to a system or device class that was not claimed in the implementation statement is implicitly assigned the verdict Not Applicable.

6.1-N Test report, timeline

The test lab SHALL include a timeline of the testing campaign as it actually occurred.

Applies To: Voting system

6.1-O Test report, compensatory procedures

The test lab SHALL list any specific election management practices that are required for the voting system to satisfy the requirements of the VVSG.

Applies To: Voting system

DISCUSSION

For example, if additional procedures must be followed in order to safeguard the secrecy of the vote, these must be documented. If a system requires unusually onerous procedural compensations because customary system safeguards are absent, this may impact certification decisions.

Source: New requirement

6.1-P Test report, warrant of accepting change control responsibility

If any changes to the system are required to complete conformity assessment, the test lab SHALL include a signed warrant from the manufacturer that those changes will be included in the product that is delivered to customers.

Applies To: Voting system

Source: New requirement

6.1-Q Test report, issues list

The test lab SHALL list and explain any concerns that SHOULD be brought to the attention of readers and/or the VVSG interpretations and maintenance processes.

Applies To: Voting system

DISCUSSION

Any unresolved concerns may be documented in the test report. "Concerns" would include ambiguities in the VVSG, interpretation conflicts, requirements that appear to do more harm than good, loopholes in the VVSG (where it is possible to satisfy the technical requirements while failing to satisfy their intent), and other issues whose resolution would require action by outside authorities.