Module 47 - T204 Part 1 of 2

T204 Part 1 of 2: How to Develop Test Procedures for an ITS Standards-Based Test Plan

HTML of the PowerPoint Presentation

(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

Slide 1:

(Extended Text Description: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words "Standards ITS Training" in green and blue on the middle left side. The word "Welcome" in white is to the right of the logo. Under the logo box is the logo for the U.S. Department of Transpotation, Office of the Assistant Secretary for Research and Technology.)

Slide 2:

(Extended Text Description: This slide, entitled "Welcome" has a photo of Ken Leonard, Director, ITS Joint Program Office, on the left hand side, with his email address, Ken.Leonard@dot.gov. A screen capture snapshot of the home webpage is found on the right hand side - for illustration only - from August 2014. Below this image is a link to the current website: www.its.dot.gov/pcb - this screen capture snapshot shows an example from the Office of the Assistant Secretary for Research and Development - Intelligent Transportation Systems Joint Program Office - ITS Professional Capacity Building Program/Advanced ITS Education. Below the main site banner, it shows the main navigation menu with the following items: About, ITS Training, Knowledge Exchange, Technology Transfer, ITS in Academics, and Media Library. Below the main navigation menu, the page shows various content of the website, including a graphic image of professionals seated in a room during a training program. A text overlay has the text Welcome to ITS Professional Capacity Building. Additional content on the page includes a box entitled What's New and a section labeled Free Training. Again, this image serves for illustration only. The current website link is: https://www.its.dot.gov/pcb.)

Slide 3:

Slide 4:

T204 Part 1 of 2:

How to Develop Test Procedures for an ITS Standards-Based Test Plan

Slide 5:

Instructor

Dave Miller,

Chair: NEMA / AASHTO / ITE

Joint Committee on ATC

Chair: 3TS Technical Committee

Principal Systems Engineer

Siemens Industry, Inc.

RC-US MO MM-ITS R&D

Austin, Texas, USA

Slide 6:

Target Audience

• Traffic management and engineering staff
• Maintenance Staff
• System developers
• Test personnel
• Private and public sector users including manufacturers

Slide 7:

Recommended Prerequisite(s)

• T101: Introduction to ITS Standards Testing
• T201: How to Write a Test Plan
• T202: Overview of Test Design Specifications, Test Case Specifications and Test Procedures
• T203 Part 1 of 2: How to Develop Test Cases for an ITS Standard-based Test Plan, Part 1 of 2
• T203 Part 2 of 2: How to Develop Test Cases for an ITS Standard-based Test Plan, Part 2 of 2

Slide 8:

Curriculum Path

(Extended Text Description: Curriculum Path: A graphical illustration indicating the sequence of training modules that lead up to and follow each course. Each modules is represented by a box with the name of the module in it and an arrow showing the logical flow of the modules and the current module highlighted. This slide focuses on the modules that lead up to the current course. The first box is labeled "T101 Introduction to ITS Standards Testing" followed by a line that connects to a box labeled "T201 How to Write a Test Plan" followed by a line that connects to a box labeled "T202 Overview of Test Design Specifications, Test Cases and Test Procedures," followed by a line that connects to a box labeled "T203 Part 1 of 2 How to Develop Test Cases for an ITS Standards-based Test Plan, Part 1 of 2," followed by a line that connects to a box labeled "T203 Part 2 of 2 How to Develop Test Cases for an ITS Standards-based Test Plan, Part 2 of 2." This box, in turn, connects to a box, colored purple to indicate that it represents this module, labeled "T204 Part 1 of 2 How to Develop Test Procedures for an ITS Standard-based Test Plan, Part 1 of 2.")

Slide 9:

abbrs and Terminology Used

• TP - Test Plan
• TDS - Test Design Specification
• TPS - Test Procedure Specification
• TPG - Test Plan Generator
• TCS - Test Case Specification
• DMS - Dynamic Message Sign
• TMDD -Traffic Management Data Dictionary
• RTM - Requirements Traceability Matrix
• RTCM - Requirements to Test Case Matrix

(Extended Text Description: abbrs and Terminology Used: A graphic illustration created for this slide that depicts the testing tasks within the overall system workflow. The topmost gray box labeled "Test Plan" includes an arrow from the lower side of that box directed downward to a second gray box labeled "Test Design Specification" that includes three arrows as follows: Arrow 1 from the left side of "Test Design Specification" is directed to the left then down to a gray box labeled "Test Case 1". Arrow 2 from the lower side of "Test Design Specification" is directed downward to a gray box labeled "Test Case 2". Arrow 3 from the right side of "Test Design Specification" is directed to the right then downward to a gray box labeled "Test Case 3". The gray box labeled "Test Case 1" includes an arrow from the lower side of that box directed downward to a gray box labeled "Test Procedure 1". The gray box labeled "Test Case 2" includes an arrow from the left side of that box directed to the left then downward to the gray box labeled "Test Procedure 1". The gray box labeled "Test Case 3" includes two arrows, one arrow from that box directed to the left and then downward, and then to the left to the gray box labeled "Test Procedure 2". The second arrow is directed downward from the lower side of "Test Case 3" to a gray box labeled "Test Procedure 3".)

See Student Supplement for Details

Slide 10:

Learning Objectives

Part 1 of 2

1. Recognize the purpose and structure of a test procedure
2. Identify the role of a Test Procedure Specification (TPS) within a test plan and the overall testing process
3. Synchronize the test procedure specification to the contract terms and conditions for successful contract execution
4. Write the reports produced at the end of testing and understand their relationship to successful procurement contracts
5. Use tools to develop the test procedures for a sample TPS structure

Part 2 of 2

6. Use the Test Plan Generator (TPG) to generate test procedures for a variety of equipment
7. Adapt the generated test procedures to procurement contract terms and conditions for successful project conclusion
8. Develop complex test procedures that pull together NTCIP elements using the TPG

Slide 11:

Learning Objective #1: Recognize the Purpose and Structure of a Test Procedure Specification

• Explain structure of a Test Procedure Specification (TPS)
• Discuss inputs and expected outputs from a test procedure
• Explain special needs from a test procedure stand point

Slide 12:

Definitions

• A test procedure provides detailed instructions for the setup, execution, and evaluation of results for a given test case (IEEE Std 829)
• Test Procedure Specification (TPS) is a document that defines the steps to execute a test and relates to test cases
• TPS is used in ITS project as part of a test plan prepared for a system acceptance

(Extended Text Description: Definitions: Three graphic illustrations created for this module, positioned vertically above one another located at the right side of the slide. The topmost graphic illustration depicts a white box containing the text "Test Procedure". A light blue arrow containing the text "Inputs are Test Cases" is directed towards and touches the left side of "Test Procedure" box. A second light blue arrow containing the text "Outputs are Test Results" touches the right side of "Test Procedure" box, directed towards the right. The middle graphic illustration depicts a white box with light blue border containing the green text "Test Procedure Specification TPS". The lower graphic illustration contains a depiction of a gray "Device Under Test (DUT)" box and two gray laptop computers labeled "Test Software" and "Data Analyzer". A white depiction of a cloud labeled "Communications" is positioned below "Device Under Test (DUT)", above "Test Software" and to the left of "Data Analyzer". )

Slide 13:

Learning Objective #1

Purpose and Structure of a Test Procedure

Purpose of Test Procedure

• A standardized test procedure provides meaningful results (outputs) to a user for evaluating the system:
  Did we build the system as intended by the project specification (user)?
• In the previous Module, T203, we learned how to develop test cases for DMS and TMDD-based ITS projects
• Continuing further, in this module we will learn how to develop TPS using IEEE Std 829 format and definitions

Slide 14:

Learning Objective #1

What does IEEE Std 829 Provide?

• Guidance and formats for preparing testing documentation:
  • Test Plan
  • Test Design Specification
  • Test Case Specification
  • Test Procedure Specification (TPS)
  • Test Reports
    • Test Logs
    • Test Anomaly Report
    • Test Report
• Testing professionals across ITS are familiar with these definitions and the sequence of the testing workflow

Slide 15:

Learning Objective #1

Purpose and Structure of a Test Procedure (cont.)

Structure of a Test Procedure (IEEE Standard 829)

1. Introduction

• Document identifier
• Scope
• References
• Relationship to other procedures

2. Details

• Inputs, outputs, and special requirements
• Ordered description of the steps to be taken to execute the test cases

3. General

• Glossary
• Document change procedures and history

Slide 16:

Learning Objective #1

Purpose and Structure of a Test Procedure (cont.)

Related Details

• Test resources
• Test schedule
• Items to be tested
• Features to be tested
• Test tasks to be performed
• Test personnel required to conduct tests
• Test risks

Slide 17:

Learning Objective #1

Where does TPS Fit in Documentation Structure?

(Extended Text Description: Where does TPS Fit in Documentation Structure? Author's relevant notes: A light blue arrow directed from the right of a white text box with light blue border containing the green text "Test Procedure Specification (TPS)". To the right of that light blue arrow is located a graphic recreated from "PCB Module-IEEE 829-1998" that depicts the six steps to execute a test: 1) Topmost "Master Test Plan", below that, "Test Design Specification", below that, "Test Case Specification", below that, "Test Procedure Specification", below that, "Test Execution" and below that, "Test Reports". Within each of these steps is depicted one or more labeled boxes showing the workflow recreated from IEEE 829-1998 without alteration in this graphic.)

Source: PCB Module-IEEE 829-1998

Slide 18:

Learning Objective #1

Inputs and Expected Outputs

A Test Procedure is Conducted on Test Inputs to Produce Test Outputs

(Extended Text Description: Inputs and Expected Outputs: A light blue arrow "Inputs are Test Cases" located at the center-left of the slide, directed towards the left edge of white box "Test Procedures with Expected Outputs". A light blue arrow "Outputs are Test Results" directed from the left side of "Test Procedures with Expected Outputs" to the left edge of light brown "Test Results:" box, which contains the text, Test Logs, Test Summaries, Discrepancy Reports-documents unexpected outputs, and The deliverables are formatted per IEEE 829. Light blue "Output Categories" box is located below "Test Results" box, which contains the text, Expected outputs ("PASS"), Unexpected outputs (requires further investigation).)

Slide 19:

Learning Objective #1

Role of Test Procedures

• Test procedure ultimately traces back to the user needs—a starting point in a SE life cycle of an ITS project:
  • Test procedure traces back to test cases
  • Test cases trace back to requirements
  • Requirements trace back to user needs

(Extended Text Description: Role of Test Procedures: Light blue "ITS Project Functional Features-Capabilities" box, with "ITS Project Functional Features-Capabilities" highlighted in yellow, located along the right side of the slide. Four colored text boxes are located within "ITS Project Functional Features-Capabilities" box. Light blue "Test Case" box is located at the upper left within "ITS Project Functional Features-Capabilities". Red "Test Procedures" box is located at the upper right within "ITS Project Functional Features-Capabilities". Green "User Needs" box is located at the lower right within "ITS Project Functional Features-Capabilities". Dark blue "Requirements" box is located at the lower right within "ITS Project Functional Features-Capabilities". Four arrows are also located within "ITS Project Functional Features-Capabilities". A red arrow is located to the right of "Test Procedure" directed towards "Test Cases". A second red arrow is located below "Test Cases" directed towards "Requirements". A third red arrow is located to the right of "Requirements" directed towards "Use Needs". A green arrow is located below "Test Procedures" directed towards "User Needs".)

Slide 20:

Learning Objective #1

Role of Test Procedures (cont.)

Test Procedure Insures the System is Built Correctly as Expected by the User

• Verifying that we built the system correctly:
  • Design used standard objects and dialogs correctly
  • Design implemented special contract provisions (if any) correctly
  • The system performs as expected by the user

Did we build the system as intended by a project specification?

-users, owners, developers, vendors

(Extended Text Description: Role of Test Procedures (cont.): Reproduction of a portion of the VEE model graphic taken from Systems Engineering for Intelligent Transportation Systems, USDOT, January 2007. Graphic depicts three levels of the right side of the VEE which are 1) System Validation level, 2) System Verification & Deployment level and 3) Subsystem Verification level. A red arrow points to the word "Deployment", which is highlighted in red.)

Source: FHWA

Slide 21:

Learning Objective #1

Special Needs from a Test Procedure Standpoint

(Extended Text Description: Special Needs from a Test Procedure Standpoint: White box with light blue outline containing the text "Test procedure" centered on the left of the slide. A light blue, double-ended arrow projecting +45 degrees and located between the "Test Procedure" box and the text to the upper right. The box to the upper right contains the text, Special Needs Outside of the Standards - Special security measures, Special access rights (permission) to data bases, Documentation control for archiving deliverables, Configuration management to reproduce the same results, Records of any automated test tools, Special skill sets required of test operators. A second light blue, double-ended arrow projecting -45 degrees and located between the "Test Procedure" box and the text to the lower right, which contains the text, Special contract provisions (if any): Special needs do not show up for the first time in the TPS, Special Needs of contracts move through TDS and TCS.)

Slide 22:

Learning Objective #1

IEEE 829 Level Test Procedure Steps

LEVEL TEST PROCEDURE # ID
LOG:	Logging tools and methods
	List ...
SETUP:	Sequence of actions to prepare for testing
	List ...
START:	Actions to begin test execution
	List ...
MEASUREMENT:	Describe how measurements are to be made
	List ...
SHUT DOWN:	Actions to suspend testing for unscheduled events
	List ...
RESTART:	Procedure to restart testing from shutdown
	List ...
STOP:	Procedure for orderly halt to testing
	List ...
WRAP UP:	Actions after execution has been completed
	List ...
CONTINGENCIES:	Actions necessary to deal with anomalies
	List ...

Slide 23:

Slide 24:

Learning Objective #1

Which of the following is a FALSE statement?

Answer Choices

1. ITS standards define objects and dialogs to be tested
2. ITS standards provide format and steps for test procedures
3. Test procedures can merge with test cases
4. Test cases trace to requirements

Slide 25:

Learning Objective #1

Review of Answers

a) ITS standards define objects and dialogs to be tested
Incorrect. Objects defined in the ITS standards. The most recent version of several NTCIP standards include dialogs

b) ITS standards provide format and steps for test procedures
Correct! ITS standards do not provide formats and steps for TPS, ONLY IEEE 829 does.

c) Test procedures can merge with test cases
Incorrect. Std 829 links test cases as inputs to a test procedure.

d) Test cases trace to requirements
Incorrect. A test cases traces to requirements.

Slide 26:

Summary of Learning Objective #1

Recognize the Purpose and Structure of a Test Procedure

• Discussed the purpose and structure of a TPS based on IEEE 829 format and definitions
• Reviewed the inputs and expected outputs in a test procedure
• Reviewed special needs that arise from outside of ITS standards and contractual requirements

Slide 27:

Learning Objective #2: Identify the Role of a Test Procedure Specification Within a Test Plan and the Overall Testing Process

• Briefly review the role of Test Design Specification (TDS)
• Briefly review the role of Test Case Specification (TCS)
• Discuss test procedure steps

Slide 28:

Learning Objective #2

Brief Review of Test Design Specification (TDS)

TDS Details What a Test is to Demonstrate

• TDS is a document that:
  • Specifies the details of the test approach
  • Specifies what is to be verified
  • Identifies the associated tests
• Each test identified in the TDS may identify and detail:
  • A test of a single requirement
  • A test of a combination of multiple requirements

Slide 29:

Learning Objective #2

Example of TDS

TDS Outlines Requirements to Be Tested

NTCIP 1203 Protocol Requirements List (PRL)

USER NEED SECTION NUMBER	USER NEED	FR SECTION NUMBER	FUNCTIONAL REQUIREMENT	CONFORMANCE	SUPPORT/PROJECT REQUIREMENT	ADDITIONAL PROJECT REQUIREMENTS
2.3.2.3	DMS Display Matrix Configuration			M	Yes	The DMS shall be 9,000 millimeters wide (0 .65535) and 2,700 millimeters high (0. .65535), inclusive of borders. The Sign's Border shall be at least 400 millimeters wide (0 .65535) and 400 millimeters high (0..65535).
2.3.2.3.1	Non-Matrix			0.2 (1)	Yes/[No]
2.3.2.3.2 (Matrix)	Matrix			0.2(1)	[Yes]/No	The pitch between pixels shall be at least 66 millimeters (0..255).
2.3.2.3.2.1	Full Matrix			0.3 (1)	Yes/[No]	The sign shall be ____ pixels wide (0..65535) and ____ pixels high (0..65535).
2.3.2.3.2.2	Line Matrix			0.3 (1)	Yes/[No]	The siqn shall have ____ lines with each line being ____ pixels wide and ____ pixels high.
2.3.2.3.2.3	Character Matrix			0.3 (1)	[Yes]/No	The sign shall be 18 characters wide and 3 characters high with each character being 5 pixels wide (0. 255), 7 pixels high (0..255).

Source: NTCIP 1203 DMS

Slide 30:

Learning Objective #2

Brief Review of Test Case Specification (TCS)

TCS Specifies Conditions and Sequence of Verification

• TCS is a document that specifies:
  • The inputs to the test
  • The outputs of the test - the predicted results
  • A set of execution conditions for the test
• Section 1: Introduction
  • Document identifier
  • References to other documents
  • Context required outside of the TCS document
  • Notation description, such as numbering systems

Slide 31:

Learning Objective #2

Brief Review of TCS (continued)

TCS Specifies Purpose of Verification

• Section 2: Test Case Details
  • Test case identifier, unique to each
  • Test objective and focus
  • Inputs required to execute the test
  • Expected outcomes and behaviors
  • Environmental needs for setup, execution and recording the results
  • Special procedures
  • Interdependencies to other test cases
• Section 3: General
  • Glossary
  • Document change procedure and change history

Slide 32:

Learning Objective #2

Role of TPS within Test Plan & Testing Process

TPS Defines the Steps to Perform the Test

• TSP is a document that:
  • Specifies a sequence of actions (steps) for the execution of a test
  • TPS based on TDS and TCS
  • In general, TPS specifies the steps of the verification such as inspection, demonstration, analysis and a test procedure

Slide 33:

Learning Objective #2

Role of TPS within Test Plan & Testing Process

Requirements to Test Case Matrix (RTCM) : Example from NTCIP 1204 ESS v03, C.2.2

ANNEX C TEST PROCEDURES [NORMATIVE] - 152

C.1 Purpose - 152

C.1.1 Scope - 152

C.1.2 Keywords - 152

C.1.3 Rules for Following Test Procedures - 153

C.2 Testing Requirements - 153

C.2.1 Field Device Test Environment - 153

C.2.2 Traceability Table - 153

C.2.3 Test Procedures - 161

Table 30 Requirements to Test Case Traceability Table

Requirement		Test Case
ID	Title	ID	Title
3,5	Data Exchange Requirements
3.5.1	ESS Manager Requirements
3.5.1.1	ESS Configuration Requirements
3.5.1.1.1	Retrieve ESS Characteristics
		C.2.3.1.1	ESS Characteristics

Source: NTCIP 1204 ESS v03

Slide 34:

Learning Objective #2

Role of TPS within Test Plan & Testing Process

Test Workflow: Continued from Module T202

• Test Workflow Steps:
  • Overall Test Plan
  • Test Design Specification
  • Multiple Test Cases per TDS
  • Multiple Test Procedures
    • Each test procedure may cover one independent test case
    • Each test procedure may cover multiple dependent test cases

Slide 35:

Slide 36:

Learning Objective #2

In addition to inputs, outputs, and execution conditions, test case specification includes:

Answer Choices

1. Test objective to provide guidance to the test operator
2. Test environment hardware and software
3. Special procedures, such as automated tools
4. All of the above

Slide 37:

Learning Objective #2

Review of Answers

a) Test objective to provide guidance to the test operator
All are true.

b) Test environment hardware and software
All are true.

c) Special procedures, such as automated tools
All are true.

d) All of the above
Correct! All are true.

Slide 38:

Summary of Learning Objective #2

Role of Test Procedure Specification within a Test Plan and Overall Testing Process

• Briefly reviewed the role of Test Design Specification (TDS) as per Standard IEEE 829
• Briefly reviewed the role of Test Case Specification (TCS) as per Standard IEEE 829
• Discussed test procedure steps and overall role within a test plan and testing process

Slide 39:

Learning Objective #3: Synchronize the Test Procedure Specification to the Contract Terms and Conditions for Successful Contract Execution

• Discuss ITS project background
• Discuss how to structure contract terms and conditions from viewpoint of project end successfully

Slide 40:

Learning Objective #3

ITS Projects Background (cont.)

• Study has shown that of 280,000 Information Technology projects surveyed, 142,000 were late or over budget; 42,000 failed and were canceled
• Factors contributing to failures:
  • Lack of user involvement
  • Few limits on project scope
  • Lack of firm requirements
  • No consensus of project success at project kickoff

(Extended Text Description: Figure 6: Pie chart graphic reproduced from FHWA Systems Engineering Handbook, 2007, located in the upper right of the slide. The pie chart indicates 24% Succeeded, 15% Failed and 51% Challenged.)

Source: FHWA SE Handbook, 2007 See student supplement for reference

Slide 41:

Learning Objective #3

Synchronizing the TPS to the Contract Terms and Conditions

Start the Project with an Eye on the Finish Line

• In the beginning: What constitutes success at the end?
• For project acceptance at the end, project terms and conditions should include:
  • Design specifications
  • Test case specifications
  • Test procedures
  • Testing of interfaces by suppliers before integration

Slide 42:

Learning Objective #3

Synchronizing the TPS to the Contract Terms and Conditions (cont.)

• Expect systems engineering and planning effort to consume over 15% of the project budget to avoid ambiguities and system defects at project end
• Good planning includes test documentation

Slide 43:

Learning Objective #3

Synchronizing the TPS to the Contract Terms and Conditions (cont.)

• Terms and conditions should include IEEE 829 elements:
  • Test plan
  • Test design specification
  • Test case specification
  • Test procedures with "fill-in-the-blank" fields for test outputs

(Extended Text Description: Author's relevant notes: VEE model graphic reproduced from FHWA Systems Engineering Handbook, 2007, located to the right of the slide text. Above the V to the left, pointing to the left side of the V, is the text "Write Terms and Conditions here..." Above the V to the right, pointing to the right side of the V, is the text "as if you are here." The point of the graphic image is to convey the idea to synchronize the TPS to the contract trems and conditions.)

Slide 44:

Learning Objective #3

Synchronizing the TPS to the Contract Terms and Conditions (cont.)

Additional Concerns to be Aware Of

• Terms and conditions could also specify:
  • Third party test equipment model-requirements
  • Test scripts as deliverables
  • Test procedure for common test equipment and test scripts
  • Example: Type of ASC, ATC, 2070, etc.

Slide 45:

Slide 46:

Learning Objective #3

The TPS should be synchronized to the contract terms and conditions...

Answer Choices

1. Project ending without unexpected issues
2. Minimizing the project planning costs and time
3. Replicating the wording of a similar prior project
4. Enforcement after the project is late and over budget

Slide 47:

Learning Objective #3

Review of Answers

a) Project ending without unexpected issues
Correct! Contract terms including the tests allow the suppliers to test deliverables throughout the project, eliminating unexpected results.

b) Minimizing the project planning costs and time
Incorrect. Studies of IT project show that investing 15% or more of the total project cost in planning will save more than15% in cost overruns.

c) Replicating the wording of a similar prior project
Incorrect. Replication of "boilerplate" acceptance wording does not provide clear acceptance criteria for differing equipment and software.

d) Enforcement after the project is late and over budget
Incorrect. Enforcing design changes for unexpected results is more expensive than planning the final acceptance before the project begins.

Slide 48:

Summary of Learning Objective #3

Synchronize the TPS to the Contract Terms and Conditions for Successful Contract Execution

• Discussed ITS project background, how they present challenges
• Discussed how to structure contract terms and conditions from the viewpoint of project end successfully

Slide 49:

Learning Objective #4: Write the Reports Produced at the End of the Testing and Understand Their Relationship to Successful Procurement Contracts

• Logs, including data, information, files, and needs that are captured during the test
• Incident report, including a failure description and the investigation process
• Summary report, providing a measure of success compared to the stated goals and scope of the test plan

Slide 50:

Learning Objective #4

Reports Produced at the End of Testing and Their Relationship to Successful Procurement

Logs, Including the Data, Information, Files and Fulfilled Requirements That are Captured During the Test

• Outputs captured during the test
  • Test Logs
  • Test Data
  • Test Information
  • Test Files
• Output documentation is tailored to each test procedure from the formats available in IEEE 829
• We will demonstrate how the outputs are populated during a test procedure that includes more than one test case

Slide 51:

Learning Objective #4

Reports Produced at the End of Testing and Their Relationship to Successful Procurement

Workflow: Test Procedure Outputs

• We have created (gray):
  • Test Plan
  • Test Specification
  • Test Cases
  • Test Procedures
• Using test cases as inputs, test procedures will create output records (green):
  • Test Logs
  • Test Data
  • Test Information
  • Test Files

(Extended Text Description: Workflow: Test Procedure Outputs: Graphic identical to Slide 9 "abbrs" with another row of four green boxes located below labeled "Output Records 1", "Output Records 2", "Output Records 3" and "Output Records 4" respectively from left to right. An arrow extends from the bottom of "Test Procedure 1" to the top of "Output Records 1" and "Output Records 2" boxes. A second arrow extends from the bottom of "Test Procedure 2" to the top of "Output Records 3" box. A third arrow extends from the bottom of "Test Procedure 3" to the top of "Output Records 4".)

Slide 52:

Learning Objective #4

Reports Produced at the End of Testing and Their Relationship to Successful Procurement

Anomaly Report, Including a Failure Description and the Investigation Process

• Discovery of anomalies:
  • Evaluate the test procedure outputs
  • Identify data that is out-of-bounds of the expected result
  • Out-of-bounds data is unexpected and requires investigation
  • AKA "incidences", "errors", "problems", "defects", "issues"
• Anomaly report includes:
  • Unexpected results in the test procedure outputs
  • A description of each unexpected result
  • Investigation plan as the next step for disposition of each anomaly

Slide 53:

Learning Objective #4

Anomaly Report: IEEE Standard 829 Format

1. Introduction

1.1. Document identifier

1.2. Scope

1.3. References

2. Details

2.1. Summary

2.2. Date anomaly discovered

2.3. Context

2.4. Description of anomaly

2.5. Impact

2 6 Originator's assessment of urgency (see EEEE 1044-1993 [B13])

2.7. Description of the corrective action

2.8. Staros of the anomaly

2.9. Conclusions and recommendations

3. General

3.1 Document change procedures and history

Slide 54:

Learning Objective #4

Anomaly Report: IEEE Standard 829 Format

• Introduction
  • Document identifier
  • Scope of testing
  • References, such as standards documents
• Details
  • Summary: What was found to be out of bounds?
  • Date, context, description: When, Where, What?
  • Impact and urgency: Major or minor impact to project cost & schedule?
  • Action: Stop and fix or investigate further?
  • Status: Open, investigating, or resolved and closed? Who decides?
  • Conclusions and Recommendations: Next steps for stakeholders
• General: Document control numbers, versions, history

Slide 55:

Learning Objective #4

Anomaly Report: IEEE Standard 829 Format

• Description
  • Test inputs used
  • Expected outputs from the plan
  • Actual results observed
  • Gap between expected and actual
  • Test procedure step
  • Test environment
  • Attempts to repeat the test
  • Test operators, who conducted the test?
  • Observers

Slide 56:

Learning Objective #4

Level Test Reports Produced at the End of Testing

Level Test Report, Providing a Measure of Success Compared to the Stated Goals and Scope of the Test

Plan

• Level test report:
  • Measure of success
  • Identify gap between current situation and stated goals
  • Clear objective metrics based on anomaly report and remaining disposition of open issues
• Clear conclusions and recommendations:
  • Ready for deployment
  • Ready with minor modifications
  • Not ready

Slide 57:

Learning Objective #4

Level Test Report: IEEE Standard 829 Format

• For each level of integration test
• Introduction
  • Document ID
  • Scope of testing
  • References, if any
• Details
  • Overview of test results
  • Detailed test results
  • Rational
  • Recommendations
• General: Document control

1. Introduction

1.1. Docunient identifier

1.2. Scope

1.3. Reference?

2. Details

2.1. Overview of test re suits

2.2. Detailed test results

2.3. Rationale for decisions

2.4. Conclusions and recommendations

3. General

3.1. Glossary

3.2. Document change procedures and history

Slide 58:

Learning Objective #4

Example of a Level Test Report

• For ASC Timing
• Introduction
  • Document ID
  • Scope of testing
  • References, if any
• Details
  • Overview of test results
  • Detailed test results
  • Rational
  • Recommendations
• General: Document control

Slide 59:

Learning Objective #4

Master Test Report Produced at the End of Testing

• For all levels of integration test
• Introduction
  • Document ID
  • Scope of testing
  • References, if any
• Details
  • Overview of test results
  • Detailed test results
  • Rational
  • Recommendations
• General: Document control

1. Introduction

1.1. Document identifier

1.2. Scope

1.3. References

2. Details of the Master Test Report

2.1. Overview of all asgresate test results

2.2. Rationale for decisions

2.3. Conclusions and recommendations

3. General

3.1. Glossary

3.2. Document change procedures and history

Slide 60:

Slide 61:

Learning Objective #4

Which of the following statements is FALSE?

Answer Choices

1. Only one test case used as input to each test procedure
2. A test procedure can use multiple test cases as inputs

Slide 62:

Learning Objective #4

Review of Answers

a) Only one test case used as input to each test procedure
Correct! This statement is false. Test cases that are independent of each other will likely be the single input to a test procedure. However, several dependent test cases are often used as multiple Inputs to one test procedure.

b) A test procedure can use multiple test cases as inputs
Incorrect. This statement is true. Test cases that require similar equipment with related functions can be tested with one procedure.

Slide 63:

Summary of Learning Objective #4

Reports Produced at the End of Testing and Their Relationship to Successful Procurement

• Learned the format and content of logs, including data, information, files, and needs that are captured during the test
• Learned the format and content of an incident report, including a failure description and the investigation process, and an anomaly report
• Learned the format and content of the level test report and summary report, providing a measure of success compared to the stated goals and scope of the test plan

Slide 64:

Learning Objective #5: Using the Test Procedure Generator (TPG) to Develop the Test Procedures for a Sample TPS Structure

• Test Procedure Generator (TPG) Tool
  • Purpose of TPG
  • How to obtain a copy of the TPG
  • Show a quick overview of installation
  • Explain each step and the results
  • Learn how to how handle an error in a sample TPS structure
• Understand pre and postconditions, and different types of steps

Slide 65:

Learning Objective #5

Using a Tool to Develop the Test Procedures

Test Procedure Generator (TPG) for a Sample TPS Structure

• TPG is a software tool developed by USDOT
• Guides the development of uniform test procedures:
  • NTCIP standards that can typically use the TPG have:
    • Management Information Base (MIB)
    • With standardized dialogs
    • Requirements Traceability Matrix (RTM) defining performance
• Creates XML scripts
  • Consistent interpretation among stakeholders
  • Eliminates errors of manual creation of scripts
  • Can be used on any automated test equipment

Slide 66:

Learning Objective #5

Using TPG to Develop the Test Procedures

Test Procedure Generator (TPG)

• TPG is used to develop test procedures from NTCIP standards with:
  • Management Information Base (MIB)
  • Data exchange dialogs
  • Requirements Traceability Matrix (RTM)
• Advantages of using the TPG:
  • Creates scripts in commonly-understood XML format
  • Creates uniform scripts for use in multiple projects
  • Creates uniform scripts for each stakeholder within a project
  • Eliminates syntax errors and manual entry errors

Slide 67:

Learning Objective #5

Using a Tool to Develop the Test Procedures

Test Procedure Generator (TPG)

• Will be able to request a copy of TPG to be added to distribution list
• Will be able to download the TPG, including:
  • TPG executables
  • TPG installation procedure
  • TPG instruction manual

(Extended Text Description: Test Procedure Generator: Screenshot taken from TPG "Installation Wizard" located in the lower left of the slide. Author's key notes: A copy of the TPG can be obtained as a no-cost download. A copy of the TPG must be initiated with a request that identifies the requestor for records, such as notification of updates, etc. The request is acknowledged with a link to download the TPG, installation instructions and operations manual.)

Slide 68:

Learning Objective #5

How to Use TPG (cont.)

Opening a New TPG Session

(Extended Text Description: Opening a New TPG Session: Screenshot taken from TPG "New Session" occupying the entire slide. Author's key notes for illustration purposes: This slide shows the dialog box and screen when opening a new session.)

Slide 69:

Learning Objective #5

How to Use TPG (cont.)

Opening a Standard

(Extended Text Description: Extracting the Table of Contents data: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the dialog box and screen when opening a standard, which then shows the commands being executed in the progress window, such as Extracting the Table of Contents data and other related processes.)

Slide 70:

Learning Objective #5

How to Use TPG (cont.)

Create a New set of Test Procedures

(Extended Text Description: New Set of Test Procedures: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the dialog box, menu and screen when creating a new set of Test Procedures. The user can select the File menu, then select New Set of Test Procedures to begin the process.)

Slide 71:

Learning Objective #5

How to Use TPG (cont.)

Test Procedure 01.00

(Extended Text Description: Test Procedure 01.00: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the windows and screen when the screen displays Test Procedure 01.00. The Current Test Procedure tab is selected, indicating the current document.)

Slide 72:

Learning Objective #5

How to Use TPG (cont.)

Define the Test Procedure Header

(Extended Text Description: Define Test Procedure Header: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the menu and screen when Define Test Procedure Header is selected from the Test Procedure menu. The Current Test Procedure tab is selected, indicating the current document.)

Slide 73:

Learning Objective #5

How to Use TPG (cont.)

Test Procedure Header Window Defined

(Extended Text Description: Test Procedure 01.00: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the header window and screen after the definition of Test Procedure. In this case, the Test Procedure is titled Determine Sign Type and Technology. The Current Test Procedure tab is selected, indicating the current document.)

Slide 74:

Learning Objective #5

How to Use TPG (cont.)

Select a Requirement

(Extended Text Description: Select Requirements: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the menu and screen when the user selects the Test Procedure menu, then selects the option for Select Requirements. The Current Test Procedure tab is selected, indicating the current document.)

Slide 75:

Learning Objective #5

How to Use TPG (cont.)

Select a Requirement From Window

(Extended Text Description: Configure Logging Service: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the dialog box and screen when the user Selects Requirements, in this case 3.4.2.2 Configure Logging Service. The Current Test Procedure tab is selected, indicating the current document.)

Slide 76:

Learning Objective #5

How to Use TPG (cont.)

Requirements Populated

(Extended Text Description: Test Procedure 01.00: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the screen when Requirements are then populated, in this case, 3.4.1.1 through 3.4.2.2, for example. The Current Test Procedure tab is selected, indicating the current document.)

Slide 77:

Learning Objective #5

How to Use TPG (cont.)

Define Variables

(Extended Text Description: Define Variables: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the menu and screen when the user selects Test Procedure, then selects Define variables in the menus. The Current Test Procedure tab is selected, indicating the current document.)

Slide 78:

Learning Objective #5

How to Use TPG (cont.)

Saving a Test Procedure

(Extended Text Description: Save Test Procedure: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the menu and screen after completion, the Test Procedure is saved. The user selects Test Procedure in the menu, then selects Save Test Procedure to save the information.)

Slide 79:

Learning Objective #5

How to Use TPG (cont.)

Closing a Test Procedure

(Extended Text Description: Close Test Procedure: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the menu and screen after Close Test Procedure is selected from the Test Procedure Menu, the procedure is being saved and the Test Procedure is closed.)

Slide 80:

Learning Objective #5

How to Use TPG (cont.)

Save XML Set of Test Procedures

(Extended Text Description: Save XML Set of Test Procedures: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the menu and screen when the user selects File then Save XML Set of Test Procedures from the menu, allowing sets of Test Procedures to be saved as a set of XML.)

Slide 81:

Learning Objective #5

How to Use TPG (cont.)

XML Representation

(Extended Text Description: XML Report: Screenshot taken from TPG occupying the entire slide. Author's key notes for illustration purposes: This slide shows the screen with the XML Representation of the example, in this case NTCIP C2F Device Interface Standard. The Reports tab is selected indicating the current XML representation.)

Slide 82:

Learning Objective #5

How to Use TPG (cont.)

Other TPG Functionality

• Delete a test procedure step
• Resort test procedure steps
• Renumber test procedure steps
• Modify test procedure identification
• Edit a test procedure
• Print a test procedure

Slide 83:

Learning Objective #5

Understanding Pre and Postcondition and Different Types of Steps

• Preconditions and Postconditions:
  • Constructed of key words
  • Key words are consistent with the common understanding of the ITS standards
  • Key words entered are validated, creating consistent test procedures that are interoperable among manufacturers

Slide 84:

Slide 85:

Learning Objective #5

Which statement is TRUE? The TPG...

Answer Choices

1. Takes test cases directly as inputs
2. Guides test procedures having MIBs, Dialogs, and RTM
3. Creates outputs in commonly understood XLS files
4. Executes test procedures automatically

Slide 86:

Learning Objective #5

Review of Answers

a) Takes test cases directly as inputs
Incorrect. The TPG does not read in test cases. It pulls information from various sections of the standard and reads non-TPG created test procedures.

b) Guides test procedures having MIBs, Dialogs and RTM
Correct! The TPG is used to develop test procedure with uniform key words, dialogs and requirements traceability matrix to eliminate manual entry errors and to enable test procedures to be reused.

c) Creates outputs in commonly-understood XLS files
Incorrect. The TPG creates outputs in commonly-understood XML scripts.

d) Executes test procedures automatically
Incorrect. The TPG does not execute test procedures. TPG XML scripts can be used as inputs to automated test equipment.

Slide 87:

Summary of Learning Objective #5

How to Use Tools to Develop the Test Procedures for a Sample TSP Structure

• Introduced Test Procedure Generator (TPG) Tool and showed how to use it to produce test procedures
• Discussed pre and postconditions, and different types of steps

Slide 88:

What We Have Learned in Part 1

1. Test Procedure Specification (TSP) inputs are test cases used to create outputs of expected results and anomaly reports of unexpected results in IEEE 829 standard format.
2. A test design specification details what a test is to demonstrate, a test case specification is a specific example that assigns values, while a test procedure defines the steps to perform the test.
3. Contract Terms and Conditions should be viewed from the project's end, including test case specifications and test procedures.
4. A master test report measures project success to stated goals.
5. Test Procedure Generator (TPG) is an automated tool that generates XML scripts using consistent key words for interoperability.

Slide 89:

Resources

• IEEE 829: IEEE Standard for Software and System Test Documentation http://standards.ieee.org/findstds/standard/829-2008.html
• NTCIP 1204 v03 ESS, www.ntcip.org
• PCB Website: Module T201, T202 Available stds training.aspx
• Systems Engineering for Intelligent Transportation Systems, USDOT, January 2007 http://ops.fhwa.dot.gov/publications/seitsguide/seguide.pdf
• Center to Field Test Procedure Generator User Manual, v1.7, Federal Highway Administration, October 18, 2012

Slide 90:

Next Course Module

Module T204 Part 2 of 2:

How to Develop Test Procedures for ITS Standards-based Test Plan

Part 2 of 2

(We will cover Learning Objectives 6, 7, and 8)

Slide 91:

Next Course Module

Homework between T204 Part 1 and T204 Part 2

• Request a copy of the Test Procedure Generator at the address:
  • blake.christie@noblis.org
• Register at the same link to receive updates
• Download and install the TPG on a computer
• Create a simple Test Procedure to become familiar with the TPG before we begin T204 Part 2
• Have the TPG up and running when we begin Part 2

Slide 92: