Module 52 - T306
T306: Applying Your Test Plan to the Electrical and Lighting Management Systems Based on NTCIP 1213 ELMS Standard v03
HTML of the PowerPoint Presentation
(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)
Slide 1:
(Extended Text Description: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words "Standards ITS Training - Transit" in green and blue on the middle left side. The word "Welcome" in white is to the right of the logo. Under the logo box is the logo for the U.S. Department of Transportation, Office of the Assistant Secretary for Research and Technology.)
Slide 2:
(Extended Text Description: This slide, entitled "Welcome" has a photo of Ken Leonard, Director, ITS Joint Program Office, on the left hand side, with his email address, Ken.Leonard@dot.gov. A screen capture snapshot of the home webpage is found on the right hand side - for illustration only - from August 2014. Below this image is a link to the current website: www.its.dot.gov/pcb - this screen capture snapshot shows an example from the Office of the Assistant Secretary for Research and Development - Intelligent Transportation Systems Joint Program Office - ITS Professional Capacity Building Program/Advanced ITS Education. Below the main site banner, it shows the main navigation menu with the following items: About, ITS Training, Knowledge Exchange, Technology Transfer, ITS in Academics, and Media Library. Below the main navigation menu, the page shows various content of the website, including a graphic image of professionals seated in a room during a training program. A text overlay has the text Welcome to ITS Professional Capacity Building. Additional content on the page includes a box entitled What's New and a section labeled Free Training. Again, this image serves for illustration only. The current website link is: https://www.its.dot.gov/pcb.)
Slide 3:
T306: Applying Your Test Plan to the Electrical and Lighting Management Systems Based on NTCIP 1213 ELMS Standard v03
© Jim Frazer 2017
Slide 4:
Instructor
James J. Frazer
President
Gridaptive Technologies
Pompano Beach, FL, USA
Slide 5:
Learning Objectives
- Describe ELMS Testing
- Describe ELMS Test Plan Application
- Identify Relevant Elements of an ELMS Test Plan
- Describe Adaptation of a Test Plan
Slide 6:
Learning Objective 1
- Describe ELMS Testing
Slide 7:
Describe ELMS Testing
The testing life cycle, the role of test plans, and the testing to be undertaken for Electrical and Lighting Management Systems (ELMS)
- Why We Test ELMS
- Purpose of an ELMS Test Plan
-
Components of an ELMS Test Plan
- Test Design Specification
- Test Case Specification
- Test Procedure Specification
Slide 8:
Why We Test
To confirm that an ELMS will work as intended
The testing process provides objective evidence that the system:
- Satisfies the system requirements
- Solves the right problem
- Satisfies the user needs
Slide 9:
Why We Test
Testing and the Systems Life Cycle
(Extended Text Description: Author's relevant description: This slide has a graphic of a blue large letter "V" which is a graphical representation of systems engineering process. Systems life cycle diagram shows testing to be undertaken at various stages, as well as the types of testing to be utilized. In this case, System Validation, System Verification and Deployment, Subsystem Verification and Unit/Device Testing are highlighted in red. A very detailed explanation of how a Vee diagram is used in PCB modules is provided below. This level of detail is not necessary and may be skipped. Explanation of V Diagram A graphic of the systems engineering process (SEP) is shown. The main graphic of the SEP is a V-shaped diagram with some additional horizontal "wings" on the left and right side of the top of the V. Starting from the left "wing" the steps are regional architecture, needs assessment, concept selection, project planning, and systems engineering management planning. At this point the steps begin to descend the left side of the V with concept of operations, system requirements, high-level design, detailed design, and software hardware and field installation. At this point the steps begin to ascend the right side of the V with unit device testing, subsystem verification, system verification and deployment, system validation, and operations and maintenance. At this point the steps are on the right "wing" of the V with changes and upgrades and retirement/replacement. The following arrows supplement the figure: 1. A time line at the bottom of the figure indicating that all steps proceed over time with the steps forming the V overlapping one another. 2. A downward arrow on the left side of the V that indicates that these steps deal with decomposition and definition. 3. An upward arrow on the right side of the V that indicates that these steps deal with integration. 4. A horizontal arrow near the bottom of the center of the V that connects the detailed design step to the unit testing step. 5. A horizontal arrow a little higher that connects the subsystem requirements step with the subsystem verification step. 6. A horizontal arrow a little higher that connects the system requirements step with the system verification step. 7. A horizontal arrow near the top of the V that connects the concept of operations step with the system validation step. Finally, major steps are separated by a barrier that is labeled "document approval." This graphical view of SEP life cycle process used to show where SSM user needs/requirements are located. In this slide, at the top of the Vee, there are three boxes in yellow.)
Slide 10:
Why We Test
To confirm that an ELMS system will work as intended
(Extended Text Description: Author's relevant description: This slide has a graphic of a blue large letter "V" which is a graphical representation of systems engineering process. Focus upon the traceability of testing to the underlying requirements and foundational user needs. Systems life cycle diagram shows testing to be undertaken at various stages, which includes Test Planning, Testing Documentation (Test Plan), and Testing Process. Note that test planning is done earlier than test design documents. Further description of the Vee diagram is found in previous slide 9.)
Slide 11:
Purpose of a Test Plan
Does the system conform to the requirements?
-
A test plan is a document describing:
- Scope (technical management)
- Approach
- Resources needed
- Schedule to complete
-
A test plan identifies:
- Test items
- Features to be tested
- Testing tasks
- Risks requiring contingency plan
Testing determines whether the system conforms to the requirements and whether it satisfies its intended use and user needs (IEEE-829-2008).
Slide 12:
Components of a Test Plan
Relationship between Components
(Extended Text Description: Author's relevant description: This slide has a graphic of the hierarchical relationship between test plan components. A test plan provides a description of the overall approach to testing all of the requirements to be verified. The Test Plan outlines the scope, approach, resources, and schedule of testing activities. Breakdown of Key Point: This first key point describes a test plan – the master document that will include the test cases. Explaining this key point shows the hierarchical structure (test plan – test design specification – test case – test procedures) that is required to develop a fully system-engineered test plan. To the left of the diagram are bullet points with arrows corresponding to the related sections of the diagram:
- Test Plan Specification
- Test Design Specification
- Test Case Specification
- Test Procedure Specification
Image Courtesy of ITS PCB Course T321, Slide 92.)
Slide 13:
Components of a Test Plan
Test Plan Specification
- Test plan specifications detail objectives, target market, internal beta team, and processes for a specific test for a software or hardware product
- Test plan specifications contain a detailed understanding of the comprehensive workflow
Slide 14:
Components of a Test Plan
Test Design Specification
- Test design specifications result in a collection of test cases that are intended to be used to test a specified set (test suite) of behavior
- A test suite contains detailed instructions or goals for each collection of test cases and information on the system configuration to be used during testing
Slide 15:
Components of a Test Plan
Test Case Specification (TCS)
A test case specification is a set of conditions under which a tester will determine whether the system is working as it was originally intended to do
Slide 16:
Components of a Test Plan
Test Procedure Specification
- Test procedure specification defines a process that produces a test result
- It is a technical operation that consists of determining the characteristics of a given product, process, or service according to a specified procedure
Slide 17:
Slide 18:
Question
Which is not a component of an ELMS test plan?
Answer Choices
- Test Facilitation
- Test Design Specification
- Test Case Specification
- Test Procedure Specification
Slide 19:
Review of Answers
a) Test Facilitation
Correct! Test facilitation is not part of an ELMS test plan.
b) Test Design Specification
Incorrect. Test Design Specification is part of an ELMS test plan.
c) Test Case Specification
Incorrect. Test Case Specification is part of an ELMS test plan.
d) Test Procedure Specification
Incorrect. Test Procedure Specification is part of an ELMS test plan.
Slide 20:
Learning Objectives
- Describe ELMS Testing
- Describe ELMS Test Plan Application
Slide 21:
Learning Objective 2
- Describe ELMS Test Plan Application
Slide 22:
ELMS Test Plan Application
Steps in Developing an ELMS Test Plan
- Identify requirements to be tested/not to be tested for each testing phase
- Identify test methodology
- Introduce and describe the Requirements to Test Case Traceability Matrix (RTCTM)
- Plan logistics of testing
- Estimate level of effort for testing
- Evaluate risks
- Plan project closeout
Slide 23:
ELMS Test Plan Application
Develop a Sample Test Plan
Identify Requirements to Test:
- Requirements are found in the Protocol Requirements List (PRL)
-
Module A306b identified how to define ELMS requirements
- See Participant Student Supplement for list of sample requirements
-
Every requirement should be tested:
- During at least one test phase
- Using at least one method
- By at least one party
- Extent of agency testing is a risk management issue
Slide 24:
ELMS Test Plan Application
Develop a Sample Test Plan
Identify Test Plan Level
-
Each test level will have its own test plan ° Prototype
- Design Approval
- Factory Acceptance
- Incoming Device
- Site Acceptance
- Burn-in
-
Often further divided
- NTCIP testing
- Hardware testing
- Etc.
Slide 25:
ELMS Test Plan Application
Develop a Sample Test Plan Approach
- Identify Test Methodology
- Inspection
- Analysis
- Demonstration
- Formal testing
-
Consider testing scenarios
- Positive test(s)
- Negative test(s)
- Boundary test(s)
Slide 26:
ELMS Test Plan Application
Develop a Sample Test Plan
Requirements to Test Case Traceability Matrix (RTCTM)
| Requirement ID | Requirement | Test Case ID | Test Case |
|---|---|---|---|
| 3.5.4.1.1.1 | Retrieve Luminaire Pole Identifier | 3.5.4.1.1 | Retrieve Luminaire Pole Identifier |
| 3.5.4.1.1.2 | Retrieve Luminaire Location | 3.5.4.1.1.2 | Retrieve Luminaire Location |
| 3.5.4.1.3 | Configure Luminaire Mode | 3.5.4.1.3.1 | Configure Luminaire Mode |
| 3.5.4.1.3.2 | Incorrectly Configure Luminaire Mode | ||
| 3.5.4.1.4.1 | Configure Luminaire Color Temperature | 3.5.4.1.4.1.1 | Configure Luminaire Color Temperature |
| 3.5.4.1.4.1.2 | Incorrectly Configure Luminaire Color Temperature |
Slide 27:
ELMS Test Plan Application
Develop a Sample Test Plan
Identify the Test Environment
(Extended Text Description: Author's relevant description: This slide includes an image of a subset of a device under test communication with a test application. A data analyzer is pictured "listening" to the communications stream. Focus upon: The test environment needs to be defined in terms of hardware and software required.)
Source: NTCIP 8007, page 13
Slide 28:
ELMS Test Plan Application
Develop a Sample Test Plan
Plan Logistics of Testing
-
Where will tests be performed?
- Safety during on-site testing
-
Who is responsible for what?
- Power
- Tools
- Tables
- Protection from elements
- Local assistance for remote testing
- What happens if testing is suspended?
Slide 29:
ELMS Test Plan Application
Develop a Sample Test Plan
Estimate effort, schedule, and budget for:
- Preparing test plan
- Preparing test cases
- Preparing test procedures
-
Performing multiple rounds of testing
- Performing tests
- Investigating problems
- Preparing test documentation
Slide 30:
ELMS Test Plan Application
Develop a Sample Test Plan
Understanding the Impact of a Failure
| Requirement ID | Requirement | Test Case ID | Test Case |
|---|---|---|---|
| 3.5.4.1.1.1 | Retrieve Luminaire Pole Identifier | 3.5.4.1.1 | Retrieve Luminaire Pole Identifier |
| 3.5.4.1.1.2 | Retrieve Luminaire Location | 3.5.4.1.1.2 | Retrieve Luminaire Location |
| 3.5.4.1.3 | Configure Luminaire Mode | 3.5.4.1.3.1 | Configure Luminaire Mode |
| 3.5.4.1.3.2 | Incorrectly Configure Luminaire Mode | ||
| 3.5.4.1.4.1 | Configure Luminaire Color Temperature | 3.5.4.1.4.1.1 | Configure Luminaire Color Temperature |
| 3.5.4.1.4.1.2 | Incorrectly Configure Luminaire Color Temperature |
Slide 31:
ELMS Test Plan Application
Develop a Sample Test Plan
Understanding the Impact of a Failure
| User Need ID | User Need | Requirement ID | Requirement |
|---|---|---|---|
| 2.5.2.1.1.1 | Retrieve Luminaire Information | ||
| 3.5.4.1.1.1 | Retrieve Luminaire Pole Identifier | ||
| 3.5.4.1.1.2 | Retrieve Luminaire Location | ||
| 3.5.4.1.1.3 | Retrieve Luminaire Mode | ||
| 3.5.4.1.1.4 | Retrieve Luminaire Zone | ||
| 3.5.4.1.1.5 | Retrieve Luminaire Vendor Information | ||
| 3.5.4.1.1.6 | Retrieve Luminaire Light Source Type | ||
| 3.5.4.1.1.7 | Retrieve Luminaire Wattage | ||
| 3.5.4.1.1.8 | Retrieve Luminaire Voltage | ||
Slide 32:
ELMS Test Plan Application
Develop a Sample Test Plan
Plan Project Closeout
- Have a plan
- Understand the impacts of accepting a failure
Slide 33:
Slide 34:
Question
Which of the following ELMS statements is false?
Answer Choices
- Every ELMS requirement should be tested
- You should only need to perform your ELMS test plan once
- Some ELMS testing may be performed by the manufacturer
- ELMS traceability tables can help you assess the impact of a test failure
Slide 35:
Review of Answers
a) Every ELMS requirement should be tested
True. Every requirement should be tested.
b) You should only need to perform your ELMS test plan once
False (correct). This statement is not true. Testing will often reveal problems; these should be fixed and the device retested.
c) Some testing may be performed by the manufacturer
True. Testing may be performed by the agency, the manufacturer, or a third party.
d) ELMS traceability tables can help you assess the impact of a test failure
True. Traceability tables allow you to identify the user needs that will not be completely fulfilled.
Slide 36:
Learning Objectives
- Describe ELMS Testing
- Describe ELMS Test Plan Application
- Identify Relevant Elements of an ELMS Test Plan
Slide 37:
Learning Objective 3
- Identify Relevant Elements of an ELMS Test Plan
Slide 38:
What Is Being Tested?
Only Project-Specific Requirements Are Tested
(Extended Text Description: This slide contains the following table with values circled in red in the Support column: rows 2, 4, 6, 7: Yes, rows 5 and 8: No:
| User Need ID | User Need | FR ID | Functional Requirement | Conformance | Support | Additional Specifications |
|---|---|---|---|---|---|---|
| 2.5.2.2.2 | Control Electrical Service | 0 | Yes / No | |||
| 3.5.5.2.1 | Control Electrical Service by Permanent/Continuous Override | M | Yes | |||
| 3.5.5.2.2 | Control Electrical Service by Transitory Override | 0 | Yes / No | |||
| 3.5.5.2.3 | Control Electrical Service by Timed Override | 0 | Yes / No | |||
| 3.5.5.2.4 | Control Electrical Service in Stagger Mode | 0 | Yes / No | |||
| 3.5.5.2.5 | Control Electrical Service by Photocell | 0 | Yes / No | |||
| 3.5.5.2.6 | Control Electrical Service by Adaptive Means | 0 | Yes / No | |||
)
Slide 39:
Test Design, Test Cases, and Test Procedures
Designing Test Case Specifications and Procedures
- Review guidance from IEEE 829-2008 and NTCIP 8007
- Apply guidance to sample dialog
-
Key Differences Between the Two Approaches
- IEEE standard approach is applicable to all ITS standards including C2C and C2F
- IEEE standard approach separates test cases from test procedures, while previous efforts combined both, such as per NTCIP 8007 information report
- IEEE standard approach allows reuse of test procedures, where agencies typically place more efforts
- IEEE standard approach includes a test plan and method to split testing into test designs, and includes test reports
Slide 40:
Test Design, Test Cases, and Test Procedures
Designing Test Case Specifications and Procedures
What Does IEEE 829-2008 Provide?
- Test Plan
- Test Design Specification
- Test Case Specification
- Test Procedure Specification
-
Test Reports
- Test Logs
- Test Anomaly Report
- Test Report
- Testing professionals across ITS are familiar with these definitions/formats
Slide 41:
Test Design, Test Cases, and Test Procedures
Designing Test Case Specifications and Procedures
NTCIP 8007 Components
-
NTCIP 8007 describes how these can be combined for NTCIP testing
- Test case identifier
- Purpose
- Inputs
- Pass/Fail criteria
-
Procedure steps
- Can reference other often used procedures
- Expected outputs
- Features tested
- Defines terms that can be used in test steps for NTCIP testing
Slide 42:
Test Design, Test Cases, and Test Procedures
Designing Test Case Specifications and Procedures
Sample Basic Dialog: Set Time Dialog
(Extended Text Description: Author's relevant description: This slide has an image of a communications message between a management station and an ELMS device. The slide is an example of a simple dialog between a Management Station and an ELMS device. A comprehensive list of dialogs is included in the standard. Test Case also seeks to test boundaries with Data objects variable ranges; that part is missing. Dialogs are tested for sequence of message content.)
Slide 43:
Test Design, Test Cases, and Test Procedures
Designing Test Case Specifications and Procedures
-
Specify Test Case First
- "What You Are Testing"
-
Then Specify Test Procedure
- "How You Run The Test"
Slide 44:
Test Design, Test Cases, and Test Procedures
Designing Test Case Specifications
Specify Each Test Case
| Requirement ID | Requirement | Test Case ID | Test Case |
|---|---|---|---|
| 2.2.1 | Set Time | 2.2.1 | Set Time |
| Test Case | ||
| Test Case 2.2.1 | Test Case Name | Set Time |
| Description | This test case verifies that the ELMS properly tracks time. It advances the clock by a user-defined amount, waits a few seconds, retrieves the time, and verifies it indicates an appropriate value. | |
| Variable | GlobalTime as defined in NTCIP 1213 V3.0 | |
| Pass/Fail Criteria | The DUT shall pass every verification step included within the Test Case to pass the Test Case. | |
Slide 45:
Test Design, Test Cases, and Test Procedures
Designing Test Case Procedures
- Data exchanges should follow defined dialogs
- Return the device to its original state (generally)
- Verification steps should cite the relevant requirement
- A test case typically tests multiple requirements
- NTCIP 8007 precisely defines standardized step types
- A "SET" operation includes nine specific verification checks related to the Simple Network Management Protocol (SNMP) response packet
Slide 46:
Test Design, Test Cases, and Test Procedures
Designing Test Case Procedures
Steps of a Sample Procedure
| Step Number | Test Procedure | Results |
|---|---|---|
| 1 | CONFIGURE: Determine the number of seconds to advance the clock in the ELMS | |
| 2 | GET the following object(s): globalTime.0 | Pass/Fail |
| 3 | RECORD the RESPONSE VALUE for globalTime.0 as Start_Time | |
| 4 | SET the following object(s): globalTime.0 = Start_Time + Time_Offset | Pass/Fail |
| 5 | DELAY for 15 seconds | |
| 6 | GET the following object(s): globalTime.0 | Pass/Fail |
| 7 | VERIFY that the RESPONSE VALUE for globalTime.0 is roughly equal to Start_Time + Time_Offset + 15 | Pass/Fail |
Slide 47:
Adapting the Test Plan
The process of adapting the test plan based on selected user needs and requirements
- We have described the components of a test plan
- We have examined the major components of test cases and test procedures in detail
- Next we will create a project-specific ELMS test plan
Slide 48:
Slide 49:
Question
Where can you find definitions for terms that can be used in NTCIP test steps?
Answer Choices
- IEEE 829
- NTCIP 8007
- ISO 9001
- Student Supplement
Slide 50:
Review of Answers
a) IEEE 829
Incorrect. IEEE 829 defines sample outlines for test documentation, but does not define steps for NTCIP.
b) NTCIP 8007
Correct! NTCIP 8007 defines a number of terms that can be used in test steps for NTCIP testing.
c) ISO 9001
Incorrect. ISO 9001 deals with quality management, but does not deal directly with NTCIP testing.
d) Student Supplement
Incorrect. The student supplement provides samples of test procedures, but it does not define the test terms.
Slide 51:
Learning Objectives
- Describe ELMS Testing
- Describe ELMS Test Plan Application
- Identify Relevant Elements of an ELMS Test Plan
- Describe Adaptation of a Test Plan
Slide 52:
Learning Objective 4
- Describe Adaptation of a Test Plan
Slide 53:
Develop an NTCIP 1213 v03 Test Design Specification
Background
Information Sources:
NTCIP 1213 v03 - National Transportation Communications for ITS Protocol Object Definitions for Electrical and Lighting Management Systems
- Protocol Requirements List (PRL)
- Requirements Traceability Matrix (RTM)
-
Testing documentation
- Required but not supplied in standard
- Must be created
Slide 54:
Develop an NTCIP 1213 v03 Test Design Specification
Background
Characteristics of the NTCIP 1213 v3.0 (ELMS) Standard
- ELMS is a Center-to-Field Communications Standard
- ELMS contains System Engineering (SE) content (the standard has a PRL and an RTM)
- ELMS does not contain Test Procedures
Slide 55:
Develop an NTCIP 1213 v03 Test Design Specification
Background
Protocol Requirements List (PRL)
- Contains user needs
- Contains functional requirements
- Describes relationship between needs and requirements
- Project-specific requirements are identified by project-level Protocol Requirements List (PRL)
Slide 56:
Develop an NTCIP 1213 v03 Test Design Specification
Background
Requirements Traceability Matrix (RTM)
- Contains functional requirements
- Contains object dialogs
- Describes relationship between requirements and object dialogs
- A project-specific Requirements Traceability Matrix (RTM) references relevant design content needed to define the inputs and outputs for the test case specification
Slide 57:
Develop an NTCIP 1213 v03 Test Design Specification
Background
Context Diagram
(Extended Text Description: Author's relevant description: This slide has an image of a communications message between a management station and an ELMS device. This is an example of the electricalserviceSwitchState dialog. Each dialog includes a numeric object identifier ID as well as an object name. Each project-specific object will need to be examined.)
Slide 58:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 1: Select Your User Needs in the PRL
Slide 59:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 2: Use Project RTM to Identify Objects and Dialogs to Be Tested
| Requirements Traceability Matrix (RTM) | |||||
| FR ID | Functional Requirement | Dialog ID | Object ID | Object Name | Additional Specifications |
|---|---|---|---|---|---|
| 3.5.5.2.2 | Control Electrical Service by Transitory Override | G.3 | |||
| 5.5.1.6 | electricalserviceSwitchMode | ||||
| 3.5.5.2.3 | Control Electrical Service by Timed Override | 4.2.13 | |||
| 5.5.1.6 | electricalserviceSwitchMode | ||||
| 5.5.1.7 | electricalServiceSwitchModeTime | ||||
| 3.5.5.2.4 | Control Electrical Service in Stagger Mode | G.3 | |||
| 5.5.1.28 | electricalserviceSwitchState | ||||
| 3.5.5.2.5 | Control Electrical Service by Photocell | G.3 | |||
| 5.5.1.29 | electricalservicePhotocellIndex | ||||
| 3.5.5.2.6 | Control Electrical Service by Adaptive Means | G.3 | |||
| 5.5.1.6 | eletricalserviceSwitchMode | ||||
Slide 60:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 3: Develop Test Case Objective
(Extended Text Description: Author's relevant description: This slide includes an image of a test case objective. In this example we have defined one objective addressing four requirements. Notice that the test case verifies that the data value of the OBJECTS requested are within specified ranges. The key text of the image includes:
To verify system interface implements (positive test case) requirements for a series of object requests for:
3.5.5.2.2 electricalserviceSwitchMode
3.5.5.2.3 electricalserviceSwitchModeTime
3.5.5.2.4 electricalserviceSwitchState
3.5.5.2.5 electricalservicePhotocellIndex
The test case verifies that the data value of the OBJECTS requested are within specified ranges. The object identifier (OID) of each object requested is the only input required. An output specification is provided to show valid value constraints per the NTCIP 1205 v01 object definitions.)
Slide 61:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 3: Develop Test Case Objective (continued)
| Test Case Output Specification | |||
| ID:TCOS001 | Title: Status Condition within the Device | ||
| Data Concept ID | Data Concept Name (Variable) | Data Concept Type | Value Constraints |
|---|---|---|---|
| 3.5.5.2.2 | elertrcialserviceSwitchMode | Data Element | |
| 3.5.5.2.3 | electrcialserviceSwitchModeTime | Data Element | |
| 3.5.5.2.4 | electrcialserviceSwitchModeState | Data Element | |
| 3.5.5.2.5 | electrcialservicePhotocellIndex | Data Element | |
Slide 62:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 4: Identify Dialogs, Inputs, Outputs
(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. This is a definition from the standard. The key is to notice how Type and Valid Range are defined here as octet string and a range of 1 to 9.)
Slide 63:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 4: Identify Dialogs, Inputs, Outputs (continued)
(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. This is another definition from the standard. The key is to notice how Type and Valid Range are defined here as integer and a range of 0 to 65535.)
Slide 64:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 5: Document Value Constraints for Inputs
(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. From the standard, we enter the value constraints into the Test Case Input Specification. The Value Constraints are highlighted in red in the table below:
| Test Case Input Specification | |||
| ID TCI201 | Title: Input Specification for electricalserviceSwitchMode (Positive test case) | ||
| Data Concept ID | Data Concept Name (Variable) | Data Concept Type | Value Constraints |
|---|---|---|---|
| 3.5.5.52.2 | electricalserviceSwitchMode | Data Element |
1 = "permanentOn" 2 = "permanentOff" 3 = "schedule" 4 = "transitoryOn" 5 = "transitoryOff" 6 = "timedOn" 7 = "timedOff" 8 = "none" 9 = "adaptive" |
)
Slide 65:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 5: Document Value Constraints for Outputs
(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. We also enter the value constraints into the Test Case Output Specification. At this point it is important to remember that negative testing is also important. The Value Constraints are highlighted in red in the table below:
| Test Case Output Specification | |||
| ID TCI201 | Title: Output Specification for electricalserviceSwitchMode (Positive test case) | ||
| Data Concept ID | Data Concept Name (Variable) | Data Concept Type | Value Constraints |
|---|---|---|---|
| 3.5.5.52.2 | electricalserviceSwitchMode | Data Element |
1 = "permanentOn" 2 = "permanentOff" 3 = "schedule" 4 = "transitoryOn" 5 = "transitoryOff" 6 = "timedOn" 7 = "timedOff" 8 = "none" 9 = "adaptive" |
)
Slide 66:
NTCIP 1213 v03 Test Design Specification
Testing Documentation
Step 6: Complete Test Case
(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. In this slide we are completing a simplified test case. Notice the additional items of Environmental Needs, Tester/Reviewer, Special Procedure Requirements, and Intercase Dependencies. Those areas are highlighted in red in the table below:
| Test Case | |
| ID TCI201 | Title: electricalserviceSwitchMode Dialog Verification (Positive Test Case) |
| Objective | To verify system interface implements(positive test case) requirements for object: electricalserviceSwitchMode |
| Inputs | Use valid inputs as defined by test case input specification |
| Outcomes | All data are returned and verified as correct: correct sequence of message exchanges, structure of data, and valid value of data content. See Test Case Output Specification for details. |
| Environmental Needs: | No additional needs outside of those specified in the test plan. |
| Tester/Reviewer | JF |
| Special Procedure Requirements | None |
| Intercase Dependencies | None |
)
Slide 67:
Developing Test Cases and Procedures for Extensions
Supporting Objects Not in the Standard
Extending the Standard complicates interoperability and interchangeability
- Not achievable unless all design details are known
-
Extensions are relatively custom solutions, resulting in:
- Increased specification costs
- Increased development costs
- Increased testing costs
- Increased integration costs
- Longer deployment timeframe
- Increased maintenance costs
Slide 68:
Developing Test Cases and Procedures for Extensions
Supporting Objects Not in the Standard
Extensions should only be considered when:
- NTCIP features are inadequate to meet needs
- Benefits of extension outweigh added costs
Slide 69:
Developing Test Cases and Procedures for Extensions
Supporting Objects Not in the Standard
Extended equipment should be designed to:
- Appropriately integrate with NTCIP-only deployments
- Minimize added complexity
Slide 70:
Developing Test Cases and Procedures for Extensions
If You Do Choose to Test Objects Not in the Standard
- Adhere to the relationships between the PRL, RTM, and RTCTM, as well as the underlying user needs and measurable functional requirements
- The main purpose of Test Design is to identify the features to be tested by a particular level test (e.g., unit test)
-
The features to be tested are included in the RTCTM
- Based on a Requirements Traceability Matrix (RTM)
Slide 71:
Test Procedure Generator Tool (TPG)
Introduction to the Test Procedure Generator
- What is the TPG?
- Why is the TPG important?
- What are the benefits?
- How do you use the TPG?
- How does it fit into testing for NTCIP Standards?
- Where does a user obtain the TPG?
Slide 72:
Test Procedure Generator Tool (TPG)
What Is the TPG and How Does It Work?
- USDOT has released the version 2.1 of the TPG tool for the ITS Standards communities
- TPG v2.1 supports development and deployment NTCIP Center-to-Field (C2F) Device Interface Standards with Systems Engineering Content
- TPG is a Windows-based software tool that uses Microsoft Word to input the NTCIP Standards and output Test Procedures
- TPG supports ITS Standard developers as well as deployers (local and state agencies) of NTCIP C2F Standards
Slide 73:
Test Procedure Generator Tool (TPG)
What Is the TPG and How Does It Work?
- For deployers and local agencies, the TPG guides the development of test procedures by: Loading and processing the standard to be implemented including the requirements, dialogs, and objects
- Basing the Test Procedures on the user-selected requirements in NTCIP C2F Standard
Slide 74:
Test Procedure Generator Tool (TPG)
What Is the TPG and How Does It Work?
-
Uses standardized and consistent language for Test Procedures development, including:
- Standard keywords, variables, and object names imported directly from the standard
- Outputs an XML file that can be consistently interpreted by vendors and testing staff for their test suites
- Standards Deployers can use the TPG to create consistent Test Procedures
- Remember: The TPG is not a testing tool!
Slide 75:
Test Procedure Generator Tool (TPG)
Benefits of the TPG
- Agencies can use the TPG to develop consistent test procedures for verifying conformance and compliance
- Using the TPG tool will reduce developmental risks, effort, and the cost of developing standards and test procedures
Slide 76:
Test Procedure Generator Tool (TPG)
(Extended Text Description: Author's relevant description: This slide includes an image that describes the focus of the TPG as test cases and test procedures. As described previously, the Test Program Generator Tool (TPG) automates steps 3 and 4 "Test Cases" and Test Procedures" of the test planning process. It does not address the Test Plan, Test Design Specifications, Test Execution, or Test Reports.)
Slide 77:
Test Procedure Generator Tool (TPG)
Role of the TPG in Testing
- Supports off-the-shelf interoperability
- Promotes the systems engineering process by giving users support in creating test procedures
- Standardized and easily available Test Procedures that are conformant to the standard help to eliminate the proprietary system elements
Slide 78:
Test Procedure Generator Tool (TPG)
Using the TPG - Start a New Session
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software splash screen. The first step in using the TPG is to start a new session by loading the NTCIP 1213 v03 standard in Word doc 2010 format. The New Session dialog shows several options, including: NTCIP C2F Device Interlace Standard Number, NTCIP Standard Major Version Number, NTCIP Standard Minor Version Number, NTCIP Standard Revision Letter (Optional).)
Slide 79:
Test Procedure Generator Tool (TPG)
Using the TPG - The Graphical User Interface
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software user interface. This slide represents the graphical user interface of the TPG Tool. A tree view (the session panel) is on the left. Session status is in a ribbon at the bottom. The interface shows TPG Menu Items, Document Tabs, Embedded Microsoft Word 2010 Document Menu Items, Embedded Microsoft Word 2010 Document, NTCIP 1213 version v03, TPG Session Status, TPG Command Status.)
Slide 80:
Test Procedure Generator Tool (TPG)
Using the TPG - Create a New Test Procedure
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "create a new test procedure" interface. The next step (Step 2) is to create a new Test Procedure.
A Current Test Procedure windows displays the following example table:
| Test Procedure: 01.00 | Select the Test Procedure->Define Header Menu Item to enter the Test Procedure Title |
| Description: | Select the Test Procedure->Define Header Menu Item to enter the Test Procedure Description |
| Require me nt(s): | Select the Test Procedure->Select Requirements Menu Item to enter the Test Procedure Requirements |
| Variable(s): | Select the Test Procedure->Define Variables menu item to enter the Test Procedure Variables |
| Pass/Fail Criteria: | Select the Test Procedure->Define Header Menu Item to enter the Test Procedure Pass/Fail Criteria |
)
Slide 81:
Test Procedure Generator Tool (TPG)
Using the TPG - Create a New Test Procedure
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "create a new test procedure" interface. Creating a new Test Procedure begins with defining a Test Procedure Title and Description.)
Slide 82:
Test Procedure Generator Tool (TPG)
Using the TPG - Select Your Requirements
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "select new requirements" interface. Step 3 is to "Select Your Requirements" by checking the checkboxes of project-specific requirements. Some example requirements include: 3.4.1.1 Retrieve Data, 3.4.1.2 Deliver Data, 3.4.1.3 Explore Data, 3.4.2.1 Determine Current Configuration of Logging Service.)
Slide 83:
Test Procedure Generator Tool (TPG)
Using the TPG - The Test Procedure
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "completed test procedure" interface. This slide represents the Test Procedure as defined so far. Example content includes: Current Test Procedure, Test Procedure: 01.00, Determine Sign Type and Technology, Description: This test case verifies that the DMS indicates that it is the sign type and uses the technology as required by the specification.)
Slide 84:
Test Procedure Generator Tool (TPG)
Using the TPG - Create Your Variables
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "create your variables" interface. Step 4 is to select your variable objects. Example Create a New Variable dialog has example variables: Select Test Procedure Variable(s): Required_Sign_Technology [eventClassClearTime], Required_Sign_Type [INTEGER].)
Slide 85:
Test Procedure Generator Tool (TPG)
Using the TPG - Create Your Test Procedure Step
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "test your procedure step" interface. Step 5 is to create your Test Procedure Step. Test Procedure Step dialog is shown.)
Slide 86:
Test Procedure Generator Tool (TPG)
Using the TPG - Test Procedure Results
(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "your procedure test results" interface. This slide represents your project-specific Test Procedure results. Example Set of Test Procedures, Pass/Fail Criteria: The device under test (DUT) shall pass every verification step included within the Test Case in order to pass the Test Case. Example columns show Test Step Number, Test Procedure, Results.)
Slide 87:
Test Procedure Generator Tool (TPG)
How to Obtain the TPG
- TPG v2.1 updates include: Compatibility with Windows 7 Professional
- Compatibility with Microsoft Office 2010
-
For more information and to acquire the TPG, please visit:
https://www.standards.its.dot.gov/DeploymentResources/Tools -
The free download package includes:
- TPG v2.1 Installation file
- TPG User Manual
- TPGSupport@noblis.org
Slide 88:
Slide 89:
Question
Which of the following statements is false?
Answer Choices
- TPG v2.1 supports development and deployment NTCIP Center-to-Field (C2F) Device Interface Standards with Systems Engineering Content
- TPG is a testing tool
- TPG is a Windows-based software tool that uses Microsoft Word to input the NTCIP Standards and output Test Procedures
- TPG supports ITS Standard developers as well as deployers (local and state agencies) of NTCIP C2F Standards
Slide 90:
Review of Answers
a) TPG v2.1 supports development and deployment NTCIP Center-to-Field (C2F) Device Interface Standards with Systems Engineering Content
Incorrect. TPG does support development and deployment NTCIP Center-to-Field (C2F) Device Interface.
b) TPG is a testing tool
Correct! False, TPG is not a testing tool.
c) TPG is a Windows-based software tool that uses Microsoft Word to input the NTCIP Standards and output Test Procedures
Incorrect. TPG is a Windows-based software tool.
d) TPG supports ITS Standard developers as well as deployers (local and state agencies) of NTCIP C2F Standards.
Incorrect. TPG supports ITS Standard developers as well as deployers.
Slide 91:
Module Summary
- Describe ELMS Testing
- Describe ELMS Test Plan Application
- Identify Relevant Elements of an ELMS Test Plan
- Describe Adaptation of a Test Plan
Slide 92:
We Have Now Completed the ELMS Curriculum
Module A306a:
Understanding user needs for Electrical and Lighting Management Systems Based on NTCIP 1213 v03
Module A306b:
Specifying requirements for Electrical and Lighting Management Systems Based on NTCIP 1213 v03
Module T306:
Applying Your Test Plan to the Electrical and Lighting Management Systems Based on NTCIP 1213 v03
Slide 93:
Thank you for completing this module.
Feedback
Please use the Feedback link below to provide us with your thoughts and comments about the value of the training.
Thank you!