Next Generation 9-1-1 (NG9-1-1) System Initiative

Proof of Concept Testing Report

Acknowledgements

The Next Generation 9-1-1 Initiative at the U.S. Department of Transportation gratefully acknowledges the contributions of all participants to the Proof of Concept. Without their time and expertise, the successful completion of the Proof of Concept would not have been possible.

Public Safety Answering Point (PSAP) Personnel:

PSAP Facilities:

    King County E-911 System, Seattle WA
    State of Montana Public Safety Services Bureau, Helena MT
    Ramsey County 911, St. Paul MN
    Kosciusko County 911, Warsaw IN
    Rochester – Monroe County Emergency Communications Department, Rochester NY

Proof-of-Concept Laboratories:

    Booz Allen Hamilton, Center for Network & Systems Innovation (CNSI)
    Columbia University, Internet Real Time Laboratory
    Texas A&M University, Internet2 Technology Evaluation Center

Next Generation 9-1-1 Initiative Project Team:

    Booz Allen Hamilton
    National Emergency Number Association
    L. Robert Kimball & Associates
    Texas A&M University Research Foundation

Vendors:

    Cisco Systems
    Emergent Communications
    INdigital Telecom
    OnStar
    Pictometry International
    RedSky Technologies
    WirelessWERX


Table of Contents

1    Introduction

2    Methodology

3    Test Site User Training
4    Initial Laboratory Site Testing
5     PSAP Site Testing

6    Total System Testing

7  Demonstration Testing

8   Lessons Learned and Future Research. 

9   Compliance Matrix
Appendix A—Acronyms
Appendix B—Glossary
Appendix C—Source References
Appendix D—POC Testing Results
Appendix E—POC Data Acquistion and Analysis Results


The following report contains the brand names of software and hardware used in the USDOT NG9-1-1 Proof of Concept, as well as the names of companies and agencies that participated.  Their inclusion does not represent an endorsement of any kind.

1    Introduction

This report documents the methodology and results of the U.S. Department of Transportation (USDOT) Next Generation 9-1-1 Initiative (NG9-1-1) Proof of Concept (POC) testing.  It describes in detail the NG9-1-1 POC test results and provides a baseline for future testing of NG9-1-1 systems.

The NG9-1-1 POC demonstration provided a realistic implementation and served as the culmination of previous NG9-1-1 efforts including—

• NG9-1-1 Concept of Operations (CONOPS)
• High-Level Requirements document
• Interim System Design Document
• Human Machine Interface Display Design Document
• POC Deployment Plan
• POC Testing Plan
• POC Data Acquisition and Analysis Plan.

These documents were used as a basis for planning, executing, and testing the NG9-1-1 POC system.  These documents are available for download at the USDOT NG9-1-1 website: http://www.its.dot.gov/ng911.

1.1    Overview

The following subsections provide brief overviews of the NG9-1-1 project, the POC, and the POC testing performed.

1.1.1    Overview of the NG9-1-1 Project

The NG9-1-1 Initiative is a USDOT research and development project that was established to help define the concept of operations, functional requirements, and system architecture and to develop a transition plan that considers responsibilities, costs, schedule, and benefits for deploying Internet Protocol (IP)-based emergency services across the Nation.  To accomplish these goals, USDOT has worked closely with public and private 9-1-1 stakeholders throughout the public safety community.

Trends in telecommunications mobility and convergence have put the 9-1-1 system at a crossroads.  The growing market penetration of both wireless and Voice-over-Internet-Protocol (VoIP) telephony, and the increasingly nomadic and mobile world they reflect, has underscored the limitations of the current 9-1-1 infrastructure.  The Nation’s 9-1-1 system, based on 1960s technology, cannot handle the text, data, images, and video that are increasingly common in personal communications and critical to future transportation safety and mobility advances.  Although improvements to 9-1-1 technology have occurred over the last four decades to enhance the data provided to the call taker (e.g., Enhanced 9-1-1 [E9-1-1], Wireless 9-1-1, and VoIP), the core 9-1-1 technology remains unchanged.  The limitations of today’s 9-1-1 infrastructure prevent the easy transport of data, critical sharing of information, and enhanced awareness that ultimately reduces decision-making ability and quality of service provided to emergency callers.

USDOT views the NG9-1-1 project as a transition enabler, further assisting the public to make a 9-1-1 “call” from any wired, wireless, or IP-based device, and allowing the emergency services community to take advantage of enhanced call delivery and advanced functional and operational capabilities through new internetworking technologies based on open standards.  By enabling the public to access 9-1-1 services through virtually any communications device, the NG9-1-1 system provides a more direct ability to request help or share critical data with an emergency services provider from any location.  In addition, call takers at the Public Safety Answering Points (PSAP) will be able to transfer emergency calls to other PSAPs and forward the location and other critical data, such as text messages, images, and video, with the call.

The intent of the USDOT NG9-1-1 project is to provide leadership, guidance, and resources to work with the public and private 9-1-1 stakeholders, and lay out a path to achieve a phased migration of nationally interoperable emergency service networks.

1.1.2    Overview of the Proof of Concept

The NG9-1-1 POC provided the opportunity to develop and deploy software and network components that demonstrate the desired capabilities of the NG9-1-1 system.  These included—

• Call origination using—

• IP User Agents (UA) such as laptop computers, IP telephones, IP wireless devices, and Session Initiation Protocol (SIP) clients (audio, data, text, and streaming video
• Cellular devices (audio, Short Message Service [SMS], Instant Messaging [IM]
• Third-party call center (audio and  telematics data
• IP and Video Relay Services (VRS) for the deaf and hard-of-hearing community (real-time text and video).

• Call support and processing using—

• Standard IP access networks
• NG9-1-1 network components such as Emergency Services Routing Proxy (ESRP) and data gateways
• NG9-1-1 databases such as Business Rules, Location-to-Service Translation Protocol (LoST), Enterprise Location Information Server (LIS), and Call Record.

• Call termination at the PSAPs using—

• IP Automatic Call Distribution (ACD) systems
• IP telephones and workstations
• Human machine interface (HMI).

To add a realistic perspective to the POC testing and demonstration, USDOT approached the PSAP community to participate in the POC effort.  Three test laboratories and five selected PSAPs hosted the infrastructure for the NG9-1-1 POC demonstration.  The three laboratory facilities that housed a majority of the NG9-1-1 infrastructure were—

• Booz Allen Hamilton Center for Network & Systems Innovation (CNSI), located at the One Dulles office in Herndon, Virginia
• Texas A&M Internet2 Laboratory, located in College Station, Texas
• Columbia University Next Gen Laboratory, located in New York, New York.

After completion of configuration and testing at the test bed locations, equipment and software was installed at the selected PSAP locations.  The five PSAP locations were—

• City of Rochester—Emergency Communications Department, Rochester, New York
• King County E-911 System, Seattle, Washington
• Metropolitan Emergency Services Board—Ramsey County Emergency Communications Center, St. Paul, Minnesota
• State of Montana—Public Safety Services Bureau, Helena, Montana
• Kosciusko County 911, Warsaw IN.

1.1.3    Overview of the Testing to Be Performed

The testing process was based on the NG9-1-1 requirements document using the Tier 1 requirements.  The requirements tested are grouped into two main segments: System Operations and PSAP Operations.  The POC Test Plan was created to describe the detailed test processes to validate the functional requirements.  By employing “use cases” or scenarios, the requirements have been integrated into industry-standard testing methodologies, which were applied to the laboratory test bed environment, the PSAP environment, and an end-to-end system test.

 

To accomplish testing, all tests used POC equipment, systems, and test data, and at no time, were live calls handled on the POC system or were the test calls sent to live 9-1-1 systems.  To accommodate the sheer number of tests and limited testing staff, the testing occurred in phases.  The most complicated components were housed at the various laboratory sites, and as a result, they were tested first by the project team.  Subsequently, the individual PSAPs were tested, using trained participants from the host agency or organization.

During the testing phases, each technology type (wireline, cellular, SMS, telematics, and IPUA) was used to deliver a call to the system.  Call origination typically occurred from the laboratories, but due to the flexibility of the POC system configuration and the physical location of the testers, some calls were generated directly from the PSAP locations. 

Regardless of technology being used, the Test Call followed the same general Call Flow.  A call was originated from a lab or PSAP onto some form of access network (Step 1 in figure below).  Within the access network, the location of the caller was acquired and additional call stream parameters were added, if these data elements were not already provided by the end user device (Steps 2,3).  The system then used the location and call stream parameters to route the call through the NG9-1-1 system to the appropriate PSAP based on various NG9-1-1 business rules (Steps 4-8).  If needed, additional call information could be obtained from external supportive data sources while the call was in transit (Step 6).  The call was then terminated at a PSAP Automatic Call Distributor (ACD) where its call stream parameters were analyzed and subsequently the call was routed to the most appropriate Call Taker based on the Call Taker’s availability and capabilities (Step 9-10).  Once the call was terminated with a Call Taker, additional external supplemental data sources could be queried for further information and a comprehensive call record was created and stored to a Call Record database (Step 11, 12).  For further details on the system architecture and design refer to the POC System Design Document.

The NG9-1-1 POC was evaluated on several levels, both objectively and subjectively.  Based on basic pass/fail criteria for each test, the system was graded to determine how it performed in a functional and operational manner.  From a subjective perspective, feedback and input from the end users was recorded to provide for avenues of future study and investigation.  Performance data was stored to measure how the NG9-1-1 POC system responded to the testing and can be used to identify opportunities to review and/or optimize future networks or systems.

The three laboratory sites were tested at the beginning of the POC, prior to any PSAP-based testing.  Although the three laboratories housed the equipment, logically they operated as a single system.  Seven use cases were tested at the laboratories, and testing was completed by placing test calls to a workstation at the Booz Allen laboratory.  During these initial tests, 47 requirements were tested, with 39 (83 percent) successfully passing.  This lower pass rate can be attributed to the limited maturity of the POC software at the time of testing.  The laboratory tests focused on the call setup and routing of calls, while the individual PSAP-based testing focused more on the call termination and handling.

At the 5 PSAPs, 26 professional call taker, dispatch, and supervisory personnel were trained to assist with the POC testing.  Involving end users in the testing process provided a valuable benefit to the overall effort.  These users were able to provide subjective feedback about functionality which someday they will come to depend on.  Their input about their needs and the difficulties currently faced in today’s 9-1-1 environment will help ensure that current problems are addressed by emerging NG9-1-1 technology.  During the PSAP-based testing, 273 functional requirements were tested, with 241 (88.3 percent) successfully passing.  While no industry benchmarks exist that gauge the success of emergency service network implementations, the team felt accomplished in having successfully demonstrated a significant portion of the NG9-1-1 concepts and use cases during the POC.

Based on the goals of the NG9-1-1 POC, the following summarizes the initial results—
 

Overall, very positive feedback was received from the POC participants, both those involved in the testing and in the demonstrations.  The POC helped to “de-mystify” NG9-1-1 by calming fears and answering pressing questions.  The 9-1-1 community generally recognizes that a fundamental transformation of the way 9-1-1 calls are originated, delivered, and handled is slowly underway.  The POC helped create a sense of urgency and movement within the community to get more involved and to start discussing the issues.  The POC generated new discussions among stakeholder organizations—a key process to the success of NG9-1-1—and one that often does not occur today.

1.2    Scope

Limitations in available time and funding made it necessary to limit the technical scope of the POC.  As a result, the project team focused its efforts on the 9-1-1 call from origination to delivery and handling by a public safety call taker.  While recognizing that the needs of an emergency caller do not stop with the call taker, dispatch and field operations that support the emergency response were deemed outside the scope of the project.  The scope limitations also narrowed the number of requirements tested as part of the POC.

A number of requirements require further evaluation, testing, and possible revision.  In addition, it was necessary to simulate certain components of the POC, including—

• Integration with Emergency Providers Access Database (EPAD)
• Business Rules Database
• Integration with dispatch and responder agencies (e.g., computer-aided dispatch systems).

The scope of this document is limited to the process and results of testing the Tier 1 functionality of the USDOT NG9-1-1 system requirements.  The testing took place during June and July 2008.  Testing occurred in a manner consistent with the POC Test Plan, and results were recorded during the test process.  Data was gathered during the testing process and later analyzed for recommendations and conclusions, which are presented in subsequent sections of this document.

1.3    Document Overview

This NG9-1-1 POC Test Report is organized into the following numbered sections:

• 1 Introduction—This section presents an overview of the project and POC.
• 2 Methodology—This section outlines the process and logistics of the POC testing.
• 3 Test Site User Training—This section describes the training that was delivered to the test sites.
• 4 Initial Laboratory Site Testing—This section outlines the testing that was conducted at each of the three laboratory sites.
• 5 PSAP Site Testing—This section outlines the PSAP testing that was conducted at each of the selected PSAP sites.
• 6 Total System Testing—This section outlines the testing that was conducted with the total system.
• 7 Demonstration Testing—This section outlines the demonstrations that were conducted during the public demonstration phase.
• 8 Lessons Learned—This section contains a description of lessons learned during the testing.
• 9 Compliance Matrix—This section contains a summary of the test results in table format.
• Appendix A—Acronyms—This appendix contains acronyms used in the test plan document.
• Appendix B—Glossary—This appendix contains a glossary of terms used in the test plan.
• Appendix C—Source Documents—This appendix lists the documents used as references to develop the test plan.
• Appendix D—POC Testing Results—This appendix details the testing results from each test site.
• Appendix E—POC Data Collection Results—This appendix contains the results from the data collection detailed in the Data Acquisition and Analysis Plan.

2    Methodology

The POC testing was designed to test the high-level (Tier 1) requirements as developed and documented in the NG9-1-1 Concept of Operations (CONOPS) and High-Level Requirements documents.  These requirements should not be considered an exhaustive list of all NG9-1-1 functional requirements but rather a fundamental set of requirements that form the baseline for future NG9-1-1 systems.

The NG9-1-1 POC system development lifecycle generated several documents that outline the functions of an NG9-1-1 system.  As part of the development process, the NG9-1-1 team developed a set of use cases that covered the basic functions of an NG9-1-1 system.  Each of these use cases was further decomposed into a set of requirements.  These were documented in the System Description and High-Level Requirements Document.  Based on these requirements, the NG9-1-1 POC Test Plan was developed.

The NG9-1-1 POC Test Plan lists each use case with its associated requirements.  A test was then developed for each requirement.  Each test associated with a requirement is described, along with the test procedure.  For each requirement, the plan provided the following information—

• Test Description—Brief overview of the test
• Test Procedures—Required test steps
• Expected Results—What was expected to happen
• PASS/FAIL—General indicator of success or failure of the test
• Results—Documentation of the test outcome.

While the plan provides a set of test steps associated with each requirement, there is considerable interrelationship among the requirements.  Consequently, a single test call was often used to test more than one requirement at a time.

Since the System Test Plan was written before all development had completed, occasionally the test procedures did not accurately reflect the proper operation of the system.  However, in many cases the actual functional requirement was still met by the software.  When this occurred, the functional requirement received a PASS, and the details of how the requirement was met were added to the notes section of the test case in Appendix D.

Each test was rated as a PASS, FAIL, or NOT TESTED:  PASS was assigned if the system was able to fully meet the requirement; FAIL was assigned if the system did not meet the requirement in its entirety; and NOT TESTED was used if no testing was conducted or the functionality was not developed in time for the functionality to be tested.

2.1    Testing Process

The testing proceeded in six phases using a top-down approach.  Major NG9-1-1 POC functionality housed in the laboratories at Booz Allen Hamilton, Columbia University, and Texas A&M was tested first.  Next, more detailed aspects of the call origination process were strategically tested at the five PSAP locations.  Finally, the entire end-to-end system was comprehensively tested to ensure all Tier 1 functionality had been provided by the software development team.  Once end-to-end testing was complete, the team conducted demonstration tests at the five PSAP locations.  During all testing phases, the team also acquired data on the operation and performance of the NG9-1-1 POC system.  Time was built into the testing process for retesting.  As software issues were discovered, the software development team would fix bugs and release new versions of the software.  The test team would then conduct basic acceptance testing of the new software release to ensure the problem had been resolved and that no new issues had arisen with the previously validated functionality.  The following subsections (2.1.1–2.1.6) give an overview of testing process and its various phases.  The subsequent sections (3–8) then go into further detail about each phase and discuss the findings and issues that were discovered during that specific phase of testing.

2.1.1    Initial Laboratory Site Testing

The laboratories were tested as a group to ensure that the main NG9-1-1 infrastructure they housed worked as intended.  The seven most significant use cases were tested to ensure basic operation of the POC system.  The Texas A&M and the Booz Allen CNSI laboratories originated calls using each technology type.  Calls were answered using the call taker workstation located at the Booz Allen CNSI laboratory, which simulated the infrastructure that would eventually be housed at each of the five PSAPs.

2.1.2    Individual PSAP Site Testing

Each PSAP was tested individually as a separate operating entity.  The equipment housed in the PSAPs was intended to work independently and was tested as such.  In addition, no conference or transfer functionality was built into the software during this initial phase of PSAP testing.  Nine use cases were tested at each PSAP.  The test team went to each PSAP and conducted the tests for a 2-day period.  The test team usually consisted of two team personnel and members of the PSAP staff.  A development and support team was also located at the laboratory sites as support for call origination.  During this phase, the network-to-network connections were also demonstrated at the Indiana state system.  All tests used POC equipment, systems, and test data.  At no time were live calls handled on the POC system or were the test calls sent to live 9-1-1 systems.

2.1.3    Total System Testing

The total system was tested as a unit.  The testing scripts developed for the POC Test Plan were modified for the actual testing with specific technologies and additional steps.  Seven tests were developed and tested.  During the 2-day test period, test team and PSAP personnel were present at each of the five PSAP locations, and the development and support team was located at the three laboratories. 

2.1.4    Demonstration Testing

At the completion of testing, demonstrations were held to allow input from  a variety of 9-1-1 stakeholders regarding the processes and systems of the NG9-1-1 POC.  A demonstration was prepared that included graphical (animated) slides describing the processes that each call source used.  These slides were used to visualize and explain the processes that occur prior to and following placement of the test call.  Over a period of 3 weeks, seven demonstrations were conducted, one at each of the five PSAP locations, one at the Texas A&M Laboratory, and one at the Booz Allen Hamilton CNSI Laboratory.

2.1.5    Data Collection

Data for the POC were collected in accordance with the Data Acquisition and Analysis Plan.  The details of the data acquisition during the POC and the analysis that was conducted are discussed in Appendix E of this document.

2.1.6    Retesting (If needed)

The POC was a research and development project.  Consequently, a series of software versions and changes to the systems were developed and integrated cyclically during the testing process.  To support the demonstration timelines, retesting occurred in an ad hoc manner during the complete testing process and whenever key features were released by the software development team.  Given the sheer number of changes to the software after PSAP testing, retesting was completed prior to moving into the end-to-end system testing or demonstration testing. 

2.2    Testing Resources

The testing personnel, software versions, and schedules are described in the following paragraphs.

2.2.1    Testing Personnel

The testing personnel consisted of three groups of personnel:

• The Testing Team consisted of seven people who performed the testing at the various sites.  One of the testing team staff was at the Booz Allen laboratory site to assist the test teams in the field for most of these tests.  In general, during the PSAP testing, two test team personnel were present at each site during the tests.  During system testing, there was usually only one person at each site.
• The PSAP Staff was critical to the success of the testing.  This staff operated the call taker workstation and provided feedback regarding the systems and processes during the testing.
• The Development and Support Team was available during the testing to troubleshoot problems and provide answers to technical questions.  The development staff also made frequent updates to the systems and software as problems were discovered.  The support staff also assisted by generating calls to the systems as needed to initiate or complete a test.

2.2.2    Software Revisions Tested

Several revisions and releases of the Call Taker HMI software occurred during the testing process.  The table below outlines these revisions.

 

2.2.3    Scheduling

Given the research-oriented nature of the project, the POC Test Plan schedule was designed to allow for flexibility.  The final testing schedule provided ample yet stringent timelines for the test team to travel to the various sites, test the NG9-1-1 infrastructure and its associated functionality, and then document the findings.

3    Test Site User Training

In preparation for the POC testing, the staff from each PSAP received a 4-hour block of training to use the call taker software.  During this training, the PSAP staff provided feedback.  The software was then revised based on the feedback.  The training covered—

• USDOT NG9-1-1 Initiative Overview
• NG9-1-1 POC Overview
• POC Testing Logistics
• POC Software Overview
• POC Software Demonstration.

The following table lists the training dates, number of attendees, and comments from each site.

4    Initial Laboratory Site Testing

At the laboratory sites, seven use cases were used for testing.  All three laboratory sites were tested at the same time.  The requirements associated with the use cases below were tested from a workstation at the Booz Allen CNSI laboratory.  The table and narrative below summarize the activities.  Detailed test results are included in Appendix D of this report.

June 4–5, 2008
On the scheduled testing dates, the software was not fully completed.  The testing team used the testing time to familiarize itself with the software and systems to be used for the POC.  The team reviewed the test scripts and determined items that needed to be completed.  These tests were not documented on the test scripts; rather, input was provided to the development team for further refinement of the systems and software. 

June 12, 2008
The testing team began the testing by updating the Call Taker HMI software to the June 6, 2008, 10:24:12 p.m. version.  The testers then reviewed the latest version of software and placed initial test calls to familiarize themselves with the functionality.  The test scripts were completed.  A few areas of the software testing was not completed:

• Bridging was not fully implemented.
• Video errors caused the video calls to freeze.
• Call queuing was not working.
• Some data was not being saved to the call detail record.

July 9, 2008 Retesting
The testing team began the testing by reviewing the software and initial test calls.  The Sipcalltaker software version tested was July 3, 2008, 7:03:08 p.m.  This testing completed the tests that had not been completed in prior test sessions.  There were some problems during the testing with slow response from services on the network, and therefore, the testing was completed with the Booz Allen firewall removed.
 

5    PSAP Site Testing

Each of the five PSAP test sites was tested with a set of test scripts.  These scripts were detailed in the POC Test Plan.  This section summarizes the activities and results.  The detailed test results are included in Appendix D of this report.

5.1    King County, Washington, PSAP Site Testing

At the King County PSAP site, the requirements associated with the nine use cases listed below were used for testing.  During testing, the PSAP staff was involved and performed many of the tester functions.

 

The test equipment was located in the offices of the King County E9-1-1 Program Office.  The various PSAPs in the King County system provided dispatch staff to operate the call taker workstation during the testing.

PSAP testers provided significant feedback, including—

• The screen refreshed inappropriately after a call was automatically received.
• Call details did not update after HOLD was activated.
• The call queue was not under the active call screen.
• Sound volume for recorded calls of people on HOLD was too low.
• Sipcalltaker reused the last touched Internet Explorer© window to update the map information rather than using a dedicated application instance.
• There did not seem to be an indication to the call taker when the caller dropped the call from his/her end.
• Each time a note in the call detail report was generated, it re-displayed all previous information in the screen, whereas call takers would expect a simple timestamp and the new note entry.
• When location information was updated for the call, another location table was created in the log detail area rather than appending changes to one location table.
• A unique identifier for each call should be attached to the call so two people can ensure they are talking about the same thing when looking at logs or playback.
• More hyperlinks should be enabled from the call detail report, such as a map of the location information.
• In the future, call takers should have the ability to check databases, such as the National Law Enforcement Telecommunications System (NLETS), National Crime Information Center (NCIC), or local databases, regarding the reporting party or subjects.

5.2    Helena, Montana, PSAP Site Testing

At the Helena PSAP site, the requirements associated with the nine use cases listed below were used for testing.  During testing, the PSAP staff was actively involved and performed many of the testing functions.

The test equipment was located in an equipment room of a telecommunications provider.  The PSAPs in the area provided staff to operate the call taker workstations during the testing and provided feedback.

June 19, 2008
The software version built on June 18, 2008, at 10:43:03 p.m. was used.  An initial conference bridge was established with the Rochester NY PSAP test site and the Booz Allen CNSI laboratory.  A cellular telephone was used because there was no wireline conference bridge telephone at the Helena site.  A brief overview of the POC, the screen, and the various available functions and features was conducted.

PSAP testers provided feedback on call taker software functions, including—

• A recommendation was made to provide functionality that would automatically take a call off hold if it had been on hold for defined time (such as 2 minutes).
• The call taker should be able to enter answers when going through the scripts (POC did not have this feature).
• Stakeholders commented that all of the provided data might not be relevant for the call taker or medical first responders—call takers would process call in the same manner.  The stakeholders questioned why some data was listed.
• It was suggested that call takers should be able to use the appropriate telematics-provided data to run inquiries via state NLETS-related systems.  Moreover, it would be helpful to be able to use supportive/supplemental data to run inquiries to other external/internal databases.
• Call takers wanted to know whether POC NG9-1-1 software could send a reply message to a text message initiator indicating that a text message was received.  Consensus of the group was that this should be possible for all calls.

June 20, 2008
The software version built on June 20, 2008, at 2:19:41 a.m. was used.  An initial conference bridge was established with the Rochester NY PSAP test site and the Booz Allen CNSI laboratory.  Cellular telephones were used because there was no wireline conference bridge telephone at the Helena site.

PSAP testers provided feedback on call taker software functions, including—
Outgoing calls from the PSAP were not stored in call records.
Color could be improved so data (such as caller location, etc.) would stand out better.  Background color could be improved in the data area associated with a call, and buttons/toggles should change colors when the cursor is passed over them and when the buttons/toggles are in use.  (Different colors would be better than just shades of a single color.)
It was suggested that call takers should have the ability to change the latitude/longitude to another view than that delivered; this would aid those who use other display methods for latitude/longitude in other systems (such as the degree/minute/second measurement system that some Global Positioning System [GPS] units use).

June 25, 2008
The software version built on June 24, 2008, at 11:57:27 p.m. was used.  Retesting took place to complete tests not previously conducted.

5.3    St. Paul, Minnesota, PSAP Site Testing

At the St. Paul PSAP site, the requirements associated with the nine use cases listed below were used for testing.  During testing, the PSAP staff was involved and performed many of the tester functions.

The test equipment was located on the PSAP’s dispatch floor.  The PSAP provided staff to operate the call taker workstation during the testing and provided feedback.

June 16, 2008
The software version built on June 15, 2008, at 9:52:17 p.m. was used.  Feedback included the following:

• Requirement D.8.10 should be reworded to indicate that the call taker screen should be refreshed, not the call detail record.
• Requirement D.11.2 should be reworded.  The system automatically placed the initial caller location information into the emergency location fields of the HMI screen, but the emergency location was added to the call detail record only if the information was changed on the screen.  The changed information entered by the call taker did not appear to update the call stream as shown by the SIP message tool.
• The read-only fields were grayed out.  It was suggested that users tend to ignore items that are grayed out.

June 17, 2008
The software version built on June 15, 2008, at 9:52:17 p.m. was used.  Feedback included the following:

• Call Detail Records should display with the most current record first.
• Address/Location entry box should be formatted in a logical manner.

5.4    Rochester, New York, PSAP Site Testing

At the Rochester PSAP site, the requirements associated with the nine use cases listed below were used for testing.  During testing, PSAP staff was involved and performed many of the tester functions.

The test equipment was located in the Training/Emergency Operations Center (EOC) room at the Rochester PSAP.  The PSAP provided staff to operate the call taker workstation and provided the following feedback:

• The ability to automatically see calls in queue would be preferred.  Tester had three calls in queue but no data in Call Queue.  On a second attempt, after two calls were made, the Call Queue refreshed and two calls were listed in queue.
• Additional calls in queue could not be answered until the first call was released.
• Audio and video recorded the caller while the call was “on busy” and on hold.
• Call takers suggested the system should display a pop-up screen asking “Are you sure?” before a call was released.
• Call taker “Ready” or “Available” status was too small.

5.5    Indiana PSAP Site Network-to-Network Testing

At the Indiana PSAP site, the requirements associated with the nine use cases listed below were used for testing.  During testing, PSAP staff was involved and performed many of the tester functions.

The test equipment was located in the Training/EOC room at the Kosciusko County PSAP in Warsaw, Indiana.  The PSAP provided staff to operate the call taker workstation and provided the following feedback:

• The provided mapping information could not be used on the call taker workstations during testing.
• After an SMS message was received, Call Taker Status had to be manually changed to “available.”
• Caller heard music in background on IP UA calls.
  

6    Total System Testing

After completion of the component testing in the laboratory and PSAP testing phases, the total system was tested.  This testing consisted of processing calls through the complete system and recording the results.  During the testing, the PSAP staff was involved and operated the call taker workstation.  This testing took place on June 25–26, 2008.

On June 26, 2008, the test sites were offered the opportunity to invite local agencies to observe the testing.  Several sites took advantage of this chance to showcase the testing that was occurring at their respective PSAPs.

To cover all of the possible scenarios, the PSAPs were configured to use the following answer types and transfer points.  A PSAP whose Answer Type was designated as “Manual” relied on Call Takers to physically acknowledge and accept incoming calls.  Alternatively, a PSAP also had the option of designating an “Automatic” Answer Type in which the IP ACD and Call Taker Software worked autonomously to acknowledged and answered incoming calls and then direct them to the appropriate Call Taker.

The basic scripts were broken down into three sections:

• Call Setup
• Call Handling
• Data Handling.

These sections of the test scripts were developed to cover each of the Tier 1 requirements.  The use cases covered in each section of the test script are listed below.

   
The specific test scripts that were used for each use case were changed from those described in the test plan.  The table below summarizes the technologies tested.  The detailed test results are included in Appendix D of this report.

During the second day of total system testing, the system was used by all five PSAP locations as well as the three laboratory sites.  This did put stress on the systems, and some problems were experienced, which are described below.

Network Response Time Extended
Call takers began to report that the call taker workstation software took a second or more to refresh the screen with the caller’s information when they answered a call.  They also noticed some of the other screens (Map, Call Detail Record) were slow to open.

Loss of SMS Calls

Several SMS calls were not delivered or delivered to the default PSAP many hours later.  Troubleshooting was done to debug these issues.  During the demonstration testing, further information was gathered to assist in the resolution. This is discussed in the demonstration testing section. 

6.1    King County, Washington, PSAP Site Testing

POC site testing took place on June 23–25, 2008, at the King County facility.  Total system testing was held on June 26, 2008, at the same facility.

Execution of test scripts for the total system went well.  However, network problems slowed the testing.  On one day of testing, calls could not be processed at the start of morning testing.  The development team was contacted, and the IP ACD software at the PSAP was shut down and restarted.  During IP User Agent (IPUA) device calls, the video call stream would lock up after several minutes of use.

There were no further problems with the testing.  Testing continued normally the rest of the time at the site and training was conducted successfully.

During the testing a different group of attendees participated each day.  Each group attending was given an overview of the USDOT POC and a PowerPoint description of how the systems operate before the attendees spent some hands-on time taking test calls.  The reaction by all groups attending was positive.

6.2    Helena, Montana, PSAP Site Testing

Execution of test scripts for the total system testing went well.  However, network problems slowed the testing, as well as the coordination of the testing with the five PSAP and two laboratory sites that were actively involved in the testing.  Testing was completed by the testing staff; no PSAP staff members were scheduled to participate on these test days.

The testing workstation used the software build of June 24, 2008, 11:25:27 a.m. 

No PSAP demonstrations were held at this site during the system testing.

6.3    St. Paul, Minnesota, PSAP Site Testing

On June 25, 2008, there was no connectivity for the system in St. Paul at the start of testing.  Support staff worked to establish connectivity.  The steps taken can be summarized as follows:

• Upon arrival at the St Paul Center on June 25, staff looked at the local router and found the T1 leased line was dead.  At about 8:30 a.m., they reported the bad circuit to the TMAC (AT&T dedicated State of Texas trouble desk.).  A ticket was opened with ASI (AT&T national backbone network group), which isolated it to a Qwest local loop in St. Paul.  A Qwest trouble ticket was opened and a local technician was dispatched.  (The router and network management system [NMS] logs indicated that the St Paul T1 had gone down at about 1:00 p.m. on Tuesday.)

• At about 1:00 p.m., the Qwest technician arrived and opened the line at the Central Office, performed a set of time domain reflectometer tests and found the incoming wire pair to be good.  He reconnected the line but the circuit still did not come up.  He then did a visual inspection of the connection in the local cross-connect box because he suspected a bad connection.  After reconnecting the local block, the circuit came up and tested error free.  
• The link was reestablished at about 2:15 pm.

There were four visitors at the Ramsay County PSAP on June 25 from 1:30 to 3:00 p.m.  Three were from the Fargo North Dakota PSAP, and one was from the Metropolitan Emergency Services Board.  The visitors arrived just as total system testing commenced and viewed the testing as it progressed through the scripts.  They asked many next generation concept questions while the test team worked with the PSAP call taker on the system testing scripts.  They were pleased with what they saw of the system testing.

6.4    Rochester, New York, PSAP Site Testing

On 6/25/2008 test calls were transferred from the Rochester PSAP to Seattle, then to Montana, then to Indiana, and finally back to Rochester.  The notes that were added to the call on the cellular and IP UA device tests were not received.  On the wireline call, Seattle and Montana successfully added and passed notes to the other PSAPs.   

On June 26, 2008, at 1:00 p.m., approximately 30 invitees were given a short PowerPoint presentation on the USDOT NG9-1-1 POC project and the POC software overview, followed by a software demonstration (Total System Test).

After the presentation, the test team made two successful outbound calls.  The tester called a telephone on the system and a cellular telephone not on the system.

6.5    Indiana PSAP Site Network-to-Network Testing

Execution of the test scripts for the total system testing went well.  However, network problems slowed the testing, as well as the coordination of the testing with the five PSAP and two laboratory sites that were actively involved in the testing.  Testing was completed by the testing and PSAP staff.

No PSAP demonstrations were held at this site during the system testing.

7    Demonstration Testing

The demonstration tests were similar to the total system tests but limited in number.  Only seven technologies were demonstrated, and testing consisted of processing calls through the complete system.  During the testing, the PSAP staff was involved and performed many of the tester functions.

7.1    King County, Washington, PSAP Site

July 15, 2008
The demonstration kicked off with a short introduction by the King County Executive.  There were 58 people in attendance, including representatives from 3 local television stations and 1 newspaper.  Two members of the Washington Office of the Deaf and Hard of Hearing were also in attendance.  Press packets were distributed by King County.

The prepared slides were presented in two sections—the NG9-1-1 overview and the call flow of the demonstration calls.  All live presentations functioned correctly and were received well by the audience.  The three television stations recorded the event.

Several questions were raised and responded to after the presentation:

• When asked if this POC would be the standard for NG9-1-1, the test team stated that this was a proof of concept to illustrate what was possible.  Other organizations would be responsible for setting standards.
• There were inquiries about several NG9-1-1 features that were not part of the POC, including receipt of still pictures from cellular telephones.  The test team stated that these features were not currently included in the POC.  The team restated that this was a proof of concept and that these features would have to be studied and integrated into production systems of the future. 

The audience enjoyed the presentation.

After completion of the demonstration, the two representatives from the Office of the Deaf and Hard of Hearing had several specific questions about the video and texting applications as they pertained to NG9-1-1 and which organizations were involved in setting the standards.  The test team answered their questions through an interpreter for about an hour.

7.2    Helena, Montana, PSAP Site

July 11, 2008
There were 12 attendees at the demonstration, including the Lieutenant Governor of Montana who introduced the demonstration.  Also in attendance were staff from the Governor’s office, state and local public safety departments, and one television station.

The prepared presentation reviewed the NG9-1-1 project and described the call flow of the call types that were demonstrated.  Two testing team members presented the information and fielded questions from the audience.  A Helena PSAP staff member operated the call taker workstation during the presentation.

The local television reporter recorded the presentation and demonstration and held interviews with selected people.

7.3    St. Paul, Minnesota, PSAP Site

July 17, 2008
There were 22 people attending the presentation and demonstration representing the Governor’s office, U.S. House of Representatives, State Senate, State House, State Patrol, Minnesota Department of Public Safety, Metropolitan Emergency Services Board, Ramsey County Board, and Ramsey County PSAP.

The prepared presentation reviewed the NG9-1-1 project, and described the call flow of the call types that were demonstrated.  Two testing team members presented the information and fielded questions from the audience.  A Ramsey County staff member operated the call taker workstation during the presentation.

The local CBS TV affiliate sent a reporter and cameraman who recorded the presentation and demonstration and held interviews with selected people.  A report was to be presented on a local newscast in the next couple of weeks.

7.4    Rochester, New York, PSAP Site

July 8, 2008
Prior to the demonstration, the test team ran the system and successfully tested all devices, including several video calls.  At approximately 11:00 a.m., the PSAP had a generator malfunction that cut off all power to the room where the demonstration was to be held.  Power was restored, and the POC workstations were brought back on line.

At 1:00 p.m., the POC demonstration was held, with approximately 30 attendees and representatives from the television and print media.

After the NG9-1-1 PowerPoint presentation, tests were conducted on all the communication devices.  Successful calls were made with a wireline telephone.  The cellular telephone call, which included location data updates that simulated a mobile cellular call, was successfully tested.  A telematics call was successfully completed.  The IP UA device delivered voice communication but video and text calls were not delivered.  There seemed to be a problem with the network.

Many of the attendees expressed interest in the next step after the USDOT POC.

7.5    Indiana PSAP Site

July 10, 2008
The demonstration took place in Warsaw, Indiana (Kosciusko County).  More than 22 people attended, including Indiana governmental and public safety personnel, as well as several press representatives.  The latter included representatives from television channels 15, 21, and 33 (all major networks) from Fort Wayne, and the Fort Wayne and Warsaw newspapers. 

Major governmental attendees included the Chairman of the Indiana Utility Regulatory Commission (IURC), several other IURC leaders, a State Senator, the Indiana Chief Deputy Treasurer, and the Indiana State 9-1-1 Coordinator.  A number of County 9-1-1 directors, police chiefs, and sheriffs attended.

After the overview of the USDOT NG9-1-1 project, all demonstration use cases were successfully performed for the audience, including text and video communication.  Because Indiana has implemented a statewide IP network (currently supporting cellular E9-1-1 service), many of the attendees were aware of IP networking in general and, more specifically, its role as the base for future NG9-1-1.  There were few questions from the audience. 

Several project team members were interviewed by television and press representatives, resulting in at least a 2-minute story on TV 15 Fort Wayne, and newspaper coverage.  After the demonstration, several attendees offered opinions that the demonstration went smoothly and was effective in information delivery. 

7.6    Texas A&M Laboratory Site

July 23, 2008
The demonstration was performed at Texas A&M laboratory site in College Station, Texas.  There were approximately 25 attendees, including a member of the state legislature, the Texas 9-1-1 Alliance Director, several County 9-1-1 directors, representatives from the project team agencies, and members of the press. 

All calling use case examples were successful, with the exception of SMS text test.  Involvement of the Texas A&M Internet2 Technology Evaluation Center in the USDOT NG9-1-1 project was emphasized, as well as related IP emergency communications development in Texas.  

Project team members were interviewed by television and press representatives.
 

8    Lessons Learned and Future Research

8.1    Introduction

The USDOT NG9-1-1 initiative was one of the first federally funded studies that attempted to comprehensively define and document a future vision for 9-1-1 systems.  Its goal was to provide a framework for future 9-1-1 systems that would allow any person access to emergency services from any device, anytime, anywhere.  The NG9-1-1 initiative followed the complete system development lifecycle from CONOPS, to system requirements, to implementation.  While the NG9-1-1 initiative successfully answered many pressing questions and proved out many futuristic concepts, as with any project, its scope was limited by time and money.  This section documents many of the issues and lessons learned from the POC and hopefully provides a starting point for future NG9-1-1 efforts.

Before actually discussing the numerous lessons learned from this project, it is important to highlight that one of the most significant findings of the POC was how diverse and complex the 9-1-1 community is and how much work there is to still be done.  The USDOT NG9-1-1 project team worked with a variety of organizations, including emergency service personnel (i.e., dispatchers, responders, etc.), industry vendors, and standards development organizations (SDO).  All stakeholders shared a common bond stemming from the importance of their mission and their desire to help people in their time of need.  However, hampering this mission is the community’s overall lack of effective ways to communicate and collaborate.  The USDOT initiative made significant headway by actively seeking the input of all of these groups to the project, sharing its efforts on an Internet website, and attending numerous meetings and conferences to conduct updates on the project.

It is imperative that this concept of information sharing continue to flourish and grow in the 9-1-1 community.  There is definitely a need for a common location where stakeholders can collaborate and share information from past, present, and future NG9-1-1 efforts.  Whether it be a repository, wiki, or combination there of, all this information should continue to be open and available to all stakeholders.  Providing a platform for communication will guarantee an increase in information sharing, leading to decreased cost and increased quality and interoperability of 9-1-1 systems.  It will provide a place of recognition and scrutiny for all NG9-1-1 efforts and ensure that the 9-1-1 stakeholder community does not repeat past difficulties.  At the conclusion of the NG9-1-1 Initiative, all results transition to the E9-1-1 Implementation Coordination Office (National 9-1-1 Office), which is charged by Congress with multiple responsibilities, including that of an information clearinghouse.  The promotion of USDOT efforts toward NG 9-1-1 will continue through the National 9-1-1 Office.

8.2    Operational Issues

Over the course of the POC the following operational issues were raised or discovered by the various NG9-1-1 stakeholders and project team.  For a successful transition to NG9-1-1 technology and systems all these issues will need to be addressed.

Regardless of the safeguards put in place, no system will operate 100-percent reliable all of the time.  However, given the time-sensitive nature and importance of 9-1-1 services, there must be processes and procedures put into place when a system operates unexpectedly.  As a call traverses an NG9-1-1 system, it must interact with an increased number of system components.  These multiple components provide advanced functionality but also increase the risk of an abandoned, lost, or dropped call.  To mitigate this risk, further research must be done regarding ways of logging calls as they traverse the system and informing call takers when unexpected behavior occurs within the system.

 

SMS has proliferated as a preferred method of mobile communication.  Given NG9-1-1’s mission and objective of “any device anywhere anytime,” it is desirable to provide call takers with the ability to receive SMS emergency calls.  However, SMS’ unreliable and connectionless properties raise new issues when considering the process by which call takers handle SMS emergency calls.

• Which call taker handles the call? 
• What type of lexicon is used?
• Does SMS tie up call taker resources like a traditional call? 
• Can a call taker handle multiple SMS calls at one time? 
• When is call taker finished with an SMS emergency call? 
• Can SMS become a reliable service through the use of acknowledgements? 

These, as well as a variety of other issues, must be vetted in the community before this technology can be successfully handled in the PSAPs of the future.

The NG9-1-1 Concept of Operations and Systems Requirements documents identified a variety of functionality deemed “Business Rules.”  Based on a review of the requirements documents, there are two types of Business Rule.  A policy-based business rule is a decision the NG9-1-1 system makes when handling or processing a call.  This, in turn, typically affects how a call is routed or queued within the system.  A second type of business rule refers to the configuration of software and/or components within the NG9-1-1 system that affects how a call taker interacts with a call.  These rules typically relate to the configuration properties of the call taker HMI and accessibility of data for the call taker.  While this functionality can all fall under the umbrella of NG9-1-1 System Configuration, work must be done to detail these concepts and further define how the system is configured and how it routes calls.  For the POC, the ability to override where a call terminates and the ability to route calls based on their call type were demonstrated.  In addition, the call taker HMI had a preferences and configuration screen.  More exhaustive work is needed to effectively define NG9-1-1 business rules and their associated functionality.

The rapid integration of cutting-edge technology is one of NG9-1-1’s major benefits.  The automotive industry is increasingly equipping their vehicles with more sophisticated systems and sensors.  Given the number of trauma-related fatalities associated with the Nation’s highways, it is imperative that PSAP call takers be able to respond to automotive crashes in a timely and effective fashion.  The NG9-1-1 POC demonstrated the ability to associate telematics data with an emergency call.  However, more detailed research should be done in the field of telematics integration.  The ability to determine the criticality of an accident while the call is in transit, route the call accordingly, adjust the priority level of the call if it is placed in queue, and automatically conference in EMS staff are just a few of the capabilities that where identified but not expanded upon during the POC efforts.

 

In NG9-1-1, additional data are associated with an emergency call that can enable new services for the caller and call taker.  Occasionally, callers required specialized services to enable effective communication and rapid response time.  A potential new feature of NG9-1-1 systems is the ability to automatically bring third parties into a call.  This is vital for individuals in the deaf or hard of hearing community, those who require foreign language interpreters, or during times when a specialist is needed (i.e., medical, structural, psychological, managerial, etc.).

 

NG9-1-1 provides the ability to accept emergency calls from both human and automated sensor systems.  This can include medical devices, home alarms, building and bank security systems, weather sensors, etc.  This raises a variety of operational issues because PSAPs are now able to accept an enormous amount of new data and call types.  How these calls are handled and routed in the system, the data that are associated with these calls, and the responsible agency or authority must all be determined.  The NG9-1-1 POC only scratched the surface in defining this use case.  Further evaluation, demonstration, and implementation are necessary. 

The current 9-1-1 system has more than 6,600 PSAPs geographically spread across the country.  For the most part, operations and authoritative decision making occurs at a local or county level thus hampering interoperability and coordination.  This grassroots operational model is not effective and does not utilize the functionality of an NG9-1-1 environment.  Fundamentally, built into the NG9-1-1 architecture and technology is a hierarchical governance model.  Using standards-based technology, local PSAPs will be able to interoperate with surrounding counties if they coordinate efforts.  Those counties will be able to interoperate statewide, and all states will be able to interoperate at a national level, assuming the necessary coordination  Accomplishing this increased coordination, given the divergent 9-1-1 funding models, will be difficult; however, given the flexibility of the technology, it is still possible.  The POC demonstrated a two-stage (Regional and Local) call processing model.  Transition issues must be successfully addressed, to be incorporated into the realistic 9-1-1 environment today.  The responsibility for infrastructure (national, regional, and local) and call handling and business rule policy at each level must be further defined.

One of the key capabilities of NG9-1-1, as defined in the CONOPS, is the ability to acquire supportive and supplemental data.  Supportive data are data that are acquired initially during call generation, or while the call is in transit, that provide critical information to assist the NG9-1-1 system with call processing and call handling.  Supplemental data are data that are acquired once a call is terminated at a PSAP and can assist the call taker in making more intelligent decisions regarding the nature of the call or how to respond to the situation.  Interfacing with external systems, such as medical records systems or telecommunications provider customer record systems, will require standard protocols, interfaces, and datasets.  Further work and cross-agency and industry surveys should be conducted to determine the standardization efforts that already exist.

An NG9-1-1 system supports a number of different call origination technologies.  Each technology will have its own media and data associated with the call.  This information can be a combination of voice, video, and/or data.  New call taker software must be designed with a flexible architecture and be standards based (IP, SIP, and XML) in order to ensure interoperability and ease of upgrade as new data sets and technologies become available.  Furthermore, interfaces for call takers must be user friendly and intuitive, creating uniform yet customizable views to cater to call taker preferences.  Given a network-based approach, a call taker should be able to maintain a profile enabling a common operating picture (COP) regardless of where or how he or she accesses the system.  This will create a much more useable framework for call taker software and provide “virtual PSAP” capabilities independent of geographic and physical location.  Further research must continue to optimize the way call takers interact and configure the NG9-1-1 system.

Transitioning to NG9-1-1 will not occur overnight or be as easy as flicking a switch.  More realistically, 9-1-1 systems will move to NG9-1-1 capabilities slowly based on available funding and priorities decided by state and local authorities.  However, it is imperative that as existing legacy systems transition, there is increased local, state, and federal coordination to achieve full interoperability among 9-1-1 systems.  One proposed method to ensure interoperability among systems is through accreditation.  The POC did not address this issue; however, additional time, resources, and stakeholder input would be necessary in defining the feasibility of an accreditation system .

The POC dealt exclusively with citizen-to-authority and authority-to-authority communication.  However, authority-to-citizen communication is just as important.  The ability to quickly warn citizens of natural disasters, terrorist attacks or other state of emergencies through public communication systems must be addressed, especially as emergency services move to IP-based systems.  Getting NG9-1-1 stakeholders involved with other agencies’ efforts, including the Department of Homeland Security’s Common Alerting Protocol (CAP) and the Federal Emergency Management Agency’s Integrated Public Alert and Warning System (IPAWS) ensures standardization, interoperability, and acceptance of other federal authority-to-citizen communication efforts.

Emergency response service requires a tight interaction among PSAPs, dispatch services, and first responders such as Police, Fire, and EMS.  While calls typically come into a PSAP for initial screening, once the nature of the call is determined, the caller is usually transferred or conferenced in with various dispatch services.  The NG9-1-1 POC identified the need to transfer to these services but did not demonstrate a method for effective handoff to these services.  This is especially important given all the new data that NG9-1-1 make available with an emergency call. 

In addition, PSAP call takers need a method of looking up the appropriate dispatch service.  Although PSAP call takers are typically familiar with services in their local area, sometimes they require services from distant counties or even other states.  EPAD is a federally sponsored effort and the desired future source for looking up emergency services for a given area.  It would be beneficial if the NG9-1-1 POC demonstrated its ability to interface with this directory as its method of determining the optimal dispatch service to contact.

8.3    Technical Issues

Over the course of the POC the following technical issues were raised or discovered by the various NG9-1-1 stakeholders and project team.  For a successful transition to NG9-1-1 technology and systems all these issues will need to be addressed.

Product selection will be key to effective NG9-1-1 deployments.  The POC demonstrated that technologies exist today that can support the requirements of an NG9-1-1 system.  However, the POC also demonstrated that the industry is still in its infancy.  There will be numerous integration challenges for NG9-1-1 system integrators moving forward.  For the POC, one of the largest design flaws involved limitations related to the selection of an open source, free conference server.  Because of product constraints, the conference server limited the number of connections that could be made to a PSAP and limited video conferencing to two parties (not ideal when demonstrating handling of a deaf caller who used of a sign language interpreter).  The NG9-1-1 vendor community is evolving and further efforts need to be made to drive this market, meet its needs and understand priorities of the key players in this industry.

As 9-1-1 systems are moved to IP-based technology more control is returned to the PSAP.  No longer do the telecommunications providers determine routing and call handling.  Given the importance of the 9-1-1 mission and the user community it serves, the NG9-1-1 system must be a managed IP network with guaranteed reliable and robust service.  Network management was not a key focus of the NG9-1-1 POC.  However, standard best practices and advanced monitoring capabilities must be enabled in production NG9-1-1 systems to ensure proper operation of the network and its associated system components.

Although the NG9-1-1 system is standards based, it will still require extensive testing to ensure optimal network and system performance.  The POC found that testing NG9-1-1 systems could be done at a basic level using commercial off-the-shelf test tools.  However, to more accurately characterize and simulate NG9-1-1 traffic requires extensions to currently available test tools, network monitoring applications, traffic generators, and packet sniffers.  The POC discovered test tools such as Wireshark and SIPStone are taking early steps to provide NG9-1-1 network and infrastructure testing capabilities.  However, as NG9-1-1 systems and standards mature the testing tools will have to continue to keep pace with the technologies they are testing.  These tools will prove invaluable to future NG9-1-1 interoperability, accreditation, and benchmarking work.

NG9-1-1’s basis on IP networking standards provides many advantages; however, it also exposes it to all of the associated vulnerabilities.  Security will play a key role in NG9-1-1 and will have to protect the system against some of the same challenges and malicious acts found on the Internet today.  The NG9-1-1 system will have to provide security policy for the network infrastructure and resources, as well as define access, authentication, and authorization for the users of the system.  The critical nature of the services offered and privacy concerns of the data make this network attractive for misuse.  The network must be highly controlled to ensure service yet flexible enough to provide open access.  Security will be a huge challenge that experts are just now beginning to address.  IP networks are not a new technology.  If the Internet can exist today and remain available to the public, the NG9-1-1 system should be able to leverage some of its best practices and diverse vendor community to accomplish its mission.

The NG9-1-1 system and emergency services it provides save lives on a day-to-day basis.  There is no downtime for emergency service personnel, and thus there must be no downtime in the network or system.  The only way to provide the level of service this system demands is to build redundancy and reliability into the network.  This includes both redundancy in the network and in its associated system components.  PSAP operators and authorities must build stringent quality and level of service agreements with their operational contactors and system integrators.  NG9-1-1 systems must have 100-percent uptime.  In addition, in the case of catastrophe, when system resources spike or are unavailable altogether, PSAPs must have appropriate overflow mechanisms built into their infrastructure to offload some of their burden yet still provide equivalent levels of service.  Given the fact that the POC was not handling “real” 9-1-1 calls, the design did not have to take into account these stringent yet necessary requirements.  Operational PSAPs must stress the importance of redundancy, reliability, and overflow, and ensure these services are built into their systems.

A variety of voice and video CODECs exist in the market place today.  Typically, they make tradeoffs among bandwidth, quality, and configurability.  For the POC, an arbitrary CODEC was chosen for voice and video based on ease of procurement and integration into the POC software.  Through the use of SIP, multiple CODECs can be used and supported.  Native to the SIP is the ability to negotiate the media format and associated CODEC while the session is being set up.  However, PSAPs should not have to support all the CODECs that currently exist.  It is recommended that the emergency response community be more proactive and conduct research to determined a list of acceptable CODECs that provide the necessary quality and performance for emergency communication.  Only by driving standards and specifications will the emergency service community ensure interoperability among all the communication devices available today.

As a call propagates through the NG9-1-1 system, it must interact with an increased number of system components.  Gone are the days of a single selective router that decides how the call should be routed.  NG9-1-1 adds new components with advanced capabilities like location determination, injections of supportive data, and policy-based call routing.  Unfortunately, these advance features also add overhead and can increase the propagation time of the call.  In addition, they increase the risk of call failure and place external dependencies on the system.  Typically, in today’s systems, it takes 9-1-1 calls an average of 7–12 seconds to propagate end-to-end.  An NG9-1-1 system should strive to provide the same level of quality.  To ensure this quality, stringent timing constraints must be place on NG9-1-1 systems.  A system must be able to respond within a predetermined amount of time or the NG9-1-1 system should have a mitigation strategy if one of its system dependencies is unresponsive or underperforming against its performance metrics.  The POC did not address this issue because it was not handling “live” emergency calls.  However, determining these metrics and enforcing them on an operational NG9-1-1 system is vital to ensure all emergency calls are handled with the appropriate level of service.

The NG9-1-1 system introduces an innovative paradigm for call routing.  The NG9-1-1 system routes calls based on a variety of information, one of the most important elements being location.  Unlike traditional 9-1-1 systems where the call is statically mapped to a PSAP and the location of the call is acquired after the call has terminated at the PSAP, NG9-1-1 differs in that it requires location up front to route the call appropriately through the system.  Therefore, each call origination technology (legacy wireline, cellular wireless, telematics, SMS, IPUA, Sensor, etc.) must provide mechanisms for injecting or acquiring an emergency callers location initially as the call is placed.  This will require an overhaul and upgrade of the location acquisition technology that currently exist today such as Automatic Location Identification (ALI), Mobile Positioning Center (MPCs) and Voice over IP Positioning Center (VPCs).  In addition, the POC showed that some technologies, such as SMS, currently have no means of location acquisition.  To effectively integrate these technologies into NG9-1-1, they must provide location and conform to the defined location data standards.  Standards bodies such as the Emergency Context Resolution with Internet Technologies (ECRIT) are addressing some of these issues.  Given the timelines of the POC and the inaccessibility to production location acquisition systems, many of the POC location acquisitions systems (ALI, MPC, VPC, LIS, SMS Positioning) were mocked up and simulated.  Further work needs to be done to characterize these systems and demonstrate effective, standards-based methods for accessing these systems.

 

Many emerging communication devices provide advance methods of communication and support many different media types, including voice, video, texting, and data.  Although the POC touched on many of these media types, it was not able to successfully demonstrate an effective method for PSAPs to consume imagery or video clips with a call.  This is largely due to a lack of standardization within the industry on how multimedia content is shared and distributed using mobile devices.  One technology that is offering this capability in a standard format is Multimedia Messaging Service (MMS).  As this technology matures and telecommunications vendors begin to support this standard, further research should be done to ensure that MMS can meet the requirements of the emergency service community and ease the integration challenge of supporting multimedia in a PSAP environment.

Geospatial capabilities are important as PSAPs transition to NG9-1-1 systems.  Moving to an IP-based network will allow advanced geospatial capabilities for the PSAP community and ease the integration with existing geospatial platforms and systems with NG9-1-1 software and infrastructure.  Some groups, such as the GEOPRIV Working Group, are already addressing some of the issues associated with geolocation in an emergency service context.  The POC attempted to conform to industry standard geospatial solutions.  It demonstrated the ability of NG9-1-1 systems to integrate with both open source solutions such as GoogleMaps, as well as proprietary, high-resolution solutions such as Pictometry.  Further information and an overview of geospatial solutions for emergency services can be found at geopriv.googlepages.com.

9    Compliance Matrix

Appendix A—Acronyms

Appendix B—Glossary

Appendix C—Source References

The following documents are primary sources of information used in this document.

1. Next Generation 9-1-1 (NG9-1-1) System Initiative: Concept of Operations.  USDOT ITS JPO.  April 2007.  http://www.its.dot.gov/ng911/pdf/NG911ConOps_ April07.pdf—This formal document provides a user-oriented vision of NG9-1-1 in the context of an emergency services internetwork that can be understood by stakeholders with a broad range of operational and technical expertise.  It is intended to communicate the vision of this system to stakeholders so that they can be actively engaged in its development and deployment. 
2. Next Generation 9-1-1 (NG9-1-1) System Initiative: System Description and High-Level Requirements Document.  USDOT ITS JPO.  July 2007.  http://www.its.dot.gov/ ng911/pdf/NG911_HighLevel_Requirements2007.pdf—This formal document provides a high-level overview of NG9-1-1 system operations.
3. USDOT NG9-1-1 Architecture Analysis Report.  USDOT ITS JPO November 2007.  http://www.its.dot.gov/ng911/pubs/NG911_FINAL_ArchAnalysis_v1.htm—This formal document provides a high-level architecture analysis of the NG9-1-1 system.
4. USDOT_NG911_POC_RequirementTableTiered_11Sept2007.xls.  Unpublished—This internal document contains the requirements developed in previous work.  This file contains the updated Tier 1 requirements.
5. NG9-1-1 Interim System Design Document.  USDOT ITS JPO.  February 2008.  Unpublished—This formal document details the interim NG9-1-1 system design.  This design was used in the preparation of the POC.
6. Proof of Concept Deployment Plan.  USDOT ITS JPO.  February 2008 http://www.its.dot.gov/ng911/pubs/NG911_proof_concept.htm—This formal document details the plan to deploy the proof-of-concept resources and to perform the POC.
7. Human Machine Interface Design Document.  USDOT ITS JPO.  January 2008.  Unpublished—This formal document details the human machine interface.  This design was used in preparation of the POC.
8. NG9-1-1 Proof of Concept Testing Plan.  USDOT ITS JPO.  June 2008.  http://www.its.dot.gov/ng911/pubs/ng911_poc_testplan_final.htm—This formal document describes the testing of the NG9-1-1 POC.
9. Data Acquisition and Analysis Plan.  USDOT ITS JPO.  May 2008.  http://www.its.dot.gov/ng911/pubs/data_acquisition.htm—This formal document describes the data collection plan and the plan to analyze the data collected.

Appendix D—POC Testing Results

[This appendix includes the specific details of each test conducted during the POC Testing.  Due to the large number of pages, Appendix D will be made available in electronic format only and as a separate document.]

Appendix E—POC Data Acquisition and Analysis Results

Overview
The Data Acquisition and Analysis task assisted the Next Generation 9-1-1 (NG9-1-1) Project Team and the U.S. Department of Transportation (USDOT) to evaluate the technical and operational success of the proof of concept (POC).  It identified gaps and current issues with the technologies used to implement the NG9-1-1 POC system.  The data acquired from the task will serve as benchmarks for future large-scale, production NG9-1-1 technology deployments and further assist in the refinement of the final NG9-1-1 transition plan.  The data will facilitate the USDOT, Standards Development Organizations (SDO), industry vendors, the Public Safety Answering Point (PSAP) operational community, and future independent evaluators in defining measures of interest for Internet Protocol (IP)-based emergency calling.  The subsequent subsections address the four main data analysis tasks of the NG9-1-1 POC: 

• Emergency Call Propagation and Timing—This subsection examines the need for calls to be routed in an efficient, time-sensitive manner through the NG9-1-1 system.
• Emergency Call Availability and Quality—Two subsections investigate the effect of NG9-1-1 Component and Interface availability and IP network performances on the quality of voice and video calls.
• Emergency Call Scalability—This subsection explores the NG9-1-1 system’s ability to scale on a need-driven basis with convergence of different media types (voice, video, and data).  It also looks at innovative NG9-1-1 call overflow mechanisms and their ability to improve overall system performance.

Each subsection gives an overview of the specific data analysis task, identifies the measures of interest that were collected and the tools used to collect that information, and then provides analysis and conclusions based on the data collected.  For further details about how the data analysis task was conducted, refer to the NG9-1-1 Data Acquisition and Analysis Plan.

Emergency Call Propagation and Timing

Description:  This activity evaluated the NG9-1-1 POC system’s ability to initiate, propagate, and terminate different types of emergency calls, including IP User Agent (IPUA), legacy wireline, telematics, and cellular.  Currently, within the United States, a wireline emergency telephone call traverses from a call originator to a PSAP call taker in an average of 7 to 12 seconds.  It is imperative that NG9-1-1 emergency calls are processed with at least the same level of performance as exists with current technology.

Measures of Interest: TAccess, TLIS, TNat_LoST, TNG9-1-1, TLoc_LoST, TSupp, TBusiness, TPSAP, TCall_Taker, TEnd-2-End

Data Collection Tools Used:  Network Time Protocol (NTP) Server, Call Origination/Session Initiation Protocol (SIP) Border Gateway/Emergency Services Routing Proxy (ESRP)/PSAP Automatic Call Distribution (ACD) Software Event Logs, Network Protocol Analyzer

Analysis/Conclusions/Points of Interest:

1. The median time for TEnd-2-End for all call types was less than or equal to 10 seconds (TEnd-2-End < = 10 sec).  Therefore, the NG9-1-1 POC environment maintained the same level of response time as today’s operational 9-1-1 system(s).
2. For the POC environment, most of the propagation time can be attributed to TPSAP (TPSAP  was approximately 3–5 seconds for most calls).  This is expected because a great deal of decision-making logic is built into the PSAP ACD.  The PSAP ACD must terminate the call and then analyze an emergency call’s call stream data to determine the “most appropriate” call taker to handle the call.  This is based on the availability and capabilities of the call takers.  This advanced decision-making process adds latency to the call.  Operational 9-1-1 systems would have to closely monitor this parameter and ensure those systems do not become overly complex.  There is a fine balance between providing advanced call routing features and physically getting a person in the loop to determine the nature of the emergency and the appropriate response action.  Therefore, this parameter should have a threshold to ensure that calls are handled in a timely fashion.
3. When larger amounts of time (> 10 seconds) were taken for the call to propagate through the system, TAccess was large and the most significant factor in the TEnd-2-End calculation.
4. Because of an external dependence on other systems outside the NG9-1-1 POC, TSupp was not measured (TSupp = 0 seconds).  For operational NG9-1-1 systems, a detailed understanding of the external system must be known in advance.  Dependence on external systems for routing and processing call can create bottlenecks in the system and severely affect the performance (response time) of the system.
5. Because the Business Rules Database resided on the same server as the ESRP, TBusiness was negligible and not measured after a few initial trials (TBusiness was assumed to equal 0 seconds).  If the Business Rules Database was maintained as a separate system, this could add network latency to the routing of an emergency call.  Operational 9-1-1 systems would have to closely monitor this parameter and ensure their systems did not become overly complex.  There is a fine balance between providing advanced call routing features and physically getting a person in the loop to determine the nature of the emergency and the appropriate response action.  Therefore, this parameter should have a threshold to ensure that calls are handled in a timely fashion.

Telematics

Cellular

Emergency Call Availability

Description:  This activity evaluated the NG9-1-1 component and interface availability.  Over the course of the POC, a Network Management System was deployed to track the uptime of system components and interfaces.  Proper operation of NG9-1-1 system components and interfaces is vital to an effective emergency response service.  Operationally, an NG9-1-1 system must provide near 100-percent uptime.  The POC is research oriented; therefore, no efforts were made to design a fully redundant or 100-percent available system.

Measures of Interest: Component Availability, Interface Availability

Data Collection Tools Used:  Network Management Software (HP Openview)

Analysis/Conclusions/Points of Interest:

1. Components (servers, laptops, etc.) in the POC did not provide the level of availability expected for an emergency response system.  Given the research-oriented nature of the POC, components were constantly being updated with software, restarted, and physically moved within the laboratories and at the PSAPs.  Because there was no redundancy (i.e., only one workstation was used per PSAP), an unstable environment existed where, in effect, the PSAP was considered “down/unreachable” during maintenance and/or upgrades.  For an operational system, much more care would have to be given to design the system to provide a more robust and reliable operating environment that could handle emergency calls 24/7/365.  It is likely that a PSAP or governing authority would have to enforce a service contract requiring the desired level of availability (e.g., Five 9s) from the operations and maintenance (O&M) systems contractor.
2. During the POC, interface availability (i.e., network connectivity) was fairly robust.  Network connections between the three labs and five PSAPs were stable and more reliable than the components that resided on the actual network.  Leasing the connections from major telecommunications providers minimized the risk of losing network connectivity.  However, as emergency response systems move to IP-based infrastructure, the results of the POC should not lead to underemphasizing the importance of a good Network Management System to ensure reliability and conformance to service level agreements.  Although network connectivity and bandwidth are viewed as a commodity in today’s world, stringent IT redundancy, resiliency, and performance best practices should still be used to ensure 100-percent operational emergency response systems.

Data Collection Results:

Component Availability

Interface Availability

Emergency Call Network Quality

Description:  This activity evaluated the NG9-1-1 POC’s IP network quality.  It is imperative that an optimized network exist to transport voice and video call streams, especially in an emergency response context.  Voice and video traffic requires time-sensitive delivery of IP packets.  Lost IP packets quickly degrade the quality of the voice and video streams, and retransmission is typically not an option.  End points were configured to send a variety of voice, video, and data traffic across the network.  Based on the traffic scenarios, network performance metrics were calculated.

Measures of Interest: Throughput, Packet Loss Latency, Jitter

Data Collection Tools Used:  Network Traffic Generator (appCritical, SimpleNets Simple Network Tester)

Analysis/Conclusions/Points of Interest:

1. The POC provided enough bandwidth to support the test cases and POC demonstrations.  Lease lines were obtained to connect the three labs and five PSAPs.  Both the commodity Internet and Internet2 networks were leveraged to use as transport for the POC emergency call traffic.  Based on the testing, all leased lines supported at least 3 megabits per second (Mbps) of bandwidth.  Those lines that leverage Internet2 for connectivity provided throughput of as much as 50 Mbps.  Based on the selected video and voice CODEC, a single call was no greater than 150 kilobits per second (kbps) for video and 64 kbps for voice.  Therefore, given the limited number calls placed during the POC, the IP network had little trouble supporting the traffic generated by the testing.  In an operational network, more care would need to be taken to ensure that the network could provide adequate bandwidth to the PSAP even during peak loads and disaster situations.  Best practices, such as dynamic network scalability, PSAP overflow, and call rerouting mechanisms, should be designed for and implemented in an operational network.
2. Based on current best practices, packet loss should be < 1 percent.  The POC network met this constraint during all testing and POC demonstrations.  Packet loss is typically an indication of an overly congested network.  This scenario did not occur during operation of the POC.  Operational 9-1-1 networks would need to monitor this metric using network management software and work with their network provider to ensure appropriate measures were put in place to handle network congestion.
3. Based on current best practices, latency should be < 100 milliseconds (ms) to support voice and video.  The POC network met this constraint during all testing and POC demonstrations.  Given the use of leased lines and small size of the network, there were only a limited number of hops that packets needed to take to traverse the system end to end.  Operational 9-1-1 networks would need to work closely with their access and transport network providers to ensure latency is minimized.
4. Jitter should be < 50 ms for voice and video.  The POC network met this constraint during all testing and POC demonstrations.

Data Collection Results:

appCritical Results:

SimpleNet’s Simple Network Tester Results:

Emergency Call Scalability

Description:  This activity evaluated the NG9-1-1 POC system’s ability to handle large call loads (call scalability).  Designing scalability into an emergency response network is imperative for handling peak load and disaster situations.  In an emergency response network, both the network and system components should be able to handle additional call load and dynamically scale according to need.

Measures of Interest: Call Rate, Maximum Concurrent Calls, Call Success, Call Failure

Data Collection Tools Used:  SIP Call Simulator (SIPStone SipP)

Analysis/Conclusions/Points of Interest:

1. The call failure rate is directly proportional to the number of concurrent calls placed in the system.  For the testing, the team injected a number of different types of calls into the system that simulated a combination of data, voice, and video emergency calls.  For the POC environment, it was determined that as the number of concurrent calls increased, the probability of call failure also increased.  This can be attributed to the bandwidth used by a call, the lack of quality of service (QoS) policy within the network, and the capabilities of the hardware/software used in the POC environment.  

• Limitations in bandwidth.  Because the system leveraged leased line circuits at both the ingress and egress of the NG9-1-1 network, as the number of concurrent calls increased, the system began to saturate with voice and video packets.  With no overflow or scalability capabilities, the system eventually could no longer support additional data packets injected in the system.  The router queues began to fill, and eventually, packets began to drop.  As new calls were injected into the system, their signaling data (setup and tear-down information) could no longer pass through the system.  At this point, calls began to fail, thus increasing the failure rate.  For operation systems, link budget analyses should be conducted prior to implementation to determine the desired number of concurrent calls that should be supported.  Important factors include the theoretical bandwidth of the network links, the audio and video CODECs supported, and whether the signaling protocol operates in a centralized or peer-to-peer mode.
• Lack of QoS Policy.  Given scheduling and product constraints, a QoS policy was not implemented for the NG9-1-1 POC.  A solid QoS policy would have given different priorities to certain IP packets.  Therefore, despite the congestion in the network, signaling information could have passed through the network with priority over the voice and video traffic.  This would have allowed the system to support more voice and video calls and alleviated some of the network bottlenecks that were created.
• Limitation in Call Processing Hardware/Software.  The POC environment was also limited as a result of some of the hardware and software selected.  To support the transfer and conferencing of calls, a cheaper conference server was selected that supported only a maximum of 15 concurrent call streams.  In addition, to expedite development of the POC, a design decision was made to pass all calls through the conference server even if only two parties (caller and call taker) would be involved in the call.  This design decision, in conjunction with the conference server product selection, placed a hard limit of 14 concurrent calls on the system.  In future work, it would be interesting to see how the system would scale if this constraint did not exist.  This constraint emphasizes the importance of a thorough vendor analysis before the selection of any equipment in a production or operational system.

Data Collection Results:

Rochester, New York, PSAP

Ft. Wayne, Indiana PSAP:

Seattle, Washington PSAP:

St. Paul, Minnesota PSAP:

Helena, Montana PSAP: