Conclusions

This section presents conclusions based on the findings from the evaluation of system performance and feedback from users and FOT developers. The objectives and hypotheses originally proposed in the Evaluation Plan are revisited in light of the data and findings.

5.1 Evaluation Hypotheses and Findings

The evaluation hypotheses associated with the FOT voice and data routing system performance and user acceptance were identified and discussed in the evaluation approach sections. Table 5-1 presents each of these hypotheses and summarizes the findings relevant to each of them.

Objectives	Hypotheses	Findings
Evaluation of Voice Routing System Performance
Testing FOT Voice Routing System Reliability	The OnStar Customer Service Representative can establish a 3-way call to the 911 system which goes to the correct PSAP considering the call location.	A total of 147 acceptance test calls were made from a portable unit (emulating OnStar system). Those calls were answered by OnStar Emergency Advisors and successfully connected with PSAPs corresponding to the vehicle location using FOT voice routing solution. Those test calls were delivered along with vehicle location data (LAT/LON), call back number (to OnStar unit), and ACN/AACN data. Based on a staged statistical sampling design, the FOT voice routing system passed acceptance testing with 94% confidence that the true system failure rate is not more than four percent. The acceptance test was conducted from August 10, 2005 through August 19, 2005 involving 22 PSAPs, OnStar Emergency Advisors, and MnDOT field test engineers who initiated test calls from pre-determined locations. The test procedures and acceptance test were developed and facilitated by Battelle. The test results were audited and validated by Battelle based on the statistical sampling design previously developed in the evaluation plan.
Testing FOT Voice Routing System Reliability	The 911 system at the responding PSAP will receive accurate transmission of LAT/LON, OnStar unit call back number, and additional ACN/AACN data (if available).
Evaluation of Data Routing System Performance
Testing Data Routing System Reliability	All OnStar incident data designed to be sent to the MnDOT SOAP server are sent reliably and accurately, and in timely fashion.	Over 41 weeks for which data were available, OnStar successfully transmitted 1,247 crash records to the MnDOT SOAP server, including 1093 SOS (emergency button pushing), 137 CAN, and 17 AACN crash records. Analyses of transmission logs indicated a small percent (3.9) of records failed in the transmission. However, those failures were attributable to known technical problems that arose during the FOT period. The FOT data routing solution has demonstrated a high reliability of 96.1%. The mean latency for data transmission from OnStar OCC to MnDOT SOAP server is 0.9 seconds.
Evaluation of User Acceptance
Assess User Acceptance (OnStar)	OnStar Emergency Advisors perceive the process and application to benefit TSP call processing.	Emergency Advisors indicated that it was easy to see that this kind of call routing and delivery would be preferable.
	OnStar Management Representatives perceive the process and application to generally benefit corporate service goals.	While OnStar’s history in processing customer 9-1-1 calls has been perceived to be positive, OnStar management indicated this was a much preferable method to handle such calls.
	OnStar prefers this solution to current practice.	While the current process of handling 9-1-1 destined calls works, the FOT solution would provide a much more effective way of accomplishing the same task. The current access to PSAP administrative line is still needed to facilitate the events of a non-emergency nature.
	OnStar believes this solution facilitates interaction with PSAPs. Automatic communication of location (and, thus, routing of the call) is inherently less error-prone.	This type of solution automates the routing and delivery of customer 9-1-1 calls in a native way. The ability to transmit call back number, call location, and ACN/AACN data saves time and reduces human errors.
	The total time of call processing is perceptibly reduced.	The automation of the call delivery process inherently saves time, particularly in situations involving manual miss-dial.
Assess User Acceptance (PSAP)	PSAP call-takers perceive the FOT application to be beneficial.	While historical experiences with OnStar-based 9-1-1 calls have been positive, most call-takers did perceive the trial and the solution involved to be beneficial.
	PSAP call-takers prefer this solution to current practice.	Conceptually, most call-takers did indicate a preference for this type of solution (delivering such calls in a native way, similar to traditional 9-1-1 calls). All the PSAPs interviewed currently receive telematics-based calls on so-called “administrative lines,” or on non-native 9-1-1 trunks dedicated for such purposes. In all instances, while calls on these lines are processed identically to native 9-1-1 calls, they do receive lower call-handling priority than calls on 9-1-1 trunks (though that only becomes an operational issue when 9-1-1 call-takers are busy on other calls). Since such lines generally do not automatically provide ANI and ALI data, the PSAPs involved attempt to capture the “caller-ID” of the calling party, and then poll their ALI database for location.
	PSAP call-takers believe this solution facilitates interaction with the OnStar Call Center.	To the extent that such call delivery minimizes routing problems, and time expended to communicate location, interaction with the OnStar Call Center is better supported.
	Automatic communication of location (and, thus, routing of the call) is inherently less error-prone	Automating the process avoids problems traditionally associated with manually communicating the location involved, and insures more accurate call routing and delivery. The transmitted LAT/LON can be plotted on PSAP’s GIS system in support of the call location identification.
	The total time of call processing is perceptibly reduced.	While the automation of this process does noticeably save time, the reliability of call routing, coupled with the native delivery of such calls is the primary benefit.
	PSAP Authority (cognizant 9-1-1 entity) perceives the process and application to be beneficial.	All interviewed PSAPs indicated a preference for native 9-1-1 call delivery for telematics-initiated calls, noting that the latter insures more accurate routing, the automatic delivery of call related data (like ANI, ALI and class of service) and priority processing.
	State 9-1-1 Point-of-Contact perceives the process and application to be beneficial.	The Minnesota State 9-1-1 Coordinator supported the conclusions of the PSAPs involved in the trial.
Assess User Acceptance (Medical Responders)	Medical responders (Mayo Clinic) perceive the ACN data to be beneficial.	Mayo Medical Transport, as both a secondary PSAP, and as a CARS data user, specifically cited the benefit of ACN incident data in responding to an emergency event.
	Medical responders (Mayo Clinic) are more informed with ACN data in response to the traffic-related incidents.	While the CARS data are limited, as is the means of access, the timely notification of an incident, along with descriptive data about the event, does facilitate emergency response. Mayo Clinic routinely accesses ACN/AACN data via CARS.
	The ACN data are provided in a timely manner via CARS.	While data delivery is timely, the information access on CARS is a manual process 32, and could be further enhanced or automated.
	The vehicle location representation on CARS is accurate and useful.	Mayo Clinic has found ACN/AACN data to be both accurate and useful.
Assess User Acceptance (State Traffic Operations)	State traffic operation users perceive the ACN data to be beneficial.	To the extent that such data facilitates incident management, and traffic operations, the data are very useful.
	Such users desire permanent deployment of the application.	Such users urged both deployment of the application, and its promotion beyond the geographic scope of the trial.
	The state traffic operation is more informed with ACN data in response to and management of traffic-related incidents.	The ACN/AACN provides a wealth of information regarding the accidents. Such information is supplemental to other traffic surveillance and incident management tools.
	The ACN data are provided in a timely manner via CARS.	The system, as it is currently working, does appear to provide ACN/AACN data in a timely way. The data routing evaluation results showed, in average, it takes less than a second for OnStar to transmit ACN/AACN data to MnDOT.
	The vehicle location representation on CARS is accurate and useful	Experiences indicate that location identification is both accurate and useful due to the provision of LAT/LON provided by the GPS-equipped OnStar vehicles.
Evaluation of Deployment Issues
Assess Expandability beyond MN	The project partners believe that the benefits of the FOT solution warrant its application beyond Minnesota.	All project team members recommended that the FOT solution and concept be further examined for real world deployment, within and beyond Minnesota.
Expandability (Multiple TSPs)	FOT public safety users believe that the benefits of the FOT solution warrant its application to other Telematic Service Providers.	The solution offers a generic, non-Telematics Service Provider specific approach to service customer 9-1-1 call delivery.
Consistency with NG-1-1	The FOT solution is consistent with standards and related work currently being conducted on NG9-1-1 migration by NENA.	The FOT solution represents a viable migratory step in that direction, though ultimately NG9-1-1 architecture may offer more effective solutions for such matters (for that part of the PSAP community that is able to migrate to such).

5.2 Conclusions

Based on the quantitative and qualitative components of the evaluation, this section presents the main conclusions from the evaluation of the MAYDAY/9-1-1 FOT.

Using a statistical acceptance sampling approach, the FOT solution passed the acceptance test with no observed failures. A sequential sampling approach was used to determine if the FOT solution passed acceptance. Ultimately, this approach resulted in high confidence (i.e., 94 percent) that the true system failure rate does not exceed a relatively low value (four percent). The true system failure rate may well be even lower (e.g., the observed failure rate was zero percent), but the sample size available for the test was only sufficient to make conclusions at a level of four percent.

The FOT voice routing solution relies on a wireless service provider to route long distance 9-1-1 calls originated from a TSP to a local MSC corresponding to the LAT/LON of the vehicle location. From there, the calls enter the 9-1-1 trunk using a modified E2 interface and emulate a wireless E9-1-1 call. This allows a phase 2 compliant PSAP to receive the calls as a native 9-1-1 call along with the LAT/LON of the vehicle location, and a call back number (to the telematics unit).

With the exception of WSP call routing, the final segment of the call delivery solution is consistent with the phase 2 E9-1-1 initiative. Feedback from the FOT team suggested that minor clarifications of ALI interface are needed due to the ambiguity in current E233 interface definition, in support of the call and data routing on the local 9-1-1 trunks.

The implementation of WSP call routing will require additional standards34 development and commitments from a WSP to implement the solution on all switches across the U.S. Given the competition from other standards development activities in the telecommunication community, the telematics-specific standards might not receive priority because of its small market share. Nevertheless, the FOT solutions are technically feasible and are consistent with wireless E-9-1-1 standards in the final call delivery stage.

It is noted, however, that WSP migration to Third and Fourth Generation wireless service may facilitate this process by providing more effective and flexible switching and routing platforms for such matters.

Over the 41 week period of the operational test, the FOT demonstrated it could reliably and quickly transmit OnStar crash data to a MnDOT SOAP server, from where it could be accessed by CARS and other third parties. The reliability was 96.1% and the few failures observed were traced to a known computer problem that would not be expected to occur in a production system. The average latency of data transmission was less than one second (i.e., 0.9 seconds).

The data routing evaluation reflects only a portion of the system’s reliability from an end user perspective since it does not include information for the links between the MnDOT SOAP server and other servers that ultimately make the information available to users (e.g., CARS). Nevertheless, it can be concluded that the first step in the data routing process does not introduce any serious lack of reliability or speed.

The PSAP user community generally found both the intent and nature of the trial to be beneficial to the effective delivery of a telematics-based emergency call that must be delivered to a Public Safety Answering Point. The routing of such calls to the correct PSAP is more accurate and reliable, and the time involved to process the calls is minimized by automatically identifying the location. The native delivery of calls in this manner also insures that the calls involved will receive the same priority as any other 9-1-1 calls.

While nearly all the PSAPs involved in the trial indicated positive historical experiences with OnStar calls, they also indicated that this type of delivery would enhance service and minimize confusion in terms of the description of location.

While the migration of the emergency response community to next generation IP-based network infrastructure may provide the ultimate solution to sharing voice and data conducive to emergency services, the data sharing mechanism utilized in the FOT does provide an effective and immediate way to take advantage of data generated during a telematics emergency event. Deploying new and advanced network infrastructure may take time, particularly in a ubiquitous way, this FOT offers an immediate step in that direction.

This solution can be deployed with minimum cost, and states should explore the opportunity represented here to take advantage of their highway and traffic information systems to benefit both incident management and emergency response.

Finally, FOT voice routing capability substantially enhances the quality of service OnStar provides to its customers. Having the service available in times of emergency is an important reason why many customers subscribe to the telematics service. Insuring more accurate and timely response to customer emergency requests improves the service involved. It also enhances the opportunity for a mutually beneficial relationship with the PSAP community, a matter of equal importance.

32 CARS does not alert users of the new events (e.g., OnStar accidents) in a particular geographic area. A user must proactively and constantly looks for new events of interest being populated.

33 E2 defines the interconnection specifications between 9-1-1 service providers.

34 NENA TIA TR45.2 9 [for CDMA] needs to be worked to standardize TSPECRS.

5.0 Conclusions

5.1 Evaluation Hypotheses and Findings

5.2 Conclusions