4. TEST RESULTS
The data discussed in section 3.1 was collected and analyzed according to the modified evaluation strategy described in section 3.4. This section presents the analysis results and a results summary regarding the system performance and system impact FOTs. Institutional and technical challenges were also assessed; however, because these are completely qualitative in nature, they are presented in section 5, Evaluation Findings. The lessons learned, which also provided important findings for this evaluation, are presented in section 6, Conclusions and Recommendations.
4.1 SYSTEM PERFORMANCE TEST RESULTS
Table 4 summarizes the system performance test results based on the discussion in section 3. Following the table is a discussion for each evaluation objective, along with the corresponding results in sections 4.1.1 through 4.1.3.
| Evaluation Objective | Hypothesis | Test Results |
|---|---|---|
| Objective #1: Document the system component performance. | The system meets functional specifications. | Achieved |
| The CAD and TMC systems will be able to link data on an incident. | Achieved | |
| Using the system improved incident response procedures. | To a significant extent, achieved through
prior projects. Project specific impact not measurable. |
|
| Objective #2: Automate the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors. | The system meets functional specifications. | Achieved. |
| The FOTs will decrease the reliance on manual methods for exchanging information. | Preliminary result - achieved. | |
| The FOTs will increase the extent and reliability of information exchanges. | Preliminary result - achieved. | |
| Objective #3: Extend the level of integration to include secondary responders such as utilities, towing and recovery, public works, and highway maintenance personnel. | Improved integration of secondary responders will reduce incident recovery time by getting required recovery personnel to the incident site as quickly as possible to begin recovery operations. | Secondary responders (ambulance, utilities, etc.) were not included in the project. |
4.1.1. Objective #1: Document System Performance
Following are the three hypotheses associated with the objective to document the system's performance:
- The system meets functional specifications.
- The CAD and TMC systems will be able to link data on an incident.
- Using the system improved incident response procedures.
The Evaluation Team relied on a combination of observations and interviews to determine whether or not the system met the functional specifications. Actually seeing the system work and finding out if the system met operator expectations were the best indicators to determine that the system successfully met system performance needs.
After the integrated system was implemented, a demonstration of the capability was held at the TMC with excellent results, as reported by the participants. A mock incident was sent from the UHP to the TMC, with the TMC having the ability to review each detail of the incident. The TMC staff was able to input updates to the incident information without needing to telephone UHP. The demonstration clearly illustrated that the technology worked properly and has excellent potential for improved sharing of incident details.
In interviews with UDOT deployment staff and operators from UDOT and their partner agencies, all described the system as meeting functional specifications. The consensus was that the system is "doing the job it was intended to do."
While observing the system in operation, information from partner agencies was displayed on the integrated system window within seconds from the time of input. For example, since UDOT operators have access to the VECC CAD Web site, incidents entered into the CAD system could be observed to display in the integrated system. The system also responded well when operators brought an outside incident (one from a partner agency) into the UDOT traffic management system through the integrated system. The time interval range varied from nearly instantaneous to taking a few seconds to populate the fields that could be populated by the partner agency data.
Through this demonstration and the observations of the working system, incidents that were brought into the UDOT traffic management system could be updated and sent out to other agencies. Data could be linked from one system to the other through incident number and geo-location. The system was designed to allow the operator to make the final determination of where incidents reported from more than one agency are the same and should be linked or not. As the test successfully showed, the system did provide this capability.
It was more difficult to determine to what extent incident response procedures were improved. Because of the close working relationship among the FOT partner agencies, improvements in field response activities could not be readily observed. In fact, all those interviewed acknowledged that little, if any, improvement occurred in field procedures. However, this outcome was expected going into the test.
There were some improvements in incident response procedures and resulting efficiencies in documenting incident management in the operations and dispatch centers in the region. In addition to improved efficiency, there are three other improvements that should be mentioned:
- Time to enter an incident first reported by a partner agency.
- Accuracy of the information in the incident record.
- Number of incidents included in the incident reporting system.
The time required to enter incidents into the UDOT incident management system that were first reported by a partner agency was reduced as a result of automated data exchange. Previously, incident data was entered manually. Minimizing the time to enter an incident is important because all of the incident information that goes to the public either by the CommuterLink Web site or 511 comes from the UDOT incident management system. The quicker an incident can be reported, the sooner the public will know about it and take appropriate action. The incidents are reported essentially instantaneously to the CommuterLink Web site and 511 once the incident is "declared", or entered, by the operator.
In interviews with UDOT operators, estimates of up to 1 or 2 minutes in time saved to get the incident entered and located were reported. This improvement was recognized as a two-fold time savings factor. First, some of the incident fields were populated from information from the partner agency. Operators estimated a10- to 50-second improvement because of fields automatically being populated. The second factor is that the system can carry geo-location information so the incident can be located automatically rather than being placed at the proper location by the operator. The operators had estimated that between 1 to 2 minutes in time savings were realized because of the automatic placement of incidents.
Limited observations showed about a 5- to 10-second savings in time to enter incident data on incidents reported by VECC. VECC data had limited data brought into the system, however; only incident type and location are transmitted by the VECC system. (See later discussions of institutional challenges in section 5 and conclusions in section 6 for more discussion.) From observations made before the integrated system was implemented, normal incidents took from 50 to 120 seconds to populate and place. In the after observations, it took about 15 seconds to fill in the empty fields from VECC incidents. With automatic placement of incidents, the process is considered complete as soon as the empty fields are populated and the operator "declares" or enters the incident. The observations showed a time savings of roughly 35 to 105 seconds, relatively close to the operator estimates. (It should be noted that the observations, both before and after, were on relatively quiet days with few incidents and no major incidents. It is easy to see where greater time savings could occur on busier days.)
It should be noted that additional improvements would result for incidents reported by any of the partners with some modifications to the system. Those modifications include more automation in sending incidents to the integrated systems for all agencies other than VECC, which sends out all of its incidents; filtering incidents at the receiving agency so only incidents of interest are displayed to operators; and populating more fields in the integrated system, especially from VECC.
Accuracy of the information included in the incident record was improved because information from the partner CAD systems was imported directly into the UDOT incident management system. This process reduced the likelihood of introducing a manual operator error when the operator would re-enter the data. In addition, the geo-location information attached with the entries from some agencies reduced the likelihood that the incident would be positioned in the wrong location.
Although the interviews with operators downplayed any possibility of improved accuracy, the manual steps that were necessary in the old system provided some probability of errors, especially in placing the incident. Even though the improved accuracy may not have been perceived as a major improvement, it is worth mentioning.
The number of incidents included in the integrated system was considerably larger than in the nonintegrated system. All VECC incident entries are transmitted to the integrated system. Prior to the integration, UDOT operators relied on scanners or the VECC CAD Web site to find out about incidents outside UHP's jurisdiction. During the FOT, the operators needed to decide on which incidents from VECC in the integrated system should be brought into the incident management system and reported to the public. Interviewees thought that they may be handling between 75 and 100 percent more incidents. The incident logs showed that there were nearly 5 times as many incidents reported per month after the integrated system as before. (See section 4.2 regarding the test results for system impact.)
In discussions with UDOT managers, the opinion was expressed that some of the increase in number of incidents was because the test overall placed more emphasis on reporting incidents due to the heightened awareness of the importance of reporting incidents. Regardless of the overriding reason, there was a large increase in the number of incidents reported.
4.1.2. Automate Information Transfer between TMC and Emergency Responders
The second objective under system performance was to automate the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors. The following three hypotheses are associated with this objective:
- The system meets functional specifications.
- The FOTs will decrease the reliance on manual methods for exchanging information.
- The FOTs will increase the extent and reliability of information exchanges.
From the discussion in section 4.1.1, it was effectively demonstrated that the system met the functional specifications.
From observations and interviews, it was demonstrated that the integrated system reduced the reliance on manual methods for exchanging information. Incidents reported by partner agencies were transmitted to the integrated system and easily imported into the UDOT incident management system. Sharing information on incidents reported by other agencies, particularly from VECC, significantly reduced UDOT operator reliance on listening to scanners to "discover" incidents of interest.
Before the integrated system was deployed, the scanners were the primary way of UDOT's finding out about incidents from agencies other than UHP. Operators kept part of their attention on the scanners at most times. When busy with other tasks, the operators indicated that it would be very easy for an operator or dispatcher to miss an incident. If the operator missed part of the message from the scanner, it would be very difficult to get the information unless the operator/dispatcher called the agency that reported the incident. Additionally, if the location or agency was part of the message that was missed, there was no method available for the operator to gather or retrieve additional information. With the integrated system, operators were still able listen to scanners, but generally only after they saw an incident come in on the integrated system and they needed to get additional information.
The number of times the operator needed to phone other agencies to get additional information also was reduced. During the before observations, operators called other agencies routinely. During the after observation, operators only rarely called other agencies.
From observations and interviews, it was proved that integration increased the extent and reliability of information exchanges. Information passed to other agencies directly from CAD so conversations only were needed to clarify information, which translated to there being a reduced likelihood of operators misunderstanding the basic aspects of the incident.
4.1.3. Integration of Secondary Responders
Secondary responders (ambulance, utilities, etc.) were not included in the FOT. Initially, UDOT wanted to incorporate ambulance services and tow trucks. This approach was not pursued because the ambulance services were transitioning from private operators to municipal services. It was too early in the transition process to incorporate ambulance dispatch in the integrated system. Since the tow industry does not use CAD, it was not practical to incorporate this entity into the integrated system.
4.2 SYSTEM IMPACT TEST RESULTS
To assess the system impacts of the CAD-TMC deployment, data was collected from the following sources:
- UDOT incident logs.
- SLCFD incident logs.
- SLCPD incident logs.
- UTA call logs for paratransit and fixed routes services.
- VECC call and radio logs.
Data from before the CAD-TMC deployment was collected for the period from April through June 2004. Data from after the deployment was collected for the same period during 2005.
Table 5 summarizes the system performance study test results based on the discussion in section 3. Following the table is a discussion for each evaluation objective, along with the corresponding results in sections 4.2.1 through 4.2.5.
| Evaluation Objective | Hypothesis | Test Results |
|---|---|---|
| Objective #1: Productivity – To determine if the CAD-TMC integration improves the efficiency and productivity of incident response. | CAD-TMC integration enhances communications among responders. | Achieved – Key issue to be addressed is that of refining information exchange to meet agency specific requirements. |
| CAD-TMC integration improves efficiency of on-scene operations. | Not measured during the evaluation. | |
| CAD-TMC integration enhances efficiency in documenting incident management. | Achieved | |
| CAD-TMC integration reduces incident clearance times. | Not measured during the evaluation. | |
| Objective #2: Mobility - To determine if the CAD-TMC integration improves mobility and reduces delays during incidents. | CAD-TMC integration enhances mobility during incident management (IM) activities. | No impact measured during the evaluation. |
| Objective #3: Capacity/ Throughput - To determine if CAD-TMC integration enhanced incident-specific traffic management plans. | CAD-TMC integration enhances incident-specific traffic management plans. | Not measured during the evaluation. |
| Objective #4: Safety – CAD-TMC integration will reduce exposure of response personnel and secondary crashes during incident response activities | CAD-TMC increases safety for response personnel. | Not measured during the evaluation. |
| CAD-TMC increases safety to the traveling public. | Not measured during the evaluation. | |
| Objective #5: Traveler Information – To determine if CAD-TMC integration will improve incident management information available to travelers. | CAD-TMC integration enhances customer satisfaction and mobility during incident management activities by improving traveler information. | Qualitative assessment: Improved ability to post incident information for public access via 511, Web site. |
| UTA Objective: To determine if the integration of the UTA CAD system improves UTA’s ability to respond to incidents. | The CAD-TMC integration will enable UTA to more effectively implement reroute decisions in response to an incident. | CAD-TMC integration provided real-time information on unplanned incidents and complemented existing UTA incident management procedures. Additional benefit from system is information provided on planned incidents, such as road closures and/or construction activities. |
4.2.1. UDOT Incident Logs
Before the CAD-TMC deployment, the UDOT incident logs contained information about 336 incidents that occurred during the period from April until June 2004 as shown in table 6.
| Month | Count |
|---|---|
| April | 50 |
| May | 114 |
| June | 172 |
However, these incident logs did not include any information about accidents; they only included information about construction and other activities that could result in road closures. The incident type information was inferred from a numeric Incident Type field, which assumed that the same codes were used for the 2004 data as for the 2005 data. The duration recorded for these incidents was not very detailed, with incidents beginning and ending on the half hour.
There were some apparent inconsistencies between the codes included in the before and after data collected. The reasons for the inconsistencies were not known, and could range from a problem in translation with regard to the definition of incidents to broader data issues. Based on questions that arose from these inconsistencies, most of the quantitative analysis that was done on the before and after data sets was rejected by the Evaluation Team.
After the CAD-TMC deployment, the UDOT incident logs contained much more detail about a much larger set of incidents, with nearly 5 times as many incidents reported per month than in the before data. An interesting aspect of this increase in incidents is that the project partner agencies were reported to be discussing issues related to the type and quantity of data that needs to be collected, such as the need for data filters to ensure that the system is not overloaded with redundant information about an incident. Table 7 presents the increased amount of incidents logged during May and June 200512 over the corresponding months in 2004.
| Month | Count |
|---|---|
| May | 678 |
| June | 950 |
| June | 172 |
The after-deployment data was also more complete than before the deployment. For example, the "City" code (a code used to identify the city in which the incident occurred) was specified for only 69 incidents in the before data, and was specified for 833 incidents in the after data with the time stamps for the incidents were usually recorded to the nearest minute. Both of these observations indicated that the information obtained from the integrated system contained more real-time traffic information and that the overall accuracy of the data was highly improved.
The fact that the after data was more complete also meant that this data would support more detailed incident analyses than the before data. Table 8 shows how incident duration varied with the type of incident and recorded the type and quality of the data being generated by the integrated system. This level of detail and accuracy was significantly improved when compared with the before project data quality.
| Incident Duration | Incident Type: Accident | Incident Type: Stall |
Incident Type: Debris | Incident Type: Congestion | Incident Type: Construction |
|---|---|---|---|---|---|
| < = 5 min | 37 | 39 | 7 | -- | 1 |
| 5 to 15 min | 82 | 22 | 7 | 2 | -- |
| 15 to 30 min | 189 | 21 | 5 | 1 | -- |
| 30 to 60 min | 515 | 34 | 9 | 3 | 2 |
| 60 to 120 min | 394 | 22 | 3 | 3 | -- |
| 120 to 720 min | 78 | 15 | 1 | 2 | 36 |
| 720+ min | 2 | 1 | -- | -- | 7 |
4.2.2. UTA Call Logs
UTA provided two types of call logs: one each for the paratransit vehicles and for the fixed route vehicles. It should be noted that UTA paratransit dispatch uses its CAD system differently than UDOT. UTA defines their accident, incident, and note fields as follows:13
- Accident: This field can be either an accident with a vehicle or a passenger. Most accidents are with passengers.
- Incident: This field is used mainly for passenger behavioral issues (some dispatchers put these in the notes field).
- Note: This field is primarily used for passenger behavioral issues, broken down vehicles, passengers that don't show, and passengers that pay for round trip fares.
The paratransit logs included 328 records covering the period from April to June 2004. Each call was classified as an accident, incident, or note, as listed in table 9. Most of the records classified as an accident were actually misclassified, and an informal review of the accidents did not identify any that were related to incidents of concern to UDOT.
| Incident Type | Number of Records Before Deployment | Number of Records after Deployment |
|---|---|---|
| Accident | 23 | 415 |
| Incident | 54 | 63 |
| Note | 251 | 377 |
4.2.3. VECC Call and Radio Logs
The VEC call and radio logs are detailed logs of calls received by VECC and radio messages exchanged with field units. For example, the before VECC call logs included 180,814 records about VECC calls. The radio logs included more than 500,000 records of radio calls. As with the SLCPD and SLCFD data, there appeared to be little overlap between these incidents and the incident monitored by UDOT. For example, of the 500,000 radio log records in the before data, only 9 had a 10-code field related to traffic.
4.2.4. UTA CAD Integration
Table 10 summarizes the planned quantitative aspect of the UTA CAD integration assessment to complement the qualitative assessment based on the before and after interviews. This table includes the objective, hypothesis, measures of effectiveness (MOE), data sources, and analysis performed. It did not prove feasible to measure the changes in time to implement and rerouting of routes based on the available data sources, so the quantitative aspect of this assessment was not completed. However, the qualitative assessment based on the before and after interviews provided considerable insight into these and other impacts of the CAD integration on UTA operations. The remainder of this section reports on these insights gained from the qualitative assessment.
| Objective | Hypothesis | MOE | Data Sources | Analysis |
|---|---|---|---|---|
| To determine if the integration of the UTA CAD system improves UTA’s ability to respond to incidents. | The CAD-TMC integration will enable UTA to more effectively implement reroute decisions in response to an incident. | Changes in time needed to implement rerouting following an incident. Changes in time needed to end rerouting once an incident has been cleared. |
Sources include UTA CAD system and UTA logs; TMC incident logs. | Descriptive statistical analysis of key measures and comparison of baseline and after cases. |
UTA did not feel that the availability of this new information tool had a significant impact on the incident management practices and capabilities of UTA dispatchers. Due to the dynamic and autonomous nature of their roles, UTA dispatchers were already empowered to make operational decisions including those needed in response to incidents. To support this role, UTA already had protocols in place for the general operational response to various scenarios. This new information did not change these practices and capabilities, but did provide a new information exchange mechanism to complement those that were previously in place.
In practice, the new messaging interface did not usually address advance notice disruptions such as road closures or delays. Instead, the new messaging interface focused on the real-time status of unexpected disruptions. UTA already had established mechanisms for receiving advance notice disruption information through other channels, including CommuterLink in particular, and relied on ongoing information sharing with each of the 72 municipalities in which the agency operated to receive information on planned road disruptions.
Although the messages did not usually address advance notice disruptions, UTA dispatchers were asked to log these as well using the documentation arm of the CAD software. Overall, the extended information in the CAD system documentation arm served as a new source of information for customer service agents, who could ask the dispatcher to query the documentation arm information to help research a customer issue.
In the past, UTA would typically become aware of an on-street incident when it was first encountered in revenue service. The primary effect of the messaging interface was that in some cases, UTA would be made aware more quickly of an incident that might affect its operations, before it was encountered in revenue service. This established a new opportunity to mitigate the impact of an incident on operations. Although a supervisor still needed to assess the incident to determine if a detour could/should be implemented, the quicker notification allowed a supervisor to be dispatched more quickly, thus reducing the time until an operational response could be implemented and mitigating the impact. However, depending upon an incident's location and duration, it was not always feasible to implement an operational response. When this occurred, the quicker incident notice did not necessarily translate into a reduced operational impact.
For unplanned incidents, UTA's procedure would be to respond to on-site supervisor feedback. With the messaging interface, sometimes the supervisor could be dispatched before the incident was first encountered by an operator. However, if a UTA operator encountered an incident first, the UTA procedure would be to contact 9-1-1. UTA did not typically take responsibility for initiating an incident report to other agencies via the messaging interface without first having a supervisor onsite to assess it (by which time the incident report has usually already been initiated by a public safety agency).
UTA has had the capability for dispatchers to notify other parts of the UTA organization about a service disruption, using one of several internal email distribution lists. UTA opted not to establish any automated linkage between incoming incident messages and this email distribution capability. UTA relied on its dispatchers to assess incoming incident messages in the context of all other available information to decide which of these will result in a UTA service disruption.
UTA felt that although it was slower than some other agencies to initiate its participation in the FOT, the agency was able to implement its solution fairly quickly as a result of having developed its CAD system in-house and not needing to negotiate the system modification with a vendor.
12 After data was not available for the month of April 2005 due to delays in system start-up operations.
13 UTA definitions for accident, incident, and note provided by email communiqué May 18, 2006, via Mr. Richard Manser (UDOT) and Mr. Nolan Hess, TransCore (system integrator).