ITS - Intelligent Transportation Systems Report ITS Home Page
Table of Contents

 

4. TEST RESULTS

This section of the report documents the results of evaluation tests for the Washington CAD-TMC FOT. This includes both quantitative and qualitative results. It should be noted that a couple of issues affected the results of the evaluation. First, the interface with the secondary responders was eliminated (see section 4.1.3). The second issue affected the extent to which the FOT resulted in impacts to traffic operations. Specifically, WSDOT TMCs already had WSP CAD data and terminals prior to implementing the integrated system. This fact limited the impact of the CAD-TMC integration, since TMC operators already had access to WSP CAD data before the deployment.

The data discussed in section 3.1 was collected and analyzed according to the evaluation strategy described in section 3.4. This section presents the analysis results and a results summary regarding the system performance and system impact FOTs. Institutional and technical challenges were also assessed; however, because these are completely qualitative in nature, they are presented in section 5, Evaluation Findings. The lessons learned, which also provided important findings for this evaluation, are presented in section 6, Conclusions and Recommendations.

4.1 SYSTEM PERFORMANCE TEST RESULTS

Table 5 summarizes the system performance study results based on the discussion in section 3. Following the table is a discussion for each evaluation objective, along with the corresponding results in sections 4.1.1 through 4.1.3.

Table 5. System Performance Test Results Summary
Evaluation Objective Hypothesis Test Results
Objective #1: Document the system component performance. The system meets functional specifications. Achieved
The CAD and TMC systems will be able to link data on an incident. Achieved
Using the system improved incident response procedures. To a significant extent, achieved through prior projects.
Project specific impact not measurable.
Objective #2: Automate the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors. The system meets functional specifications. Achieved.
The FOTs will decrease the reliance on manual methods for exchanging information. Achieved previously through placement of CAD terminals at TMCs. Enhanced through project.
The FOTs will increase the extent and reliability of information exchanges. Preliminary result - achieved.
Objective #3: Extend the level of integration to include secondary responders such as utilities, towing and recovery, public works, and highway maintenance personnel. Improved integration of secondary responders will reduce incident recovery time by getting required recovery personnel to the incident site as quickly as possible to begin recovery operations. Not achieved during the evaluation period.


4.1.1. Objective #1: Document the System Component Performance

Following are the three hypotheses associated with the objective to document the system’s performance:

The Evaluation Team relied on a combination of observations and interviews to determine whether or not the system met the functional specifications. Actually seeing the system work and finding out if the system met operator expectations were the best indicators to determine that the system successfully met system performance needs.

Through interviews with the WSDOT system developers and the WSDOT project manager, the Evaluation Team obtained qualitative assessments verifying that the system performed according to expectations and the functional specifications. The only major concern, identified during direct observations of the integrated system, was regarding latency in the system from the time when an entry was entered into and displayed on the WSP CAD, until it was displayed in CARS, the integration platform. The existence of this latency was identified during the development of system requirements and design. As mentioned in section 2, the latency is a product of the way in which the WSP CAD system was designed. There was additional latency noted in presenting information in the 511 system or on the WSDOT traffic information Web site. This also was foreseen during the project conceptualization phase.

The latency was caused by several factors. As previously described, the WSP CAD system posted updates to the integrated system every 2 minutes. Thus, while an incident may well be identified by a WSDOT operator viewing a CAD terminal on a real-time basis, the WSP CAD system did not push this information to the CARS system for up to 2 minutes. A second factor is that the CARS system also has up to a 2-minute lag when pulling the information provided by the WSP CAD into CARS.

The latency did not affect whether the system met its functional specifications. However, it is important to note that the significance of the latency differed, depending on the area and the view of the WSDOT operators involved. Operators in more rural regions were reported to be satisfied with the system and its performance. In urban areas, there were more mixed feelings about the system and its inherent latency issues. Only one operator interviewed in the Seattle TMC said he used the integrated system. The other operators entered incident information directly into CARS as soon as they saw an incident of interest displayed on the WSP CAD terminal. They did not feel they could wait for the incident to display in the integrated system to complete the entry and send it to the travel information systems.

It is important to realize that the operators in the Seattle and Tacoma area had access to WSP CAD terminals before the integrated system was implemented. The operators were used to seeing incidents on the CAD terminal, deciding which ones were of interest, and entering the appropriate information into CARS. Because of this experience, any latency from the display on the CAD terminal to the integrated system would be noticeable. Operators in rural regions did not have access to WSP CAD terminals prior to the integrated system and getting the information, even with some latency, greatly improved the information and timeliness of incidents that they report to the travel information systems.

An incident entered in the WSP CAD system had a unique case number assigned to it. This case number allowed updates to a given incident to be easily tracked and linked to the original record. The integrated system used the case number to update records in the CARS system. In addition, operators had the ability to update information on an incident and link information manually.

It was difficult to determine to what extent incident response procedures were improved. Because of the close working relationship between WSDOT and WSP for years prior to the FOT, improvements in field response activities could not be observed. In fact, all those interviewed acknowledged that little if any improvement occurred in field procedures. However, this was expected going into the test.

There were some improvements in incident response procedures in the WSDOT operations centers. Improving the efficiency of documenting incident management is discussed covered in the next section. In addition to improved efficiency, there are two other improvements that should be mentioned:

The time required to complete the entry of incidents into the WSDOT CARS reporting system was reduced for those incidents that operators accepted from the WSP CAD system. In some cases, the latency problem was significant enough that operators didn't wait for the incident to come through the integrated system. In those cases, no change in time to enter occurred.

Previously, all CARS incident data was entered manually. The operators who used the integrated system had most of the CARS fields populated by the integrated system, which saved them time and allowed them to focus on other duties. The operators who used the integrated system either had no access to a WSP CAD terminal, or they had other duties, such as radio dispatch or tunnel control system operation, in addition to monitoring and reporting incidents.

The operators who mostly entered the data directly into CARS without waiting for the integrated system to report generally had access to WSP CAD terminals and were dedicated to traffic management tasks. Minimizing the time to enter an incident was deemed important because the operators generally had multiple duties, even if they were dedicated to traffic management. The quicker an incident could be reported, the more time the operators had to perform their other duties.

Accuracy of the information included in the incident record was improved when the integrated system was used because information from the WSP CAD system was imported directly into the CARS system. This reduced the likelihood of introducing a manual operator error when the operator would re-enter the data. In addition, the geo-location information attached with the entries from the WSP reduced the likelihood that the incident would be positioned in the wrong location.

Although the interviews with operators downplayed any possibility of improved accuracy, the steps that were necessary for manual entry provided some probability of errors, especially in placing the incident. Even though the improved accuracy may not have been a major improvement, it is worth mentioning.

4.1.2. Automate Information Transfer between TMC and Emergency Responders

The second objective under system performance was to automate the seamless transfer of information between traffic management workstations, police, and EMS CAD systems from different vendors. The following three hypotheses are associated with this objective:

From the discussion in section 4.1.1, it was effectively demonstrated that the system met functional specifications.

From observations and interviews, it was demonstrated that the integrated system reduced the reliance on manual methods for exchanging information when the operators chose to use it. Incidents reported by WSP CAD were transmitted to the integrated system and easily imported into CARS. For regions that did not have WSP CAD terminals, this information sharing was the only reliable way to receive incident information. Where CAD terminals were not available, WSDOT operators relied on scanners, calls from WSP, and radio calls from WSDOT field personnel to find out about incidents. It was very easy to miss incidents because the operators didn't hear scanner reports or because WSP dispatchers or WSDOT field personnel were too busy to call in the incident. WSP dispatchers or WSDOT field personnel were too busy to call in the incident.

From observations and interviews, integration increased the extent and reliability of information exchanges. Information passed from WSP directly through CAD to CARS so conversations were needed only to clarify information. There was a lower likelihood of misunderstanding basic aspects of the incident. In locations where WSP CAD terminals existed, this benefit was already realized. However, where the terminals weren't available, the ability to focus voice communication only on details that need clarification was valued as a tremendous benefit.

4.1.3. Integration of Secondary Responders

Secondary responders (ambulance, utilities, etc.) were not included in the FOT. Skagit County EMS was originally going to be included in the FOT, but did not participate because the agency could not see the value for its operation. Historically, there were not enough incidents in the Skagit area to be useful enough to justify the agency's participation.

4.2 SYSTEM IMPACT TEST RESULTS

To assess the system impacts of the CAD-TMC deployment, data was collected from the following sources:

Data from before the CAD-TMC deployment was collected for the period from April through June 2004. Data from after the deployment was collected for the same period during 2005.

Table 6 summarizes the system performance study results based on the discussion in section 3. Following the table is a discussion for each evaluation objective, along with the corresponding results in sections 4.2.1 through 4.2.5.

Table 6. System Impact Test Results Summary
Evaluation Objective Hypothesis Test Results
Objective #1: Productivity –To determine if the CAD-TMC integration improves the efficiency and productivity of incident response. CAD-TMC integration enhances communications among responders. Achieved with WSDOT and WSP.
CAD-TMC integration improves efficiency of on-scene operations. Not measured during the evaluation.
CAD-TMC integration enhances efficiency in documenting incident management. Partially achieved; further reductions will enhance results.
CAD-TMC integration reduces incident clearance times. Not measured during the evaluation.
Objective #2: Mobility – To determine if the CAD-TMC integration improves mobility and reduces delays during incidents. CAD-TMC integration enhances mobility during incident management (IM) activities. No impact measured during the evaluation.
Objective #3: Capacity/ Throughput –To determine if CAD-TMC integration enhanced incident-specific traffic management plans. CAD-TMC integration enhances incident-specific traffic management plans. Not measured during the evaluation.
Objective #4: Safety - CAD-TMC integration will reduce exposure of response personnel and secondary crashes during incident response activities. CAD-TMC increases safety for response personnel. Not measured during the evaluation.
CAD-TMC increases safety to the traveling public. Not measured during the evaluation.
Objective #5: Traveler Information - To determine if CAD-TMC integration will improve incident management information available to travelers. CAD-TMC integration enhances customer satisfaction and mobility during incident management activities by improving traveler information Not directly measured. Increased number of incidents posted to traveler information systems indicates improved flow of information to public.


Although the impacts of the CAD-TMC FOT on productivity, mobility, safety, capacity, and throughput are further documented here, it is recommended that the reader takes care when interpreting these results.

The high degree of operational integration between WSP and WSDOT that existed before the CAD-TMC FOT meant that the impact of the FOT on operational productivity would be limited. Many of the operational benefits that could be achieved through the CAD-TMC integration had already been accomplished through other means, such as providing a CAD terminal in the individual TMCs.

Other potential benefits of the integration were negated by technical limitations of the integration. For example, most TMC operators reported that the long lag time before the automated system made CAD incidents available to CARS induced them to continue to use the manual, pre-FOT approach for populating CARS with incident data.

Thus, while this section documents the system impacts of the Washington CAD-TMC FOT, these impacts may under-represent the impacts that might be found in a similar integration at other locations.

4.2.1. Impact on Productivity

Initially, it was expected that there could be significant improvements in productivity resulting from the FOT. The availability of CAD data at the TMC could help TMC operators respond more quickly when an incident occurred, and that the automated inclusion of the CAD data in the TMC systems could help TMC operators respond to incidents more quickly and efficiently. Improved availability of traffic data for WSP dispatchers could improve their ability to manage incidents. The following sections address the impact of the FOT on the productivity of regional operations.

Impact on WSP Operations

The Evaluation Team had anticipated collecting after project data for the period of April through June 2005. However, the system deployment was somewhat delayed and data was collected for only May and June. Even with this limited amount of data, the data collected post deployment did show an impact on WSP operations as shown in figure 2. Overall, there was about a 10 percent increase in the number of WSP CAD incidents recorded following the integrated CAD-TMC deployment, which reflected the improved documentation function established through the FOT.

Bar graph shows an approximately 10 percent increase in the number of WSP CAD incidents recorded following the integrated CAD-TMC deployment compared to the before deployment case.
Figure 2. CAD Incidents Before and After the CAD-TMC Deployment.

Although there was an increase in the overall number of incidents recorded once the system became functional, the data do not show that there was any significant change in WSP incident response times before and after the deployment, as shown in figure 3.12 This reflects the maturity of the Washington State incident response program and the significant operational benefits already achieved prior to the deployment, as discussed previously in the report.

Bar graph shows data sets for 911 calls, ADMIN (representing an 'other' category), and FIELD INITATED (radio or cellular call ins). There is only a slight difference in before and after arrival times at incidents reported via 911 calls. There is no before data for the 'other' category, and there is no data for either before or after for arrival times when incidents are reported via radio or cellular phone.
Note: 911 includes all incidents reported by 911 calls. Field initiated includes all incidents reported by radio or cellular phone calls from response personnel. ADMIN is a catch-all category that includes all other reported incidents.
Figure 3. Arrival Time for CAD Incidents.

Impact on WSDOT Operations

The CAD-TMC deployment was expected to improve WSDOT operations through:

These impacts were expected to result in more incidents included in CARS, with correspondingly improved traveler information. In addition, these impacts were expected to result in more efficient operations due to operators spending less time entering information into CARS. As noted earlier in this report, technical difficulties limited the extent to which these potential benefits were achieved. However, the initial results obtained from CARS indicated that as system deployment progressed, the number of incidents included in CARS increased by about 25 percent in the last four weeks of data collection based on the "before" and "after" project data collected. Figure 4 shows the overall increase in reported incidents shown in CARS. The figure compares the before and after project data for the periods of April-June 2005 and April-June 2006, respectively. The slow build-up of after project reported incidents was the result of technical delays in system deployment and operation. The system was not completely implemented until about week 4 of the "after" data collection time period. The results presented in Figure 5A demonstrated that the most immediate impact of the expected project benefits, as discussed earlier in this section, was obtained in highly urbanized I-5 corridor, which includes Seattle. The results presented in Figure 5b show that these same benefit has not yet been fully realized, due in part to the fact that the rural regions of the State do not have as many incidents to report.

Bar chart shows steady increase in 'after' project reported incidents up to week 13.
Figure 4. Number of CARS Incidents Before and After Deployment.

Figure 6 reinforces the observation presented in figure 5 by showing that the number of CARS incidents, relative to the number of CAD incidents, increased after the CAD-TMC deployment. A concern expressed by WSDOT personnel in the before deployment interviews was that prior to deployment, it did not appear that all WSP CAD incidents were being captured by WSDOT. The increase in CARS incidents recorded after the deployment indicated that the automated transfer of incident data did record those incidents not previously reported.

Bar chart shows increasing number of CARS incident in 'after' case in the northwest region up to week 13.
Figure 5A. Number of CARS Incidents, Northwest Region.

Bar chart shows that number of CARS incident reported in the 'after' case were not significantly different from those reported in the 'before' case, nor was there an increasing trend in the 'after' case up.
Figure 5B. Number of CARS Incidents, Rest of State.

Figure 6 reinforces the observation presented in figure 5 by showing that the number of CARS incidents, relative to the number of CAD incidents, increased after the CAD-TMC deployment. A concern expressed by WSDOT personnel in the before deployment interviews was that prior to deployment, it did not appear that all WSP CAD incidents were being captured by WSDOT. The increase in CARS incidents recorded after the deployment indicated that the automated transfer of incident data did record those incidents not previously reported.

Bar chart shows that number of CARS incidents reported in the 'after' case began to increase beyond the number reported in the 'before' case about 11 weeks into the study, and about 4 weeks after implementation.
Figure 6. Number of CARS Incidents Relative to CAD Incidents.

The Impact on the Operations of Secondary Responders

The original design of the Washington CAD-TMC deployment called for development of a SECONDARY ALERT Interface that would make incident information available to secondary responders outside of WSP and WSDOT, such as local EMS providers, tow truck dispatchers, and local utility companies. Two types of SECONDARY ALERT interfaces were to be developed. The first was to insert information directly into secondary responder CAD systems (e.g., for Skagit County EMS), and the second was to be a special public responder Web site that would include information from CARS especially tailored for secondary responders. As the project progressed, the secondary responders did not show much interest in these interfaces, so the interfaces were not developed. Consequentially, the Washington CAD-TMC FOT did impact the operations of secondary responders.

4.2.2. The Impact on Mobility, Safety, Capacity, and Throughput

The original plans for the CAD-TMC evaluation called for an assessment of the impact of the CAD-TMC deployment on mobility, safety, capacity, and throughput. The concept behind these plans was that improved TMC and CAD operations (e.g., faster response because of access to CAD information and more reliable traveler information based on CAD information) could result in improved incident response. Further, the improved incident response was expected to improve mobility, safety, capacity, and throughput. Because the CAD-TMC deployment had little impact on TMC and CAD operations, an impact on mobility, safety, capacity, and throughput was not possible.



12 The incident logs included incidents with arrival times of more than 39,000 minutes (equivalent to about 27 days). These apparently errant arrival times were eliminated by including only incidents in which the total incident time was less than 1 hour. The "Field Initiated" arrival times were zero because these incidents were initiated by field officers arriving at the scene of an incident.

Previous | Next