3. EVALUATION STRATEGY

3.1 EVALUATION STRATEGY OVERVIEW

The evaluation strategy developed for the CAD-TMC integration FOT was designed to address both ITS JPO and UDOT goals and objectives for the project. The goals and objectives for this evaluation were developed using an iterative approach involving extensive review by ITS JPO and the two affected States: Utah and Washington.

The Evaluation Team reviewed all available project documentation, including the application submitted to ITS JPO by each State in response to ITS JPO;s Request for Applications distributed on May 16, 2002. Based on this review, the Evaluation Team presented high-level goals and objectives in its proposal submitted in response to ITS JPO;s RFP of March 7, 2003. These proposed goals and objectives were reviewed with the ITS JPO Contracting Officer;s Technical Representative (COTR), the ITS JPO Public Safety Coordinator, and the Mitretek Analyst on May 6, 2003, and then again during a joint June 2, 2003 kick-off meeting with Utah State DOT representatives.

The proposed goals and objectives were revised based on these meetings, and presented to the ITS JPO COTR, the ITS JPO;s Public Safety Coordinator, and the Mitretek Analyst on June 16, 2003, and to Utah and Washington State during evaluation strategy briefings conducted on June 25 and June 26, 2003, respectively. The final evaluation and objectives presented in this plan reflect the input obtained from ITS JPO and the two States throughout this process.

3.2 GOALS AND OBJECTIVES

The Evaluation Team used the following high-level, ITS JPO-established FOT goals and objectives as the starting point for developing goals and objectives for the evaluation:

The FOT will demonstrate the feasibility of automating the seamless transfer of information between traffic management workstations and police, fire and Emergency Medical Services (EMS) CAD systems from different vendors.
The FOT will incorporate ITS standards such as IEEE 1512 and National Transportation Communications for ITS Protocol (NTCIP) into the integration of public safety and transportation information systems. Other standards areas that will have to be addressed are those pertaining to Geographic Information Systems (GIS).
The FOT will extend the level of integration to include secondary responders such as utilities; towing and recovery; public works; and highway maintenance personnel.

The ITS JPO further identified a number of specific quantitative goals and objectives to be assessed during the evaluation, in particular, to:

Determine how the FOT enhances communications among responders.
Assess the extent to which the FOT enhances efficiency in documenting incidents.
Determine how the FOT enhances on-scene operations.
Measure the extent to which the FOT reduces incident clearance times.

The ITS JPO also specified that the final evaluation report include an assessment of institutional and technical challenges, and a summary of lessons learned and benefits, both qualitative and quantitative.

The high-level goals established for the CAD-TMC integration FOT by UDOT included:

To demonstrate that open communication between the law enforcement and transportation agencies can improve emergency response and traveler information distribution. This open communication involves State agencies and county, municipal, and local government agencies.
To demonstrate how this information exchange can be done without placing additional burdens on the already busy emergency response and radio dispatch staffs.

The State also adopted the high-level goals and objectives for the FOT established by the ITS JPO described previously: automating the seamless exchange of data; using the appropriate ITS standards; and integrating local-, municipal-, and county-level emergency responders.

In developing goals for the evaluation, the ITS JPO- and UDOT-determined objectives were used to identify final evaluation goals that incorporated elements of both. The proposed goals were reviewed with both the ITS JPO and the State to ensure consistency and to ensure that data was available to conduct tests to support the evaluation.

As shown in table 3, the final evaluation goals and objectives were designed to enable the assessment of project performance as compared to both the ITS JPO and UDOT goals.

**Table 3. Final Evaluation Goals and Objectives**
Evaluation Goal	Evaluation Objectives
Assess System Performance	Automate the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors. Incorporate ITS standards such as IEEE 1512 and NTCIP into the integration of public safety and transportation information systems. Also, address standards related to GIS and sharing data between map databases from different vendors. Extend the level of integration to include secondary responders such as utilities; towing and recovery; public works; and highway maintenance personnel.
Assess System Impact	CAD-TMC integration will improve productivity and efficiency. CAD-TMC integration will improve mobility. CAD-TMC integration will improve safety. Assess CAD-TMC integration with 511/Internet interface. Assess the integration of the UTA CAD and the impact on transit operations.
Assess Institutional Challenges and Technical Issues	Identify institutional and technical challenges and document how they were resolved.
Identify Lessons Learned	Develop a Lessons Learned Summary. Identify institutional and technical challenges and document how they were resolved.
Summarize Benefits	Develop a Benefits Summary.

The Evaluation Plan articulated how to assess the degree to which the goals and objectives presented in table 3 would be met. The following studies and assessments were developed to assess these goals and objectives, as discussed in sections 3.2.1 through 3.2.6.

3.2.1. System Performance Study

The system performance test was designed to:

Describe the environment in which the FOT will operate that could affect the applicability of the CAD-TMC concept to other sites and the interpretation of the system impacts data. This will help other potential deployment users to better understand the applicability of the CAD-TMC concept to their sites.
Identify key performance measures that should be met by similar deployments to achieve the system impacts observed by the FOT deployment. This will help other deployment users identify and focus on the performance goals needed to achieve similar results. Also, document the design basis for these performance measures to help other deployment users adjust these measures to better suit their local conditions.
Calculate and document the key performance measures for the system as it was deployed. This will help identify limitations in the deployed system that might affect the observed system impacts. Also, identify and document other performance measures that were gathered by the deployment team (e.g., during component and integration testing). While this data was not as critical to the evaluation as the key measures, the data should be available from the deployment team to reduce the cost associated with reporting the data.
Identify other factors that affect the deployed system's performance. After the system is deployed, users may identify other factors that could make the system more useful and knowledge that could benefit others in developing similar systems.

In addition to these activities related to evaluating the performance of the deployed system, the system performance study was intended to:

Evaluate the degree to which ITS standards such as IEEE 1512 and NTCIP were incorporated into the deployed system.
Address the approach used to share data between map databases from different vendors and GIS standards that were applied.

3.2.2. System Impact Study

System impacts were evaluated using elements of the framework provided by the ITS JPO's National ITS Program Goal Areas: Mobility; Capacity/Throughput; Productivity; Safety; and Customer Satisfaction.¹¹ The evaluation sought to quantify and document the benefits across these measurable areas for two very broadly defined beneficiary groups: incident responders and travelers. The system impact study was designed to:

Determine if the CAD-TMC integration improves the efficiency and productivity of incident response.
Determine if the CAD-TMC integration improves mobility and reduces delays during incidents.
Determine if CAD-TMC integration enhances incident-specific traffic management plans.
Determine if the CAD-TMC integration will reduce exposure of response personnel and secondary crashes during incident response activities
Determine if CAD-TMC integration will improve incident management information available to travelers.

3.2.3. Institutional Challenges Assessment

The institutional challenges were identified and documented primarily through stakeholder interviews. Interviews with project stakeholders provided the primary information source for identifying challenges and the processes by which they were resolved. These interviews were conducted on a before and after deployment basis.

The institutional challenges study was intended to:

Document inter-agency cooperation at the State level, in particular, the processes used for identifying and solving problems.
Assess how county and municipal agencies are integrated into the program (VECC, SLC).
Identify what information is shared, and how the agencies determined that this was the right information to share.
Document how UHP and UDOT determined what the information availability would be for exchanges between the CAD-TMC systems.
Document how frequently the information provided through the project is used by:
- Responders.
- Travelers.
- Media.
Document how these end-users used the information provided, and identify how the information was used.
Determine if end-users found the information useful, and why or why not.
Assess how the various CAD vendors were able to establish working relationships and share data.

3.2.4. Technical Challenges Assessment

The technical challenges assessment documented how the FOT teams addressed technical challenges such as overcoming the barriers associated with incompatible and/or proprietary systems. In conducting the assessment, the Evaluation Team primarily relied on interviews with technical staff at each participating agency to identify the specific challenges addressed and evaluate how those challenges were resolved. Results from this assessment are presented in section 6.

3.2.5. Lessons Learned Assessment

The lessons learned assessment summarized lessons learned during the other portions of this evaluation. The Evaluation Team also explicitly requested information on lessons learned during interviews and meetings associated with the evaluation. Results from this assessment are presented in section 6.

3.2.6. Benefits Summary Assessment

The Benefits Summary documents benefits derived from the all of the individual studies in this evaluation. Results from this assessment are presented in section 6.

3.3 DATA COLLECTION

Both qualitative and quantitative data were collected for the before (baseline) and after the FOT deployment. The collection of before data focused on establishing a baseline that was used to measure the impact of the FOT deployment. Collection of after data provided data that was compared to the baseline data to determine the impact of the FOT deployment.

Qualitative data collection was conducted for both the before and after data collection phases using the following methods:

Stakeholder/Vendor Interviews. The Evaluation Team interviewed stakeholders/ vendors in person or via phone as the primary means to collect the qualitative information/data needed to successfully perform the CAD-TMC integration FOT evaluation. Stakeholder interviews also were used as a means of identifying issues relevant to the CAD-TMC evaluation. Stakeholder agencies interviewed included UDOT, UHP, VECC, UTA, SLCPD, and SLCFD.
Site Visits. The Evaluation Team conducted periodic site visits with appropriate stakeholders/vendors to collect needed data not easily transmitted via phone, e-mail, or other convenient means.
Observations. Visual observations were used as a means of collecting data that is not otherwise documented or easily conveyed. An example of this included documenting the activities of CAD and TMC operators before and after the new system was deployed to identify any changes in day-to-day procedures or work requirements.

Quantitative data were obtained for the periods of April through June 2004 (before) and for April through June 2005 (after). The qualitative data collected were used to gain user impressions of system performance and impacts, and to identify institutional/technical challenges and lessons learned. Quantitative data were used to assess system performance and system impact.

Sources for quantitative data collected through this FOT are presented by agency, as listed below. Under each data source are the specific field headings from which data were pulled.

UDOT:
- Incident Management System records:
  1. DIRECTION
  2. INCIDENT ID
  3. COUNTY CODE ID
  4. ESTIMATED END
  5. SCHEDULED START
  6. SCHEDULED END
  7. ACTUAL START
  8. ACTUAL END
  9. CONFIRM TIME
  10. LAST UPDATE TIME
  11. POSITION
  12. POSTION
  13. INCIDENT TYPE ID
  14. LOCATION TYPE ID
  15. PRIMARY ROUTE
  16. SECONDARY ROUTE
  17. LOCATION TEXT

UTA:
- Dispatch System message logs (Fixed Route Operations):
  1. LOG NUMBER
  2. LOG DATE
  3. LOG TIME
  4. CLEAR TIME
  5. DIRECTION
  6. MINUTES LATE
  7. PROBLEM
  8. COMMENTS
- Dispatch System message logs (Paratransit Operations):
  1. LOG NUMBER
  2. LOG DATE
  3. LOG TIME
  4. ACCIDENT/INCIDENT/NOTE
  5. CLEAR TIME
  6. LOCATION
  7. PROBLEM
  8. COMMENTS

VECC:
- CAD system message logs (Call records):
  1. RECORD NUMBER
  2. CALL TYPE (LAW/FIRE/EMS)
  3. CALL NATURE
  4. PRIORITY OF CALL
  5. WHEN OCCURRED EARLIEST
  6. WHEN OCCURRED LATEST
  7. WHEN REPORTED
  8. WHEN MODIFIED
  9. HOW RECEIVED
  10. RESPOND TO ADDRESS
  11. CITY CODE
  12. GEOBASE ADDRESS ID
- CAD system message logs (Radio logs):
  1. LOG DATE
  2. X POSITION
  3. Y POSITION
  4. UNIT
  5. CALL ID
  6. AGENCY
  7. DESCRIPTION
  8. SHIFT
  9. CALL TYPE

SLCPD:
- CAD system message logs:
  1. DATE
  2. TIME
  3. CASE TYPE
  4. LOCATION
  5. X POINT
  6. Y POINT
  7. SEVERITY
  8. NUMBER VEHICLES
  9. NUMBER INJURED
  10. NUMBER KILLED

SLCFD:
- CAD system message logs:
  1. INCIDENT TYPE DESCRIPTION
  2. INCIDENT BEGIN TIME
  3. INCIDENT END TIME

Qualitative data were collected through interviews with and observations of the following agencies:

UDOT. Before and after interviews were conducted with UDOT TOC personnel in July 2004 and September 2005. Before and after interviews were conducted with system development personnel in July 2004 and September 2005. Evaluation Team members also observed the operation at the TOC in July 2004 and September 2005. Various field observations and interviews with field personnel occurred at different times during the evaluation periods. UDOT personnel also were involved with several meetings to discuss the integration project, the most notable of which was the agency partner meeting held September 7, 2005.
UHP/DPS. Input was received from key field and dispatch personnel through interviews and meetings during the evaluation period.
UTA. Before and after interviews were conducted with UTA in August 2003 and October 2005. UTA also provided feedback during the agency partner meeting in September 2005.
VECC. Primary input from VECC occurred during the agency partner meeting in September 2005.
SLCPD. Primary input from SLCPD occurred after it came online, during the agency partner meeting in September 2005.

3.4 MODIFICATION TO EVALUATION SCOPE

An Interim Project Review (IPR) of the ITS JPO Public Safety Program-funded Utah CAD-TMC integration FOT was held on January 28, 2005. Participants included the Joint Program Office ITS Public Safety Coordinator, the COTR, the Mitretek Analyst, the Utah Project Manager, representatives from other stakeholder agencies, the system integrator, and the Evaluation Team. The purpose of the IPR was to:

Provide the project team with a status report on evaluation activities, in particular, on the status of baseline data collection.
Obtain an update on the status of project implementation.
Discuss next steps:
- When to collect after project data.
- When to complete evaluation activities.
- Assess potential benefits of expanding the scope of the evaluation.

No significant developments beyond the original scope of the evaluation were identified during the IPR. A decision was reached by the meeting participants that the evaluation would be completed within the existing scope and schedule. It was determined that the SLCFD system would not be online and tested until April 2005; therefore, SLCFD would not be included in the final data collection activities, which were scheduled for and conducted between May and June of 2005. Further discussion determined that additional activities would include the after interviews and operation observations.

¹⁰ ITS JPO, ITS Public Safety Program brochure, titled "DOT Projects in Utah, Washington State Will Demonstrate Public Safety, Transportation Integration System."

¹¹ Additional information regarding the ITS Evaluation Guidelines - ITS Evaluation Resource Guide can be accessed from the ITS Joint Program Office Web site at: <http://www.its.dot.gov/EVAL/eguide_resguide.htm>.

Previous | Next