3. EVALUATION STRATEGY

3.1 EVALUATION STRATEGY OVERVIEW

The evaluation strategy developed for the CAD-TMC integration FOT was designed to address both ITS JPO and WSDOT goals and objectives for the project. The goals and objectives for this evaluation were developed using an iterative approach involving extensive review by the ITS JPO and the two affected States: Utah and Washington.

The Evaluation Team reviewed all available project documentation, including the application submitted to the ITS JPO by each State in response to the ITS JPO's Request for Applications distributed on May 16, 2002. Based on this review, the Evaluation Team presented high-level goals and objectives in its proposal submitted in response to the ITS JPO's RFP of March 7, 2003. These proposed goals and objectives were reviewed with the ITS JPO's Contracting Officer's Technical Representative (COTR) and the Mitretek Analyst on May 6, 2003, and then again during a June 2, 2003 kick-off meeting with Washington State DOT representatives.

The proposed goals and objectives were revised based on these meetings, and presented to the ITS JPO COTR, the ITS Public Safety Coordinator, and the Mitretek Analyst on June 16, 2003, and to Utah and Washington State during evaluation strategy briefings conducted on June 25 and June 26, 2003, respectively. The final evaluation and objectives presented in this plan reflect the input obtained from ITS JPO and the two States throughout this process.

3.2 GOALS AND OBJECTIVES

The Evaluation Team used the following high-level, ITS JPO-established FOT goals and objectives as the starting point for developing goals and objectives for the evaluation:

The FOT will demonstrate the feasibility of automating the seamless transfer of information between traffic management workstations and police and EMS CAD systems from different vendors.
The FOT will incorporate ITS standards such as IEEE 1512 and National Transportation Communications for ITS Protocol (NTCIP) into the integration of public safety and transportation information systems. Other standards areas that will have to be addressed are those pertaining to Geographic Information Systems (GIS).
The FOT will extend the level of integration to include secondary responders such as utilities; towing and recovery; public works; and highway maintenance personnel.

ITS JPO further identified a number of specific quantitative goals and objectives to be assessed during the evaluation, in particular, to:

Determine how the FOT enhances communications among responders.
Assess the extent to which the FOT enhances efficiency in documenting incidents.
Determine how the FOT enhances on-scene operations.
Measure the extent to which the FOT reduces incident clearance times.

ITS JPO also specified that the final evaluation report include an assessment of institutional and technical challenges, and a summary of lessons learned and benefits, both qualitative and quantitative.

The high-level goals established for the CAD-TMC integration FOT by WSDOT included:

To demonstrate that open communication between the law enforcement and transportation agencies can improve emergency response and traveler information distribution. This open communication involves State agencies and county, municipal, and local government agencies.
To demonstrate how this information exchange can be done without placing additional burdens on the already busy emergency response and radio dispatch staffs.¹⁰

The State also adopted the high-level goals and objectives for the FOT established by ITS JPO described previously-automating the seamless exchange of data; using the appropriate ITS standards; and integrating local-, municipal-, and county-level emergency responders.

In developing goals for the evaluation, the Evaluation Team used the ITS JPO- and WSDOT-determined objectives to identify final evaluation goals that incorporated elements of both. The proposed goals were reviewed with both ITS JPO and the State to ensure consistency and to ensure that data was available to conduct tests to support the evaluation.

A comparison of the final evaluation goals as combined with the ITS JPO and WSDOT goals for the project is shown in table 4. This table also identifies the objectives that were developed to support the evaluation goals. As can be noted, the final evaluation goals and objectives were designed to enable the assessment of project performance against both ITS JPO and USDOT goals.

**Table 4. Combined ITS JPO and WSDOT Evaluation Goals**
ITS JPO	WSDOT	Evaluation Goal	Evaluation Objective
System Performance
The FOT will automate the seamless transfer of information between traffic management workstations and police, fire and EMS CAD systems from different vendors.	Enable the automated electronic exchange of data between agencies on a real-time basis. This electronic data exchange will address the one outstanding need that still exists between WSDOT and WSP.	Document system component performance.	Determine the feasibility of automating the seamless transfer of information between traffic management workstations and police, fire, and EMS CAD systems from different vendors.
The FOT will incorporate ITS standards such as IEEE 1512 and NTCIP into the integration of public safety and transportation information systems. Other standards areas that will have to be addressed are those pertaining to Geographic Information Systems (GIS).	The State has committed to using ITS standards and to develop a system that conforms to the National ITS Architecture.	Document system component performance.	Investigate the benefits of incorporating ITS standards such as IEEE 1512 and NTCIP into the integration of public safety and transportation information systems. Also, address standards related to GIS and sharing data between map databases from different vendors.
The FOT will extend the level of integration to include secondary responders such as utilities, towing and recovery, public works and highway maintenance personnel.	Integrate local, county, and municipal government emergency management and response agencies (fire and rescue, law enforcement).	Document system component performance.	Determine the benefits of extending the level of integration to include secondary responders such as utilities, towing and recovery, public works, and highway maintenance personnel.
System Impact
Computer requirements. FOT enhances communications among responders. FOT enhances efficiency in documenting incidents. FOT enhances on-scene operations. FOT reduces incident clearance times. FOT improves information available to traveling public and media.	Expected project impacts include: Reduce the time needed to deploy assets to respond to an incident. Reduce exposure of response personnel. Reduce secondary collisions resulting from the initial incident. Reduce the time needed to post incident-related information on the State's traveler information systems (Internet, 511), as well as provide information to the media. Improve the quality of information provided to the media and traveling public.	Conduct a system impact study.	Determine whether CAD-TMC integration improves: Productivity and efficiency. Mobility. Safety. Integration with 511/Internet interface.
Institutional and Technical Challenges
Assess institutional and technical challenges.	Improved inter-agency relationships, in particular understanding of each agency's role and duties, to dissolve institutional barriers between the two agencies.	Identify institutional and technical challenges.	Document process by which institutional and technical issues were resolved.
Lessons Learned
Document lessons learned.	Not specifically established, but fully supported.	Identify lessons learned.	Document lessons learned that would be of benefit for other jurisdictions considering similar deployments.
Benefits
Summarize benefits.	Not specifically established, but fully supported.	Identify benefits from integrating the CAD-TMC system.	Identify qualitative and qualitative benefits achieved by the deployment that can be used by other jurisdictions to obtain support (programmatic, technical, funding) for similar deployment.

The Evaluation Plan articulated how to assess the degree to which the goals and objectives presented in table 4 would be met. The following studies and assessments were developed to assess these goals and objectives, as discussed in sections 3.2.1 through 3.2.5.

3.2.1. System Performance Study

The system performance study was designed to:

Describe the environment in which the FOT will operate that could affect the applicability of the CAD-TMC concept to other sites and the interpretation of the system impacts data. This will help other potential deployment users to better understand the applicability of the CAD-TMC concept to their sites.
Identify key performance measures that need to be met by similar deployments to achieve the system impacts observed by the FOT deployment. This will help other deployment users identify and focus on the performance goals needed to achieve similar results. Also, document the design basis for these performance measures to help other deployment users adjust these measures to better suit their local conditions.
Calculate and document the key performance measures for the system as it was deployed. This will help identify limitations in the deployed system that might affect the observed system impacts. Also, identify and document other performance measures that are gathered by the deployment team (e.g., during component and integration testing). While this data is not as critical to the evaluation as the key measures, the data should be available from the deployment team to reduce the cost associated with reporting the data.
Identify other factors that affect the deployed system's performance. After the system is deployed, users may identify other factors that could make the system more useful and knowledge that could benefit others in developing similar systems.

In addition to these activities related to evaluating the performance of the deployed system, the system performance study was intended to:

Evaluate the degree to which ITS standards such as IEEE 1512 and NTCIP were incorporated into the deployed system.
Address the approach used to share data between map databases from different vendors and GIS standards that were applied.

3.2.2. System Impact Study

System impacts were evaluated using elements of the framework provided by the ITS JPO's National ITS Program Goal Areas: Mobility; Capacity/Throughput; Productivity; Safety; and Customer Satisfaction.¹¹ The evaluation sought to quantify and document the benefits across these measurable areas for two very broadly defined beneficiary groups: incident responders and travelers. The system impact study was designed to:

Determine if the CAD-TMC integration improves the efficiency and productivity of incident response.
Determine if the CAD-TMC integration improves mobility and reduces delays during incidents.
Determine if CAD-TMC integration enhances incident-specific traffic management plans.
Determine if the CAD-TMC integration will reduce exposure of response personnel and secondary crashes during incident response activities.
Determine if CAD-TMC integration will improve incident management information available to travelers.

3.2.3. Institutional Challenges Assessment

The institutional challenges were identified and documented primarily through stakeholder interviews. Interviews with project stakeholders provided the primary information source for identifying challenges and the processes by which they were resolved. These interviews were conducted on a before and after deployment basis.

The institutional challenges study was intended to:

Document inter-agency cooperation at the State level, in particular, the processes used for identifying and solving problems.
Assess how county and municipal agencies are integrated into the program (Skagit County EMS).
Identify what information is shared, and how the agencies determined that this was the right information to share.
Document how WSP and WSDOT determined what the information availability would be for exchanges between the CAD-TMC systems.
Document how frequently the information provided through the project is used by:
- Responders.
- Travelers.
- Media.
Document how these end-users used the information provided, and identify how the information was used.
Determine if end-users found the information useful and why or why not.
Assess how the various CAD vendors were able to establish working relationships and share data.

3.2.4. Technical Challenges Assessment

The technical challenges assessment documented how the FOT teams addressed technical challenges such as overcoming the barriers associated with incompatible and/or proprietary systems. In conducting the assessment, the Evaluation Team primarily relied on interviews with technical staff at each participating agency to identify the specific challenges addressed and evaluate how those challenges were resolved. Results from this assessment are presented in section 6.

3.2.5. Lessons Learned Assessment

The lessons learned assessment summarized lessons learned during the other portions of this evaluation. The Evaluation Team also explicitly requested information on lessons learned during interviews and meetings associated with the evaluation. Results from this assessment are presented in section 6.

3.3 DATA COLLECTION

Both qualitative and quantitative data were collected for the before (baseline) and after the FOT deployment. The collection of before data focused on establishing a baseline that was used to measure the impact of the FOT deployment. Collection of after data provided data that was compared to the baseline data to determine the impact of the FOT deployment.

Qualitative data collection was conducted for both the before and after data collection phases using the following methods:

Stakeholder/Vendor Interviews. The Evaluation Team interviewed stakeholders/ vendors in person or via phone as the primary means to collect the qualitative information/data needed to successfully perform the CAD-TMC integration FOT evaluation. Stakeholder interviews also were used as a means of identifying issues relevant to the CAD-TMC evaluation. Stakeholder agencies interviewed included WSDOT and WSP.
Site Visits. The Evaluation Team conducted periodic site visits with appropriate stakeholders/vendors to collect needed data not easily transmitted via phone, e-mail or other convenient means.
Observations. Visual observations were used as a means of collecting data that is not otherwise documented or easily conveyed. An example of this included documenting the activities of CAD and TMC operators before and after the new system was deployed to identify any changes in day-to-day procedures or work requirements.

Quantitative data were obtained for the periods of April through June 2004 (before) and for April through June 2005 (after). The qualitative data collected were used to gain user impressions of system performance and impacts, and to identify institutional/technical challenges and lessons learned. Quantitative data were used to assess system performance and system impact.

Sources for quantitative data collected through this FOT are presented by agency, as listed below. Under each data source are the specific field headings from which data were pulled.

WSDOT:
- Condition Acquisition and Reporting System (CARS) message logs:
  1. ROUTE
  2. MILEPOST FROM
  3. MILEPOST TO
  4. DATE REPORTED
  5. START DATE
  6. END DATE
  7. INCIDENT TYPE
  8. INCIDENT DESCRIPTION
- Incident Response Database:
  1. ACTION TAKEN
  2. AGENCIES INVOLVED
  3. CLOSURE REASON
  4. DATE OF INCIDENT
  5. DATE OF REPORT
  6. INCIDENT DESCRIPTION
  7. LANES OPEN
  8. MILE POST
  9. RESPONSE TIME
  10. CLEARANCE TIME
  11. START OF INCIDENT
  12. TIME ARRIVED
  13. TIME CLEARED
  14. TIME NOTIFIED
- Revenue Science Web Page Usage Reports:
  1. WSDOT All Web site Pages (Total)
  2. WSDOT Traveler Information Pages (Total)

WSP:
- CAD system message logs ("A" records):
  1. CREATE TIME INCIDENT
  2. EVENT NUMBER
  3. CLOSED TIME INCIDENT
  4. INCIDENT TYPE ID
  5. ADDRESS
  6. MAP X
  7. MAP Y
  8. PRIMARY UNIT ID
  9. ARRIVAL TO CLOSE
  10. CREATE TO ARRIVAL
  11. CALL SOURCE ID
  12. ARRIVAL TIME
  13. INCIDENT TYPE DESCRIPTION
- CAD system message logs ("C" records):
  1. EVENT NUMBER
  2. CHILD EVENT NUMBER
  3. ASSOCIATED INCIDENT TIME

Qualitative data were collected through interviews with and observations of the following agencies:

WSDOT. Before and after interviews were conducted with WSDOT TMC and Information Technology (IT) personnel in August 2004 and October 2005. Before and after interviews were conducted with system development personnel in August 2004 and November 2005. Team members also observed the operation at the TMC in August 2004 and October 2005. Various field observations and interviews with field personnel occurred at different times during the evaluation periods.
WSP. Input was received from key field and dispatch personnel through interviews and meetings during the evaluation period.

3.4 INTERIM PROJECT REVIEW

An Interim Project Review (IPR) of the FOT was held in January 2005. Participants included the ITS JPO, the Washington State Project Manager, representatives from other stakeholder agencies, the system integrator, and the Evaluation Team. The purpose of the IPR was to:

Provide the project team with a status report on evaluation activities, in particular the status of baseline data collection.
Obtain an update on the status of project implementation.
Discuss next steps:
- When to collect after project data.
- When to complete evaluation activities.
- Assess potential benefits of expanding the scope of the evaluation.

No significant developments beyond the original scope of the evaluation were identified during the IPR. A decision was reached by the meeting participants that the evaluation be completed within the existing scope and schedule.

¹⁰ ITS JPO ITS Public Safety Program brochure "DOT Projects in Utah, Washington State Will Demonstrate Public Safety, Transportation Integration System."

¹¹ Additional information regarding the ITS Evaluation Guidelines - ITS Evaluation Resource Guide can be accessed from the ITS JPO Web site at: <http://www.its.dot.gov/EVAL/eguide_resguide.htm>.

Previous | Next