Mn/DOT Driver Acceptance: IVI FOT Evaluation Report

EXECUTIVE SUMMARY

This white paper provides findings from surveys and interviews for the evaluation of driver acceptance as a component of Battelle’s independent evaluation of the Mn/DOT Intelligent Vehicle Initiative (IVI) Field Operational Test (FOT), sponsored by the U.S. Department of Transportation (USDOT). The overall objective of this white paper is to report on the perspective and experiences of drivers and their supervisors regarding the feasibility and benefits of advanced safety systems for specialty vehicles. During the winter months of 2001-2002, Mn/DOT tested technologies designed to provide operators of snowplows, ambulances, and a state patrol car a means to maintain desired lane position and avoid collisions with obstacles during periods of low visibility. The technologies that constitute the Intelligent Vehicle Safety Systems (IVSS) include the side- and forward-looking radars (and associated collision warning system); head-up display; a GPS-based lane departure warning system that included a visual, audible, and haptic alarm; and magnetic lateral guidance that was a backup to the primary, GIS-based lane-keeping system that was activated only upon the deterioration or loss of the GPS signal.

The evaluation of driver acceptance addressed elements of the following four evaluation objectives associated with Goal Area 2: Assess Impacts on Driver Acceptance, as presented in the Mn/DOT IVI Evaluation Plan:

Objective 2.1. Determine the vehicle operator perceptions of the usability of the IVSS technologies.

Objective 2.2. Determine perceived effects of the IVSS technologies on operator training requirements, job satisfaction, stress, workload, and fatigue.

Objective 2.3. Determine perceived effects of the IVSS technologies on the driver in terms of behavior risk modifications and changes in driver vigilance.

Objective 2.4. Determine perceptions of product quality, value and maturity and establish customer willingness to pay.

Findings and conclusions from this evaluation of driver acceptance must be interpreted in light of both unusually mild winter weather that afforded very few low-visibility driving “events” for which the IVSS was primarily designed, and technologies that were not always functioning to specification. These issues affected driver acceptance and are discussed in this report.

Data Collection Procedures

Members of the Battelle evaluation team met with many of the specialty vehicle operators in the two group-training sessions held in late 2001 and outlined plans for the evaluation. Initial baseline interviews were conducted with 12 drivers and 4 supervisors in December 2001, followed by the first Internet survey of 18 drivers in January 2002. The objective was to obtain background information on the drivers and to assess their expectations for the performance and likely benefits of IVSS in their specialty vehicles. In April 2002, after about three months of driving using the new technologies, a second Internet survey (13 drivers) and in-person interviews (12 drivers and 3 supervisors) were conducted to evaluate their experiences with the technologies and any changes in perceptions. Findings from both the surveys and interviews are integrated in this report to give an overall picture of drivers’ and supervisors’ perspectives on the technologies and their experiences with them. In addition, results from the analysis of the three different driver groups are aggregated where they are similar and discussed separately where they are significantly different. To maintain confidentiality, individual driver identity is not revealed.

Analysis and Findings

Background. Several background questions were asked of the drivers to gain a better understanding of what their thoughts and perceptions of these IVI safety technologies were before they had any significant contact with them. The drivers[1] selected for this FOT were very experienced, with reported experience ranging from 10 to 36 years. Only 11% of the drivers reported they never had taken evasive maneuvers such as hard braking or sudden lane changes to avoid an accident. Therefore, we would expect that these technologies would be viewed as beneficial, and should help reduce the need for sudden evasive driving maneuvers, especially under low visibility conditions. Before the start of this evaluation, drivers were aware of problems[2] with the performance of the technologies. Nevertheless, 83% of the drivers said they expected collision avoidance[3] and lane-keeping would likely be useful to them in their driving. This indicated that drivers were willing to give the technologies a fair test and were hopeful they would experience their intended benefits.

Driver Perceptions of Usefulness. Even though the kind of low visibility weather conditions (e.g., blowing snow, heavy fog) under which these technologies were designed to be used were rare during the evaluation period, and notwithstanding technical problems with the IVSS, the drivers tried out all aspects of the system under actual operating conditions, including several low visibility conditions. Insights into their perceptions are based both on responses to the Internet survey and in-person interviews. Although there were variations in the drivers’ initial perceptions of the benefits of the collision avoidance and lane-keeping systems, some drivers tended to be skeptical of these benefits after having actual driving experience with systems in which the problems had not been fully resolved. This reaction should be expected. Moreover, some thought these systems could interfere with driving tasks and impact driving workload. Comparative survey results are highlighted in Table ES-1.

In general we see at the end of the evaluation period (second survey) the drivers reported reduced agreement with the potential benefits (collision avoidance and stress/fatigue reduction) of the systems and greater concerns about the technology interference with driving tasks and increased distraction and effort associated with the use of the IVSS technologies, compared with their expectations entering into this evaluation (first survey). Also as a generalization, drivers are somewhat more positive regarding the benefits associated with lane-keeping versus collision avoidance. Although less than half the drivers said they wanted either of these systems to be kept on their vehicles in the future, more expressed a desire to keep lane-keeping compared with collision avoidance. Many of the drivers found that the IVI systems were helpful in snowy and low visibility conditions when they were working properly, but they were concerned about having them work consistently and reliably.

In the in-person interviews drivers reported on the specific aspects of the systems that they liked or that were frustrating to them. The snowplow operators liked knowing where they were positioned with respect to the roadside when snow covered the road. Drivers in general thought the IVSS offered great potential in increased confidence and reduced stress of driving in low visibility conditions, though few expected measurable productivity benefits. Some drivers discussed such issues as glare and reflections off the head-up display (HUD) combiner, vibrations and lack of clarity in seeing road detail using the HUD, problems with night vision, and apparent false readings presented by the collision avoidance system. These kinds of issues caused some drivers to stop using the systems, or to only use them for testing purposes under good driving conditions.

When asked whether they would be better off driving without these types of high technology systems in their vehicles, 39% disagreed in both the first and second surveys and less than 10% agreed. Over half the respondents in each survey were undecided on this question, which suggests their abiding willingness to give the technologies a chance to prove themselves. Many felt, because of weather and technical constraints, that they hadn’t had adequate opportunity to experience the real benefits of these technologies, and they looked forward to that opportunity, once the bugs could be worked out.

Driving Behavior. Half of the respondents on the initial survey said they expected their driving would change as a result of having both the collision avoidance and lane-keeping systems on their vehicle. Though they weren’t asked to say specifically how their driving might change, this response indicates there is clear potential for underlying changes in driving habits. On the second survey, 46% of the drivers said that their driving had not changed as a result of the collision avoidance system, and 70% of the drivers said it had not changed as a result of the lane-keeping system. Drivers interviewed in-person mentioned they thought these safety technologies would make them more alert, more relaxed, and probably more careful about safely managing such driving tasks as following distance. This suggests they didn’t think they might engage in more risky driving behaviors under the assumption that the technologies would keep them out of trouble.

Perceived Mental Workload. Mental workload refers to the amount of mental effort, concentration, or focus that drivers think it takes to perform their driving tasks. There was general consensus among these drivers that the level of workload is quite high when operating their vehicles under the worst winter driving conditions without any IVSS technologies, and that this workload level is reduced by the IVSS technologies. However, the average reduction in workload actually experienced by these drivers (second survey) was about half as much as they expected (first survey). A few drivers reported an increase in workload and others reported no reduction in workload at all due to the IVSS, but on average drivers indicate that the level of mental workload is reduced somewhat by these systems.

Perceived Liability. Citing liability concerns, ambulance operators said they were reluctant to use the technologies when a patient’s life was at risk in an emergency driving situation. Both the ambulance and snowplow driver groups were split 50-50 on their perception of the overall safety benefits of the IVSS, with 38% agreeing that they provided a safety benefit, and 38% disagreeing overall, with no clear differences between the two driver groups.

Supervisors’ Perspective on the IVSS. Overall, supervisors would like to see these kinds of systems on their vehicles if their reliability increases and if the costs can come down. They all thought the systems’ test should run another year to gain greater exposure to the kinds of conditions in which they are designed to help drivers.

Conclusions

This evaluation of driver acceptance was hampered by both the lack of low visibility weather “events” and the initial performance problems experienced with some of the IVI technology systems in each of the three specialty vehicle categories. Because of these factors, driver perceptions measured by these surveys and interviews appear to have reflected the particular circumstances of the test along with the actual functionality and safety benefits they were able to experience from the technologies. Nevertheless, drivers and supervisors remained generally optimistic that, if the technology problems can be resolved, the IVSS technologies hold significant potential to enhance driver confidence and performance while operating specialty vehicles under very difficult driving conditions. The participants in this test agreed that the technical problems with the IVSS needed to be fixed and more evaluation time under adverse weather conditions was needed to confirm and quantify benefits.

1.0 INTRODUCTION

1.1 Background on IVI Field Operational Tests

The United States Department of Transportation (USDOT) established the Intelligent Vehicle Initiative (IVI) as a major component of the Intelligent Transportation System (ITS) program. The intent of the IVI is to improve the safety and efficiency of motor vehicle operations significantly by reducing the probability of motor vehicle crashes. These safety improvements could also show secondary benefits such as increased transportation mobility, productivity, or other operational improvements.

In 1999, USDOT entered into cooperative agreements with four partnerships to conduct Generation 0 Field Operational Tests (FOTs) of advanced intelligent vehicle safety systems (IVSS). These systems are expected to begin production preparations before the end of fiscal year 2003. Although the scope of the IVI Generation 0 FOT program included light passenger vehicles and transit vehicles, USDOT selected one FOT involving specialty vehicles and three FOTs involving commercial trucks. The Minnesota Department of Transportation (Mn/DOT). was selected to conduct the specialty vehicle FOT. The USDOT selected a Battelle-led team to work with each partner to perform an independent evaluation of the technologies being tested.

Mn/DOT deployed IVSS technologies designed to provide operators of snowplows, ambulances, and state patrol cars drivers a means to maintain desired lane position and avoid collisions with obstacles during periods of low visibility. Key among these technologies was vision enhancement, lateral guidance, and collision warning systems. The primary evaluation goal of the FOT was to determine the potential safety benefits of IVSS. Specifically, how many crashes, injuries, and fatalities could be avoided if all such vehicles were equipped with these technologies? It was also important to understand how these technologies affected driver performance. For example, did drivers drive more safely? And, how did these technologies affect driver stress level and workload? The secondary goals of these evaluations included the estimation of other benefits (mobility, efficiency, productivity, and environmental quality), evaluation of system performance, and assessments of other factors that affect development and deployment of these technologies. These factors included user acceptance, product maturity, manufacturability, and institutional and legal issues.

These were the original goals of the Mn/DOT FOT. However, the IVSS being tested in the Mn/DOT FOT are designed for use in snow accompanied by low visibility conditions. Thus, such conditions were necessary to achieve the goals and objectives of the Mn/DOT FOT as well as its independent evaluation by Battelle. However, the winter of 2001-2002 in the area of the test corridor turned out to be unusually warm and relatively devoid of snow. During the period of the FOT, December 21, 2001 – March 31, 2002, there were only two snowfalls of significance. Furthermore, according to measurements made by the Mn/DOT FOT partnership, there was no occasion during the FOT in which the visibility was very low (defined as less than 100 meters) and there were only 15 minutes when visibility was in the 100 to 199 meter range. Thus, in the words of Mn/DOT project management: “At no time during the FOT were any of the specialty (vehicle) operators exposed for sustained periods to the kind of conditions for which the DAS (Driver Assistive System) was designed.”

Recognizing that the original evaluation objectives could not be met due to the mild winter weather, FHWA modified the evaluation Statement of Work to direct that “Battelle will prepare a Project Report that describes each system tested and details the evaluation plan and test plans developed.” They also directed Battelle to prepare this white paper on driver acceptance based on feedback received during driver interviews and surveys.

1.2 The Mn/DOT IVI FOT

The Mn/DOT IVI FOT was conducted by a partnership including state and local government, industry, and the University of Minnesota. Table 1 lists the partnership organizations and their roles. URS/BRW Provided administrative and program management support to Mn/DOT on the project.

Table 1. Roles of the Mn/DOT FOT Partners

ORGANIZATION	ROLE
Mn/DOT Office of Advanced Transportation Systems (OATS)	Overall project manager as caretaker of Minnesota Guidestar Program. Facilitated contracts preparation and approval.
University of Minnesota (Intelligent Vehicle Laboratory, Human Factors Research Laboratory, and the Department of Applied Economics)	Technical lead & system integrator. Human factors support & evaluation. Benefit-cost analysis.
Mn/DOT - District 8	Provided 2 snowplows with operators. Resident district for magnetic tape installation. Provided office space in Hutchinson Area Transportation Systems (HATS) building.
Mn/DOT - Metro Division	Provided 1 snowplow with operators.
Minnesota State Patrol	Provided 1 state patrol car with operator.
McLeod County	Provided 1 snowplow with operators.
Hutchinson Health Care	Provided 1 ambulance with operators.
3M Corporation’s ITS Project Office	Provided magnetic lateral guidance tape and sensor technologies.
Altra Technologies, Inc. (ATI)	Provided side-looking radar system.

The Mn/DOT IVI effort was focused on improving mobility and reducing the number and severity of specialty vehicle (especially snowplow) crashes with other vehicles and roadside equipment such as guardrails and traffic control devices. Such crashes sometimes occur under low-visibility conditions caused by fog, rain, blinding snow, and darkness. Specific goals of the FOT included:

· Reducing the number and severity of specialty vehicle collisions as well as rear-end collisions involving the public’s vehicles hitting the backs of snowplows,

· Improving the productivity and efficiency of snowplow and emergency vehicle operations, and

· Successfully integrating systems and technologies tested in earlier Mn/DOT projects.

Overall, the Mn/DOT FOT proposed to build upon and to extend several ITS technologies investigated in past and ongoing efforts in the state of Minnesota. The purpose of the FOT was to establish safety benefits. The IVSS were focused on providing specialty vehicle drivers with assistance during low-visibility conditions. In the FOT there were four snowplows, one state patrol automobile, and one ambulance equipped with the technologies, as well as an infrastructure to support them. A number of distinct yet related systems were integrated into the IVSS using on-board processing.

The FOT was conducted from December 22, 2001 to March 31, 2002. During the FOT, the test vehicles operated on their usual state and county highway routes. The primary test road for the FOT was a 45-mile section of Minnesota Trunk Highway 7 (TH-7) that runs east-west between the I-494 beltway in Minnetonka (a community on the western side of Minneapolis) and the City of Hutchinson. There was also a 4-mile section of McLeod County Road 7 extending northeast from Hutchinson that was included in the FOT.

1.3 Organization of This Document

This paper is divided into four sections. Section 2 describes the approach that we took to evaluating driver acceptance and discusses the data collection procedures. Section 3 presents the analysis of the data collected and highlights findings from the evaluation as they relate to each of the outcomes discussed in Section 2. Section 4 provides the evaluator’s conclusions on what we learned from drivers and supervisors concerning their acceptance of the IVSS.

2.0 APPROACH

2.1 Evaluation Goals and Driver Acceptance Objectives

The U.S. DOT originally suggested five goal areas:

Goal 1: Achieve an in-depth understanding of the benefits of IVI technologies

Goal 2: Assess driver acceptance of IVSS

Goal 3: Assess system performance

Goal 4: Assess product maturity for deployment

Goal 5: Address institutional and legal issues that might impact deployment

Because the benefits of the IVI technologies fall into five different categories (safety, mobility, efficiency, productivity, and environment), Goal 1 was divided into five separate sub-goals, corresponding to benefit categories. As noted in 1.1 above, weather conditions limited the opportunities to evaluate the IVSS technologies under the conditions for which they were designed (i.e., limited visibility). As a result, FHWA determined that Battelle’s efforts should focus on two activities: (1) documenting the evaluation methods and lessons learned, and (2) performing an assessment of the drivers’ acceptance of IVSS. This report presents findings related to the second goal. The methodologies and partial analyses completed for the original goals are available in a separate document (report in preparation on the Mn/DOT Intelligent Vehicle Initiative Field Operational Test Evaluation Methods and Lessons Learned, Battelle, 2003).

The evaluation of driver acceptance addressed elements of the following four evaluation objectives associated with Goal 2: Assess Impacts on Driver Acceptance, as presented in the Mn/DOT IVI Evaluation Plan (Battelle Memorial Institute, 2001, p. 38):

Objective 2.1. Determine the vehicle operator perceptions of the usability of the IVSS technologies.

Objective 2.2. Determine perceived effects of the IVSS technologies on operator training requirements, job satisfaction, stress, workload, and fatigue.

Objective 2.3. Determine perceived effects of the IVSS technologies on the driver in terms of behavior risk modifications and changes in driver vigilance.

Objective 2.4. Determine perceptions of product quality, value and maturity and establish customer[4] willingness to pay.

Additional information on relevant driving experience and experience with computers and other “high tech” vehicle control or information systems was obtained in order to explore background factors and driver characteristics that might help explain the degree of observed driver acceptance.

2.2 Overview of Approach

Evaluation methods included in-person interviews with drivers and their supervisors and Internet-based surveys of the drivers. These were used to gather baseline information before the drivers had significant experience with the new IVSS technologies and later after they had experience with the technologies under the winter conditions for which they were designed. These data collection procedures are discussed in more detail in Section 2.

An objective of the initial baseline Internet survey (18 drivers) was to assess driver expectations for the use of the safety technologies and to ask drivers about their experiences with early versions of the technologies. It was known at the outset that there had been significant technical problems with the performance of the GPS in particular that resulted in incorrect or unusable displays of roadway information that could not be corrected prior to the start of the Field Operational Test (FOT). The final Internet survey (13 drivers) sought to identify changes in driver perceptions based on their experiences with the IVSS. The baseline driver interviews (12 drivers) and final interviews (12 drivers) supplement the objective data collected in the surveys with a more open-ended, subjective discussion of expectations, experiences, and issues with the technologies. In addition to the data collected from the drivers, baseline and final interviews were conducted with selected supervisors (4 in the first interview and 3 in the second interview) in order to obtain their perspective on these safety systems. Findings from all these data are integrated in this report.[5]

As we learned from the driver and supervisor in-person interviews conducted in December 2001, early problems with the technologies appeared to cause some drivers to have reduced expectations regarding the potential to experience benefits from these systems at the outset of the evaluation. The final Internet survey and interviews sought to evaluate whether and how driver attitudes, perceptions, and behaviors with regard to each of the IVSS technologies changed as the drivers gained experience using the technologies and to the extent that the bugs were worked out.

Three other factors are known to have had an impact on driver responses and observations obtained from the surveys and interviews.

1. Drivers who participated in the surveys and interviews were operating very different vehicles under different conditions, associated both with the vehicle type and with the geographic areas in which they operate. For example, snowplows operating in more rural environments encounter very different driving conditions from snowplows operating closer to the city, in “urban corridors,” and snowplows may operate very differently and under different conditions from ambulances or state patrol cars. Notwithstanding these differences, however, there was substantial agreement among the drivers on many of the topics covered in this evaluation. Where significant differences occurred in driver responses, these are discussed separately.

2. As was true for the entire IVI evaluation, the generally mild weather conditions that occurred between the baseline data collection and the final data collection approximately three months later significantly limited driver opportunity to experience the use and benefits of these safety systems. The evaluation timeframe provided at most two short instances of the kind of low visibility driving conditions that were considered essential to test the merits of the systems and offer the drivers sufficient opportunity to arrive at their sense of IVSS utility and potential benefit.

3. The technologies themselves were not fully debugged by the end of the second survey. This meant that the drivers were not able to report on a set of technologies that were performing up to their design specifications.

Data collected at the initial and final time points allow for descriptive analysis of data on driver expectations, perceptions, and experiences at those time points, and also allow for a comparative assessment of any changes in responses and perceptions over the time period covered by this evaluation. Data from the same or similar questions asked at both points in time are analyzed to determine any changes in perception over time. Changes in perceptions are examined for groups of drivers (group averages for example) and at the individual level for the ten drivers who participated in both the first and second Internet surveys (to examine any changes in responses by the same person at both time points). In addition, where possible, comparisons between the survey responses and the objective systems data are provided as a way to discern how accurately drivers monitor their behavior and the accuracy of their perceptions of the system’s performance.

This evaluation was conducted in parallel with a similar but independent evaluation conducted by the University of Minnesota (2002). Evaluators from both teams met periodically to discuss and coordinate plans for surveying and interviewing drivers, both to enhance the quality and comparability of the two evaluations, and to minimize the burden on the drivers to meet with the evaluators and respond to questions.

2.3 Conceptual Model

Figure 1 presents a conceptual model that illustrates sets of factors expected to influence how specialty vehicle drivers might be affected by the IVSS technologies. These factors were examined in the driver and supervisor surveys and interviews, and they include driver background, driver expectations about the IVSS, external conditions affecting the use of IVSS, and how these interact to influence driver perceptions and experiences with the IVSS. The first step (Baseline Perspective) is to take account of pre-existing experience and perspectives that can directly impact the outcomes of interest, as well as influence these outcomes through their indirect effects on driver expectations about the new technologies, their experiences and reactions while using the technologies, and their attitudes towards the technologies’ benefits. These baseline conditions include training, driving experience, level of comfort with any kind of new technology, and the extent to which their organization and fellow drivers support or criticize the technologies.

Taken together, these conditions and factors directly affect the likelihood there will be driver trust that the technologies even have the potential to offer benefits. Another key set of conditions affecting the outcomes include whether the technologies work as they are supposed to and whether the external driving conditions and environment are conducive to a successful outcome. In the case of this evaluation, we know that some aspects of the technologies were not functioning correctly, or at all, and we also know that the needed low visibility weather conditions that were critical for an adequate test of the intended use of the technologies were almost non-existent during the evaluation period. In spite of these problems, most of the drivers were willing to put the technologies to the test where they could, and they were quite willing to share their experiences and opinions with us.

Factors Affecting Driver Acceptance of IVSS Technology

Figure 1. Factors Affecting Driver Acceptance of IVSS Technology

2.4 Data Collection Procedures

Several alternative strategies for collecting data from the drivers were considered, including written surveys, telephone and in-person interviews, and Internet surveys. In-person interviews were implemented because they provided a means of gathering attitudes, opinions, and anecdotal information not easily gathered by other instruments. We selected the Internet approach for the surveys because we felt this would be of interest to the drivers and would motivate them to complete the survey, as well as provide a manageable approach with the expectation of a high participation rate. The purposes of these interviews and surveys are shown in Table 2 in the order in which they were administered.

Table 2. Mn/DOT Interviews and Surveys

Data Collection Method	Dates of Implementation	Purpose
First Interview	Dec. 12 – 13, 2001	Gather baseline driver and supervisor attitudes, perceptions and expectations of the systems.
First Survey	Jan. 7 – 27, 2002	Gather baseline information from the drivers on their experiences with technology and their expectations of the systems.
Second Survey	April 2 – 11, 2002	Gather information after deployment of the IVSS technologies regarding driver uses of these systems, effects on driving behavior, and perceptions of benefits.
Second Interview	April 11, 2002	Gather qualitative information on driver and supervisor acceptance of IVSS, and an understanding of any changes in their attitudes and perceptions.

2.4.1 Interviews

White Paper

Mn/DOT Driver Acceptance: IVI FOT Evaluation Report

Table 2. Mn/DOT Interviews and Surveys

Dates of Implementation

Purpose

Mn/DOT Driver Acceptance:
IVI FOT Evaluation Report