Evaluation Resources

Several resources are available that provide background, context, and guidance to transportation system operators, planners, and policy makers seeking to understand the impacts of automated vehicles, whether in the context of evaluating specific pilot deployments, or to increase their general knowledge of the ways automated vehicles may affect the transportation system.

• Considerations for Evaluating Automated Transit Bus Programs.¹ This guide aims to assist transit stakeholders with designing and implementing evaluations of automated transit-bus programs. In designing evaluations, transit agencies and other stakeholders should identify program goals and audiences affected by the technology; develop a logic model that maps project inputs, activities, and outcomes; choose an appropriate evaluation design; and collect and analyze data on key performance indicators related to their program goals.

• Evaluation Methods and Techniques: Advanced Transportation and Congestion Management Technologies Deployment Program.² This Evaluation Methods and Techniques document assists Advanced Transportation Congestion Management Technologies Deployment (ATCMTD) program grantees in designing and executing robust evaluations that fulfill their reporting requirements. The document offers an overview on evaluation, including best practices related to designing and executing an evaluation. It also discusses methods and analytic techniques, including best practices on benefit-cost analysis, survey and interview methods, and emissions and energy measurement. Additionally, this document provides technology-specific guidance on evaluating Adaptive Signal Control, Connected Vehicles and Automated Vehicles. Finally, it provides grantees with a recommended set of performance measures that address the FAST Act reporting requirements.

• Trilateral Impact Assessment Framework for Automation in Road Transportation.³ Members of the Trilateral Working Group on Automation in Road Transportation (ART WG)⁴ developed this framework.  It is meant for policy makers to support policy analysis and long‐range scenario‐based planning, for automakers and after‐market equipment manufacturers to better understand the potential benefits of their offerings, for designers of field tests to ensure that the information gathered maximizes the value of the test, and for those making the impact assessment as a starting point in design of their evaluation work. The framework provides recommendation for classifying automation implementations and determining impact areas to be assessed. It presents the impact mechanisms through which automated driving is expected to impact our life, covering both direct and indirect impacts. In addition, it provides recommendations for experimental procedure and data sharing.  

Smith, S., Innamaa, S., Barnard, Y., Gellerman, H., Horiguchi, R., & Rakoff, H. (2017, July 19). Where will Automated Vehicles take us? A Framework for Impact Assessment [Poster]. Automated Vehicles Symposium, San Francisco. https://higherlogicdownload.s3.amazonaws.com/AUVSI/14c12c18-fde1-4c1d-8548-035ad166c766/UploadedImages/2017/PDFs/Proceedings/Posters/Wednesday_Poster%202.pdf
Innamaa, S., Smith, S., Barnard, Y., Rainville, L., Rakoff, H., & Horiguchi, R. (2018). Trilateral Impact Assessment Framework for Automation in Road Transportation (p. 42). https://connectedautomateddriving.eu/wp-content/uploads/2018/03/Trilateral_IA_Framework_April2018.pdf

• Key performance indicators for assessing the impacts of automation in road transportation Results of the Trilateral key performance indicator survey.⁵   This report documents a survey, which was designed to investigate views on the importance of different key performance indicator (KPIs) for expressing the impact of automation in road transportation in several impact areas. It documents the rating results and additional KPIs proposed by the 77 experts in Europe, US and Japan who filled in the survey. Indicators assessed include vehicle operations, adoption/use of automated driving, safety, energy/environment, travel behavior, network efficiency, asset management, costs, public health, land use, and economic impacts.

Innamaa, S., & Kuisma, S. (2018). Key performance indicators for assessing the impacts of automation in road transportation: Results of the Trilateral key performance indicator survey (VTT-R-01054-18). https://www.vtt.fi/inf/julkaisut/muut/2018/VTT-R-01054-18.pdf

• FESTA Handbook, Version 7. The FESTA Handbook, now in its 7th edition, is the product of a European Commission support action that compiled methodology and created a handbook to guide upcoming Field Operational Tests (FOTs). https://connectedautomateddriving.eu/publication/festa-handbook-7th-version/