Lessons Learned

Lessons Learned From Recent Secure Credential Management Systems (SCMS) Operation with the CV Pilots

The CV Pilot programs have been at the forefront of deploying devices integrated with an SCMS and have generated lessons learned from that early experience. These lessons learned have been captured from experiences with the CV Pilot programs, primarily during calendar year 2020. Although captured from the CV Pilots, these lessons learned are likely applicable to other organizations deploying CV devices.

Background

The SCMS is a public key infrastructure (PKI) system that provides credentials and certificates to CV devices to enable trusted communications. CV devices, such as onboard units (OBU) in vehicles and roadside units (RSU) within the infrastructure will sign their messages with the certificates received from the SCMS. When other devices receive these messages, they can validate those signatures and ensure they are coming from a trusted source so that applications within the device can act on those messages. An OBU will validate messages from other OBUs to ensure they can provide safety of live warnings such as forward collision warnings. RSUs will validate signal request messages from vehicles such as buses to enable transit signal priority applications. The SCMS provides the backend systems for OBUs and RSUs to request and download certificates over the devices lifetimes, as these certificates are typically only valid for one week increments and need to be updated over time. Additional resources on SCMS can be found at: https://www.its.dot.gov/resources/scms.htm

The CV Pilot programs were some of the first deployments to utilize devices fully connected to the SCMS. The goals of this effort were to demonstrate that the key concepts of the SCMS were feasible and that there were SCMS providers capable of meeting the certificate needs of deployed CV devices. Although successful, the integration of the CV Pilots devices with the SCMS have generated several lessons learned for deployment agencies, CV device vendors and SCMS providers. The following are the most recent lessons learned from the CV Pilot deployments.

Lessons Learned

• Issue 1: Onboard Unit (OBU) Top-Off Failures. OBUs are required to request updated certificates (periodically) while driving through the CV environment. An individual certificate is valid for only one week. An OBU may be capable of storing up to three years’ worth of certificates, but depending on the deployment site, may only be able to obtain valid certificates for shorter periods.  Over time, the OBUs will eventually need to reach back to the SCMS to download additional certificates (this activity is known as certificate top-off). Some deployments limit the number of future weeks of certificates, necessitating more frequent top-off requests. A limited number of RSUs were provisioned within the environment to offer SCMS connection capable of supporting the top-off service. This led to situations where many OBUs were trying to download top-off certificates from the same RSU at the same time, and some OBUs would not be able to download all of their top-off certs before exiting the RSUs coverage area. The certificate download would then timeout before the OBU reached another RSU advertising an SCMS connection, creating a certificate download error and failure.
  • Lesson Learned: The deployer worked with the SCMS provider to troubleshoot these issues. The SCMS provider implemented a new auditing tool within the SCMS that logs all interactions between the device and the SCMS with both error codes and timestamps. Deployment agencies should take advantage of this auditing tool to help them troubleshoot download requests if they are having these types of issues within their environment. Additionally, the IEEE 1609 Working Group is developing a more formal technical solution within IEEE 1609.13 “Reliable Data Transport Mechanisms for Multiple Receivers”.  These issues were primarily observed in RSU-sourced download requests, and deployment agencies may want to consider other methods for downloading certificates if they are available.

• Issue 2: Certificate Download Deactivation. A network issue in one deployment site resulted in CV devices not connecting to the SCMS for a few weeks. The SCMS expected devices to connect and request certificates at least once within a two-week period. The failure to do so for this deployment triggered a deactivation feature within the SCMS, and when those devices network connections were restored, the SCMS denied their requests for new certificates.
  • Lesson Learned: The SCMS provider updated the deactivation configuration to permit an OBU to remain dormant for a longer period. Where operationally possible, and where the security risk is low, it is better for devices to download more than a few weeks of certificates at a time.

• Issue 3: RSU Application Certificate Top-off. A likely network issue was causing RSUs to request new application certificates and the SCMS would then generate the new application certificates and post them for the RSU to download. The RSU would fail to receive the initial response from the SCMS and would then request a second set application certificates, which violated an internal SCMS policy that limits the number of certificates an RSU can request in a two-week period.
  • Lesson Learned: The SCMS provider updated their backend system and would just respond with the initial set of application certificates that were generated if the RSU had not downloaded those certificates yet. This was likely a common real-world networking issue that this fix will hopefully resolve moving forward.

• Issue 4: Enrollment Certificate Expiration. A few vendors have requested enrollment certificates with only a 75-day validity period. The vendors will then ship devices, but they will not be installed and operated until the 75-day period has elapsed, which results in all future certificate requests being denied. This currently requires devices to start the enrollment process from the beginning, which can usually only be accomplished by the vendor.
  • Lesson Learned: A vendor provided an example script for creating enrollment requests, which used a 75-day validity period. It is likely vendors are copying this script without changing the validity period included in the example. Vendors should set their validity period to more operationally realistic periods (e.g. 3 years) or support a re-enrollment mechanism. All users should adopt, as a standard practice, reviewing all default settings and adjusting them to reflect their operational policies and expectations. At a higher level, when integrating collaboratively developed capabilities, do not leave default values in place if their effect and purpose are not clearly understood.

• Issue 5: Pseudonym Certificate Authority (PCA) Certificate Validity Issue. Due to decisions made during the original Crash Avoidance Metrics Partnership (CAMP) led SCMS activities, the PCA was designed with a shorter validity period to minimize security risks. The PCA is the certificate authority that generates all of the pseudonym certificates that the OBUs will use to sign their Basic Safety Messages (BSM). The PCA will expire in roughly two and a half years (January 2023). Many OBUs request pseudonym certificates for three full years, causing an issue when requesting three years’ worth of certificates after January 2020.
  • Lesson Learned: The SCMS vendor has developed a patch that will be deployed by the end of July. Currently, the SCMS has been updated to only generate certificates through January 2023. After the new PCA Certificate is in place and the patch is applied, devices will again be able to request three years of certificates. Moving forward the decision for how long the validity periods are for the different certificate authorities will reside with the SCMS Manager and will be subject to vote by the SCMS Manager voting members.

• Issue 6: Local Certificate Chain File (LCCF) Implementation. On August 5, 2020, an SCMS provider created a new Enrollment Certificate Authority (ECA) and PCA due to the situation described in Issue 5. When this was implemented operationally, a new LCCF was provided to all devices that connected to the SCMS that listed the new ECA and PCA as trusted certificate authorities. Most RSUs have two weeks’ worth of certificates and during the week of August 19th, the RSUs started signing their messages with certificates from the new PCA. When this occurred, the OBUs were no longer able to utilize the RSU’s advertised services because they no longer trusted the certificate authority that created the certificates. This was due to issues with the way the OBUs processed the new LCCF file.
  • Lesson Learned: Device vendors and deployment agencies should ensure that the processing of the ancillary files associated with the SCMS, such as the LCCF, are tested fully. It is also recommended that vendors stay up to date on software updates for applications and libraries within their device. In some instances, these LCCF processing issues were already addressed in a previous software update that was not on applied to the effected OBUs.