Verification Testing & Independent Measurement System

Outline

• Purpose of NIST IMS
• Phase I Testing Activities
• Using the IMS
  • Example Multiple-Threat Verification Test
  • IMS Description
  • Procedure for analyzing warnings during a test
• IMS forward collision warning range accuracy
• Using IMS data to improve warning system performance
• Questions

Purpose of NIST
Independent Measurement System

• Provide “independent” means to measure test variables to determine if system passes or fails a test
• Measurements:
  • Time of warning in the cab
  • FCW: Range, range-rate and time-to-collision at warning
  • LDW/LCW: Lateral distance and lateral velocity with respect to lane markings
  • Miscellaneous run-validity variables
• Support debugging and system improvements

IMS Testing Activities in 2007-08

• Integration and IMS Acceptance testing
  • Integration and fit testing – mechanical, electrical and data collection
  • IMS Acceptance – demonstrate IMS accuracy to USDOT and UMTRI
• Phase I Verification Testing (September-November 2007)
  • Verify system meets performance requirements
• Phase I Extension Verification Testing (January-March 2008)
  • Follow-up tests to assess system fixes and improvements

IMS Testing Activities in 2007-08

• 22 Separate Test Activities
  • 11 LV, 11 HT
  • 16 test track tests
  • 6 on-road tests
• 3 Track Sites
  • Dana
  • TRC
  • Marshall

Using the NIST IMS

• Describe example test – Multiple Threat Test 1
• Describe components of IMS Sensors and Electronics
• Measuring time of warning
• Measuring range to POV at time of warning
• Measuring lateral distance to lane marking at time of warning

Example Multiple-Threat Track Test

• Subject vehicle driver encounters slower moving POV and receives FCW warning
• SV driver attempts to change lane and receives LCW warning

IMS sensors and FOV

• Dual rotating-laser scanners for measuring range and range-rate to forward and side objects
• Forward and side calibrated cameras for lane position

IMS Sensors and Electronics

• External cameras and laser scanners mounted to vehicle front
• Dash camera and microphones mounted in cab
• Electronics fit in back seat

How are FCW warning ranges determined

• Step 1: During test run
  • Capture video and audio (30 Hz) using DV deck
  • Capture IBEO data with video (10 Hz) using laptop with IBEO capture program.

How are FCW warning ranges determined

• Step 2 – Transfer DV video to disk and analyze video frame by frame to determine first frame warning is heard. The video overlay GPS time stamp is the DV time of warning.

How are FCW warning ranges determined

• Step 3 – Use NIST software to:
  • Correct IBEO time stamp using video-overlay GPS time stamp
  • Select POV range points in laser scan prior to DV warning time and in laser scan after DV warning time. (Recall, DV video is 30 Hz while IBEO scans are 10 Hz.)
  • Calculate warning range using DV warning time to interpolate between laser scans.

How are LCW warning ranges determined

• Step 1 – Align calibration sticks with front wheels
• Step 2 - Identify pixel locations on calibration stick
• Step 3 – Locate warning time video-frame, click on pixel at edge of lane marker, warning range is interpolated distance between calibration marks.

IMS Forward Collision Warning Range Accuracy

• Most challenging accuracy requirement is measuring range to stopped POV at 20 m/s (45 mi/h)
• NIST measured uncertainty of laser scanners: ± 0.85 m (95% confidence) up to 20 m/s
• Three approaches to evaluate uncertainty
• 1 – NIST developed dynamic tests with uncertainty analysis
• 2 – Demonstration tests at Dana track using down looking video and calibration marks on the ground
• 3 – Calibrated forward camera
• IVBSS team agreed to accept the IMS measurements to determine whether the system passed or failed a test!

NIST Dynamic Test

• Ground truth setup:
  • Place reflectors on ground at 20 m intervals from target
  • An optical switch on vehicle generates an event (within 1 ms) when vehicle drives over the reflector
  • GPS unit time stamps the event (within 1 µs).
• Drive over reflectors at various speeds – max was 20 m/s (45 mi/h)
• Use event time to interpolate between laser scans

IMS Acceptance Testing

• UMTRI painted 1 m markings on track. Used IMS video camera to capture range to stopped POV at time of warning.
• For moving POV, used a camcorder in POV to capture range at time the audible warning transmitted from SV to POV via a walkie-talkie.
• Very cumbersome and IMS values appear more accurate

Forward camera ranging

• Calibrate forward camera focal length given known width of vehicle
• Warning ranges matched IMS, big problem is selecting correct pixels, especially when vehicle/camera bounces (see run 5)

Using IMS data to improve warning range accuracy

• Analyze warning range errors (R FCW – R IMS) as a function of range rate (R dot)
• Fit line to errors to model warning range offset and timing delay

Calibrate IVBSS warning system

• Apply calibration correction to data: