GAO report criticizes CSA scoring, calls for changes

Updated Feb 5, 2014
GAO CSA Unsafe Driving BASIC
Figure 2 shows the fall in carrier Unsafe Driving violation rates as the number of inspections increases.

The DOT system used to score and compare motor carrier safety records is flawed and particularly unfair to small carriers, an independent government watchdog agency has concluded. While trucking interests have refrained from shouting “we told you so,” the industry has praised the study.

Among carriers that received an SMS score in Unsafe Driving, carriers with fewer than 20 vehicles are more than 3 times as likely to be identified as above the intervention threshold than carriers with 20 or more vehicles. (Click the image for five key GAO report slides.)Among carriers that received an SMS score in Unsafe Driving, carriers with fewer than 20 vehicles are more than 3 times as likely to be identified as above the intervention threshold than carriers with 20 or more vehicles. (Click the image for five key GAO report slides.)

The U.S. Government Accountability Office on Monday published the report, “Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers,” an analysis that supports many of the essential objections cited by the program’s critics.

The report recommends the Federal Motor Carrier Safety Administration revise the system after conducting a “formal analysis” that specifically identifies both the limitations in the data used to calculate carrier scores and the limitations in the resulting scores, including their reliability for their intended purposes – especially significant, given an upcoming rulemaking that would allow a carrier to be declared unfit to operate based solely on the Safety Management System data results.

CCJ uncovered last spring in an in-depth report on the data from the first two years of CSA’s implementation the same findings: The overall data quality for CSA is subpar and small carriers are basically discriminated against in the system. CCJ also reported in its CSA’s Data Trail series the lack of correlation between enforcement patterns and accident rates. Click here to see CCJ’s coverage.

The CSA system and its risk-based approach nonetheless “holds promise,” GAO notes in its report.

“Assessing risk for a diverse population of motor carriers – many of which are small and inspected infrequently – presents several significant challenges for FMCSA,” GAO says. “As a result, the precision and confidence of many SMS scores is limited, a limitation that raises questions about whether SMS is effectively identifying carriers at highest risk for crashing in the future.” 

Among the shortcomings, the GAO study determines that:

• Data quality is inconsistent as states vary on inspection and enforcement practices, as well as timely crash reporting.

• Most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Indeed, 593 of the approximately 750 regulations GAO examined were violated by less than 1 percent of carriers; only 13 consistently had an association with crash risk in at least half of the GAO tests; and only two violations had sufficient data to consistently establish a “substantial and statistically reliable” relationship with crash risk across all tests.

• Data elements used to calculate violation rates for the Unsafe Driving BASIC and Crash Indicator are based on information that is self reported by the carrier. Inaccurate, missing, or misleading reports by a carrier could directly influence their SMS scores.

• The “data sufficiency” standard used by the FMCSA to determine when a carrier qualifies for a score is too low, and small carrier scores are more likely to be inflated by the methodology and prone to significant fluctuations. Similarly, a small carrier whose latest inspection report both qualifies the carrier for a score and elevates that score above a BASIC threshold will have a “very difficult time” improving its score to get below the threshold.

• While SMS is successful at identifying a group of high risk carriers that have a higher group crash rate than the average, further analysis shows that a majority of these high risk carriers did not crash at all, meaning that a minority of carriers in this group were responsible for all the crashes. “SMS is intended to provide a safety measure for individual carriers, and FMCSA has not demonstrated relationships between groups of violations and the risk that an individual motor carrier will crash,” GAO says.

Finally, GAO developed and tested an alternative method that sets a single data sufficiency standard, based on the relevant measure of exposure – either at least 20 inspections or at least 20 vehicles (depending on the BASIC) – and eliminates the use of safety event groups.

For five out of six BASICs, the Crash Indicator and the high-risk designation, the alternative system identified a higher percentage of individual carriers above the intervention threshold that actually crashed compared with FMCSA’s existing method. (FMCSA’s broader net does, however, capture a slightly higher number of carriers responsible for crashes, 85.9 percent versus 78.2 percent.)

“Overall, these results raise concerns about the effectiveness of the existing SMS as a tool to help FMCSA prioritize intervention resources to most effectively reduce crashes,” GAO says.

In its formal response to a draft report, FMCSA “expressed what it described as significant and substantive disagreements with some aspects of our analysis and conclusions,” GAO notes.

FMCSA objected to the suggestion of specific changes to its SMS methodology, so GAO, in the final report, revises its recommendation and calls for the formal analysis “to inform potential changes to the SMS methodology.”

In an email follow-up with CCJ on Monday, FMCSA Director of Communications Marissa Padilla notes that under the Obama administration, the agency has more than tripled the number of unsafe bus and truck companies it has taken off the road.

“While we are always looking for ways to improve our safety oversight methods, and will carefully consider the GAO’s latest recommendations, research shows that CSA is already more effective at identifying motor carriers with a greater risk of crashing than the system we replaced in 2010,” Padilla says, referring to SafeStat.

Still, leaders at American Trucking Associations praised the GAO review as “comprehensive, thoughtful and balanced,” and called on the agency to make immediate changes to the program.

“While ATA has long supported CSA’s objectives, we can’t help but agree with GAO’s findings that the scores produced by the program don’t present an accurate or precise assessment of the safety of many carriers,” says ATA President and CEO Bill Graves.

ATA also calls for FMCSA to remove all carrier scores from public view.

“Since scores are so often unreliable, third parties are prone to making erroneous judgments based on inaccurate data, an inequity that can only be solved in the near term by removing the scores from public view,” says Dave Osiecki, ATA executive vice president and chief of national advocacy.

Likewise, until the GAO findings are addressed, it “would clearly be improper” for FMCSA to proceed with the plan to use CSA data as a basis for its Safety Fitness Determination rulemaking, ATA says.

An industry group that has sued FMCSA in an effort to remove carrier scores from public view also welcomed the GAO report, and calls the recommendations “strong.”

“SMS’s bias against smaller carriers is well known,” says Tom Sanderson, point man for the Alliance for Safe, Efficient and Competitive Truck Transportation. “The GAO study is emphatic in making this point.”

While agreeing with the recommendations, ASECTT “continues to also believe that until the system has been fully validated for individual carriers, SMS data should not be presented to the shipping public and the FMCSA should live up to its federal duty to determine which carriers are safe to operate on the highways and shut down the unsafe operators.”

ASECTT’s full analysis is here.

The DOT’s Office of the Inspector General is also preparing an analysis of CSA.