CSA is ‘unreliable’ and here’s why – and what to tell customers, says ATA

Updated Dec 10, 2013

Trucking continues to question the accuracy and application of data at the heart of the federal government’s regulatory program, and a powerful political voice for the industry has issued a white paper that spells out the points of contention in a way that customers – and maybe even Congress – can understand.Alert

At issue is the statistical correlation between government-assigned trucking company safety scores, based on industry-wide trends, and whether the math holds true for individual carriers.

The American Trucking Associations on Monday issued a white paper on the reliability of scores from the Federal Motor Carrier Safety Administration’s Compliance, Safety, Accountability program in evaluating the safety of individual trucking companies.

“ATA continues to support the objectives of CSA and to call for improvements to the program,” says ATA President and CEO Bill Graves. “However, data and methodology problems continue to plague the system and the accuracy and reliability of companies’ scores.”

More simply, as an ATA representative told CCJ, the organization expects member carriers to share the report with third parties – such as shippers and insurers – who might be using CSA scores to evaluate trucking company safety performance.

In its paper, ATA examined data and research on the connection between CSA scores and crash risk, as well as how problems with the data and methodology produce “an imperfect and unreliable measure” of a carrier’s safety record.

Related: CSA is a ‘whim,’ transport law expert argues

Researchers have arrived at “mixed conclusions” with respect to the reliability of Safety Management System scores in identifying unsafe, or crash-prone, motor carriers. Some found virtually no correlation between scores and crash rates in any of the measurement categories, the report states.

Most significantly, ATA suggests that the high number of “exceptions,” or carriers that don’t fit the statistical model, “leads to the conclusion that SMS scores alone … are, at a minimum, unreliable.”

The American Transportation Research Institute (ATRI), “using a better prediction model,” found a positive relationship between scores and crash risk in three of the publicly available measurement categories (BASICs) but also found that scores in two others bear an inverse relationship to crash risk.

ATRI, the research arm of ATA, also pointed out that the number of alerts that a carrier has been assigned is a strong indicator of crash risk. However, the strength of the relationship varies depending on the BASICs in which the carrier has alerts, since scores in some BASICs more strongly correlate with crash risk than those in others, according to the paper.

“It may make sense for FMCSA to use scores in those categories that correlate positively with crash risk to prioritize companies for enforcement review,” Graves says. “In the process, FMCSA can verify whether or not the scores paint an accurate picture. But third parties need to know that for the purposes of drawing conclusions about individual carriers, the scores are unreliable.”

Related: CSA enforcement disparity, rhyme or reason?

ATA continues to point to a number of “data and methodology problems that plague the system”:

– A substantial lack of data, particularly on small carriers who comprise the bulk of the industry, hinders the system’s ability to render meaningful scores of comparative performance.

– Regional enforcement disparities likely cause fleets of all sizes operating in jurisdictions with robust enforcement practices to be perceived as less safe than those operating in other regions.

– The questionable assignment of severity weights to individual violations can skew carriers’ scores; and

– The underreporting of crashes by states, the use of crashes that were not caused by motor carriers, and the increased exposure to crashes experienced by carriers operating in urban environments, all affect the significance of Crash Indicator BASIC scores.

To review the full white paper, click here.