ATA questions validity of CSA as a measure of safety risk for individual carriers

(EDITOR’s NOTE: This is Part 3 of a series on CSA, based on several presentations, discussion and follow-up from the recent American Trucking Associations Management Conference and Exhibition. Part 1 focuses on whether CSA is, essentially, a new safety rating. In Part 2, FMCSA explains changes to the carrier safety information website. In Part 4, a data analyst and an FMCSA official debate the state-by-state differences in enforcement.) 

Playing the percentages is fine for setting enforcement priorities, but should the Federal Motor Carrier Safety Administration be able to publicly label carriers as unsafe based on statistical probabilities?

FMCSA graphicFMCSA graphic

The American Trucking Associations has doubts, and expressed them in a CSA-focused session at the recent Management Conference and Exhibition. But a leader of the Compliance, Safety, Accountability program was there to insist that the package works as designed.

Rob Abbott, ATA vice president for safety policy, outlined ATA’s position, starting with the good news.

“It’s very important that we pause and point out there are many, many positives in CSA right now,” Abbott said.  “And I think we’d all have to agree that since they rolled out the SMS, there’s been a greater focus on safety by fleets and drivers.”

Specifically, he noted that as a “performance-based” system that focuses on roadside inspections, CSA is significant improvement over previous programs that tended to focus on paperwork.

Likewise, he credits FMCSA for being generally responsive to industry suggestions on program improvements.

ATA’s main concern, however, is whether the scores are truly reliable measures of safety performance.

In terms of data quality, the state-to-state variations in enforcement can skew carrier scores, for example. ATA also questions the methodology, especially crash accountability and some of the severity weights that seem “illogical” to carriers.

“We are comparing scores and saying a fleet is relatively safe or unsafe, but people who look at it say,  ‘what does relatively safe or unsafe mean?’” Abbott said.

And, with sufficient data to evaluate and score only 40 percent of all carriers under the program’s Safety Management System – which essentially ignores hundreds of thousands of small carriers – just how effective can the dataset be?

“Are the fleets they have data on representative of all fleets nationwide? We really don’t know,” Abbott said.

ATA also questions the relationship between scores and crash risk.  While some BASICs do demonstrate a positive relationship to crashes, others not so much – indeed, some have an inverse relationship to crash risk.

Granted, setting priorities is important for the agency. But what should those priorities be, especially when compliance doesn’t equate to safety?

“Is it appropriate to say that this fleet does have a high score, so therefore they’re likely to be unsafe because the averages work out that way?” Abbott said. “Let’s be honest. There are a number of fleets that have high scores they deserve. But can we make these individual determinations? How strong are these relationships? How much less safe are they?”

According to an ATRI study, fleets that had a high score in at least one BASIC are 30 to 40 percent more likely to be involved in a crash.  But, considering that a 200 percent difference would be double the crash risk, or that smoking increases the risk of lung cancer by 2,300 percent, to use Abbott’s example, just how significant is that 30 percent?

“It’s kind of modest,” he said. “Is the program good enough to prioritize carriers?”

It most certainly is, says Bill Quade, associate administrator for enforcement and program delivery and the man who oversees all of FMCSA’s enforcement programs.

“I tend to be a little bit defensive about this program,” he said in his opening remarks. “I’ve worked a long time on it and I am, frankly, very, very proud of what we’re doing with it.”

Quade discounts those who criticize CSA as an incomplete work in progress, and who say the program’s constant changes prove that it’s still not ready.

“When we rolled CSA out in 2010 we were absolutely confident it was going to do what we need it to do, and it did do that,” Quade said. “Did we improve it? Yes. Did Apple improve the iPhone 2, 3, 4, 5? Are we going to continue to improve it? Yes. It was absolutely ready.”

He points to the numbers: Violations per roadside inspection are down 14 percent. Driver violations are down 17 percent.

“We started paying attention to every violation instead of just the out-of-service violations,” Quade said. “These are really significant changes and very positive changes.”

He refutes the suggestion that the dataset is too limited to be effective.

“There’s nothing I would like more than to have sufficient data to evaluate every single carrier.  But we only have enough data to evaluate 40 percent of the carriers,” Quade said. “But that 40 percent are in 92 percent of the crashes. We feel like, for the most part, we are targeting the right 40 percent.’

Additionally, there 53,000 carriers with one or more BASICs above the threshold, or 10 percent, and they have almost 50 percent of the crashes, according to FMCSA.

The SMS website has hosted 48 million user sessions, a “tenfold increase” over the previous program.

“That is a visibility to safety metrics that has never existed before,” he said. “That visibility is what makes CSA the program that it is.”

Still, that visibility, or the chance of carrier data being misunderstood or misconstrued, is what trucking companies worry about. And while FMCSA is in the process of improving the SMS public presentation, Quade admits the system isn’t a perfect picture of the safety risk of each and every carrier.

“On a macro level, the data tells me that we’re doing a good job. I’m aware that on a micro level, sometimes the program isn’t everything. There are safe companies with high scores. There are companies that are not safe that don’t have high scores,” Quade said. “One of our challenges in regulating an industry that’s as large and diverse as the trucking industry, with the resources that we’ve been given, is that a lot of times we have to put our efforts and faith in the macro. We can’t ignore the micro, but on a macro level we feel like the program is doing what we need it to do.”

(Coming up: The agency’s own SMS data is used to expose vast differences in enforcement priorities across the country, and Quade shows that he is indeed defensive about CSA.)