csa

CSA’s Distorted Rankings, Part 3: The ‘ugly little secret’ of CSA

By Todd Dills on

CSA'S DATA TRAIL

The article below is part of an ongoing, in-depth series on the U.S. Department of Transportation's Compliance, Safety, Accountability program that analyzes federal inspection, investigation and crash data and offers original reporting. Overdrive and CCJ editors have built a site dedicated to hosting the stories, interactive maps and downloadable data at CCJdigital.com/csa.

This is the third and final part of the CSA’s Distorted Rankings series, which is part of the broader CSA’s Data Trail series. Click here to see Part 1 of the series, and click here to see Part 2

inspection scales

One initiative to improve Compliance Safety Accountability scoring was an April 30 draft report issued by the CSA Subcommittee of FMCSA’s Motor Carrier Safety Advisory Committee. Principle areas of recommendation are as follows:

Violation severity weights

The current weightings given to individual violations in the Safety Measurement System are point values on a scale of 1-10, with 10 the most severe.

A presentation to the subcommittee by Dave Madsen of the Volpe National Transportation Systems Center, a chief architect of the SMS, illustrated what he called the “ugly little secret” of the system: It implies “a level of precision that doesn’t exist.” Only broad groupings of violations, in FMCSA’s view, correlate to crash risk statistically. Severity weighting, Madsen said, might be better if simplified to a basic low, medium and high system.
CSA_Pie_lead_0614 copy

Tests of such a three-tiered system show that “the companies that move in and out” of alert status in the individual categories don’t change much, Madsen said. Nevertheless, the current system has the added problem of contributing to misperceptions about how the SMS works.

“Everyone’s focused on the individual violation and the points assigned to it. People think ‘points on licenses,’ and third parties take this stuff and create driver scores. … We don’t calculate driver scores,” at least not for public consumption, Madsen added. “In general, we’re trying to get patterns, and we’re losing the forest for the trees with what we have now.”

The CSA Subcommittee recommended such a modification of the weighting system, but its chief concern with the quality of the system went further. “FMCSA should explore further improving correlation of violation groupings within [Behavioral Analysis and Safety Improvement Categories] to crash risk,” its report reads. This would be in keeping with the ultimate goal of identifying unsafe carriers more effectively.

Subcommittee members also offered other things to test relative to violation severity weights – from an even simpler low-high system, with only two weights, to a more detailed examination of what enters the measurement system in the first place.

Carrier peer groupings in the BASICs

Related

CSA’s Distorted Rankings, Part 1: CSA’s flawed scoring doesn’t stop third parties from judging carriers

Three and a half years after Compliance Safety Accountability began its radical scrambling of how trucking safety is regulated and scored, carriers and owner-operators continue ...

One CSA problem is that a carrier can experience a dramatic swing in its percentile ranking when it moves from one peer group to another. It’s often a particularly hard reality at the small-carrier level.

Madsen gave the MCSAC subcommittee the example of a carrier in the smallest peer group in the Hours of Service Compliance BASIC, which includes carriers getting five to 10 relevant inspections. Assume a carrier with 10 inspections and an absolute measure score of 2.39, putting it right on the 65th percentile in the BASIC, or at the level of FMCSA’s intervention alert threshold.

If that carrier then gets a clean inspection, what is typically advised as a way to improve BASIC scores, its absolute measure score falls to 2.25 in Madsen’s example. However, its percentile ranking in the Hours BASIC, its hours-performance face to the public, shoots up to 76 because it’s now being compared to the next-highest carrier peer group, those with 11-20 relevant inspections.

Madsen also presented on so-called “dynamic peer groups” – a concept FMCSA has considered to guard against such severe changes. The idea is that the lines separating peer groups are not set in stone, but are more fluid and narrowly defined.

Say a carrier has five inspections, he said. “In a dynamic safety event group, [that carrier is] compared to those with five, and I look to those immediately to the left and right” with four to six inspections as well.

The CSA Subcommittee adopted the recommendation to implement such a peer grouping system. It also offered these suggestions for testing:

Related

CSA’s Distorted Rankings, Part 2: SMS, safety rating system disconnect compounds problems

The difference in results between CSA rankings and the safety rating system is marked. In some quarters, eliminating that disconnect is seen as at least ...

Changing the peer group being compared more broadly beyond current exposure measures (i.e., number of inspections/power units/vehicle miles traveled). Other peer grouping considerations could include characteristics of operations, routes, number of violations, geography of where carriers received inspections, etc.

Consider separating motorcoach operations from truck operations.

Achieve more uniform crash reporting from states.

Crash exposure should be taken into account in the Crash Indicator BASIC, and not just vehicle miles traveled. In more high-traffic areas, crash risk is higher.

Todd Dills

Todd Dills is Senior Editor of Overdrive magazine and writes from Nashville, Tenn. He frequently covers business, regulatory and lifestyle topics for the magazine and at OverdriveOnline.com. His work on the “CSA’s Data Trail” series in Overdrive about the federal CSA program was awarded the highest honor in trade journalism – the “Grand Neal” – by American Business Media at the 2014 Jesse H. Neal Awards. Dills’ Channel 19 blog covers a grab bag of on-highway hearsay, owner-operator news and driver views from the roadways the nation over. His work in trucking journalism builds on a background of news feature, fiction and other creative writing and editing. Find him here at the Channel 19 blog and via his Twitter feed, or send tips to tdills@randallreilly.com or via phone at 205-907-2481.

2 comments
Bjhime
Bjhime

Relativity at its best!  Why not establish a baseline by means of a compliance review and then measure performance against that?  Oh, wait, FMCSA can't handle all the work, so they elect the expedient means and cheat the industry in the process.

BrianLoysen
BrianLoysen

The dirty little secret of CSA is that in every state the Highway Patrol now has Carte Blanche to harass vehicles to look for the 400+ maintenance violations they can assess points for.

They aren't identifying risky behavior and/or crash behavior.  Ever heard said that certain states DISCOURAGE violation free inspections?


The greatest proof to how serious any governing body is about safety (especially at the local level) is when you call the courts to hear how much that fine for having light problems on your equipment is and you wade through all their fines and realize at fault passenger car accidents often carry less of a fine that your vehicle violations does.