ASECTT: Study finds SMS percentile scores arbitrary, invalid

user-gravatar Headshot

Asectt Logo

The Alliance for Safe, Efficient and Competitive Truck Transportation on Tuesday, June 26, released the results of a study it commissioned of the correlation between individual carrier percentile rankings and crash frequency based upon data obtained from the Federal Motor Carrier Safety Administration’s data bank.

ASECTT said the study by Dr. Inam Iyoob, director of engineering at Transplace, shows that with respect to individual carriers, percentile rankings of carriers both above and below the arbitrary “monitoring thresholds” are not valid predictors of crash frequency.

The study found that regression analysis shows that Safety Measurement System percentile scores account for less than 1 percent of the variation in crash frequency for each of these Behavior Analysis Safety Improvement Categories.

ASECTT said the study shows FMCSA’s data cannot be used to predict the crash performance of individual carriers, even though the agency claims SMS scores are correlated to the average crash frequency of hundreds of carriers at each percentile integral.

“Consumers of freight transportation do not select ‘average’ carriers – they select individual carriers,” ASECTT said in a statement. “The agency study offers no proof that SMS methodology is a predictor of individual carrier safety performance at any percentile level.”

“I can’t see any useful purpose in averaging the crash data of hundreds of carriers in each of 100 different percentiles and then calculating a regression of the average values,” said Iyoob. “The purpose of regression analysis is to explain variation. Averaging hundreds of carriers at each percentile eliminates most of the variation in the data. It is not statistically accurate to say the SMS methodology and BASIC percentile scores are an accurate predictor of carrier safety predicated upon the crash data the agency uses to justify its conclusions.”

ASECTT acknowledged that while unsafe driving and driver fatigue logically do impact crashes, the group said the study shows the way the Unsafe Driving and Fatigued Driving BASICs are captured, calculated and interpreted by FMCSA does not show any correlation to crashes. “Hence, usage of SMS data for carrier selection will unduly favor some and penalize others, and thus should be avoided,” ASECTT said.

FMCSA Administrator Anne Ferro – updating attendees of the CCJ Spring Symposium in Birmingham, Ala., on the agency’s latest rulemakings – said the industry can be on the lookout for a Safety Fitness Determination proposal and perhaps a rulemaking in 2013.

The SMS replaced SafeStat to determine which fleets should be eyeballed for roadside inspections and to provide fleets with better visibility. Under the Safety Fitness Determination, FMCSA could use that data to replace the onsite inspection process.

Ferro said fleet feedback from the CSA preview is critical. “Some in the industry are very concerned, and we need to know what those concerns are so that we can address them,” she told Symposium attendees.

Ferro estimated a current population of about 500,000 freight carriers and said that about 490,000 of those have no fatality numbers. “We have 200,000 carriers with sufficient data to analyze for all carriers, and 92 percent of all crashes are attributed to those carriers,” she said. “They represent 80 percent of the power units.”

Crash history is one of the stronger predictors of future crashes, said Ferro, discussing how FMCSA is addressing crash accountability in CSA’s BASICs. Ferro said the agency calls this process “crash weighting” – preventable vs. nonpreventable.

“We started to answer that question through police reports, but those vary from state to state,” she said. “Over the course of the 90-day filing period, the content may change. So our challenge is if we have the final final report?”

The American Trucking Associations has asked Ferro to release the full results of the agency’s study on using police reports to determine crash accountability. “FMCSA continues to use crashes that motor carriers did not cause nor could have prevented in measuring motor carrier safety performance,” Graves said.

FMCSA on May 25 announced a 60-day extension of the time that motor carriers have to comment on proposed improvements to SMS. During the extended comment period that runs through July 30, FMCSA is conducting further outreach to carriers to encourage them to review how proposed enhancements will affect their SMS score and take action to correct potential safety problems before the changes are implemented.

Carriers can access the SMS Preview through either the CSA Website or the FMCSA Portal using the “CSA Outreach” link. For more information about CSA, visit Language recently was added to the CSA Website that underscores the lack of crash responsibility in crash data.

“We’ve added language to all our Websites to more clearly articulate [the fact that] crash information doesn’t include preventability information,” said Bryan Price, FMCSA senior transportation specialist.

According to the information section of the CSA main site, “a caveat is placed wherever Crash Indicator-related values are shown. The caveat states, ‘A motor carrier’s crash assessment (Crash Indicator BASIC measure and percentile) and the list of crashes below represent a motor carrier’s involvement in 24 months of reportable crashes without any determination as to responsibility.’ ”