WASHINGTON – The National Academies of Sciences, Engineering, and Medicine released its long-awaited review of the Federal Motor Carrier Safety Administration’s Safety Measurement System (SMS) today and concluded that several features of the system need improvement.

The 132-page mandated report outlines a number of recommendations to the SMS.

The report recommends “over the next two years, FMCSA should develop a more statistically principled approach for the task, based on an item response theory (IRT) model — an approach that has been applied successfully in informing policy decisions in other areas such as hospital rankings.  If the model is then demonstrated to perform well in identifying motor carriers that need interventions, FMCSA should use it to replace SMS.”

The review also found major flaws in a number of data quality issues. The report recommended the FMCSA should work together with states and agencies to improve the collection of data on vehicle miles traveled and crashes, since this data is often missing.

“FMCSA should investigate ways of collecting data that will likely benefit the recommended methodology for safety assessment,” the report reads. “This includes data on carrier characteristics—including information on driver turnover rate, type of cargo, method and level of compensation, and better information on exposure.”

The review argued that compensation levels are relevant because it is known that drivers who are paid well have fewer crashes.

Those who conducted the review were unable to recommend whether SMS percentile ranks should be part of the public domain.

American Trucking Associations officials were pleased with the review, saying  it confirmed much of what the association has been advocating about the program for a while.

“We appreciate the work the National Academies of Science has done in helping motor carriers, FMCSA and the general public learn about the limitations of CSA,” said ATA president and CEO Chris Spear. “This report has confirmed much of what we have said about the program for some time: the program, while a valuable enforcement tool, has significant shortcomings that must be addressed and we look forward to working with FMCSA to strengthen the program.”

Specifically, ATA said the study validated the trucking industry’s concerns about the inclusion of certain types of violations in the CSA system, that geographic enforcement disparities can have a significant impact on carriers’ scores and that the collection and use of clean inspections is critical to the accuracy of the program.

ATA director of safety policy Sean Garney added: “We also see great potential in the Academies’ recommendation that FMCSA overhaul the current CSA methodology in favor of a new, more adaptive, data-centric model with the potential to address serious flaws in the system. To maximize CSA’s potential, there is an urgent need to address issues regarding data sufficiency and accuracy – specifically when looking at crash and carrier exposure data. Until more study – and correction – of these issues is complete, ATA strongly believes FMCSA should continue to keep CSA scores out of the public domain.”

Steve Bryan, president of Vigillo, a SambaSafety Company offered up his comments in a blog post on June 29.

Some of his comments, read: “I hope the FMCSA accepts the recommendations of NAS and seriously begins to work on the next phase of CSA, or whatever we call it.  FMCSA must strike a more collaborative relationship with the industry, and demonstrate absolute transparency in how they are collecting, analyzing, and taking action on this new data model.  NAS suggests that outside experts be hired, perhaps from academia, maybe there are some of us in the industry that know a little about this too.  It’s a moment in time where perhaps, a less combative, more inclusive partnership can be developed and we can re-boot from the very controversial launch of CSA in 2010.”