What the audit was

In 2026, UMassSafe reviewed Massachusetts crash data for calendar year 2025. This review included every recording of a Vulnerable User, which consisted of 4,098 records on 3,929 reports. Four further fields (Distracted By, Origin/Destination, Contact Point, and Primary Injury Area) were tabulated for completeness later in 2026, by which time late-finished reports had brought the 2025 total to 4,134 records; their tables use that total. The review asked two questions of each of the seven Vulnerable User fields: Is the field completed, and does the value match what the officer wrote in the narrative?

The work was funded by a federal traffic records grant through the Executive Office of Public Safety and Security, with crash data provided by the RMV Crash Data Program and held in the UMassSafe data warehouse.

What the audit found

Where the data is thin. The fields that describe who was involved and what was struck are in good shape. The fields that describe circumstance and behavior are not, and they all fail the same way: Unknown, Not Reported, or Other in place of a specific code.

  • Origin/Destination – fewer than one record in six names where the person was actually going. Other accounts for 47 percent on its own.
  • Distracted By – names a real distraction on 4 percent of records. This is the field that should be capturing phone use.
  • Safety Equipment – helmet use is unresolved on four in ten bicyclists.
  • Contributing Circumstances – Unknown or empty on 29 percent.
  • Type, Contact Point and Primary Injury Area – filled in on 99, 80 and 79 percent. These are working.

Where the values are wrong.

  • Device classification is mostly right: in about 3 percent of records the narrative describes a different kind of road user than the code says. The largest pattern is an electric bike coded as Motorized Bicyclist (20 records); an e-bike is a Bicyclist.
  • Catch-all codes stand in for answers the narrative already supplies. Other was used for an ordinary action 167 times, and In roadway is routinely used for people the narrative places on a sidewalk, a shoulder, or in a bike lane.

What would improve it. Only the second of these is about officers.

  • Fix the reporting systems first. These are the cheapest wins. Three agencies account for most of the missing Sequence of Events data, and one of them leaves the field empty on every record. Safety Equipment completion tracks the crash-report vendor rather than the department. And 80 Action records carry a code the form does not offer. Those are RMV and vendor fixes.
  • Train on the device codes. An e-bike is a Bicyclist, a stand-up electric scooter is not a kick scooter, and a roadway worker is not an emergency responder. Each of these now has a specific accuracy check on its field page.
  • Name the Unknown habit in review. On most of these fields the narrative already contains the answer, which is how this audit found the gaps in the first place.

How to read the results tables

Each Vulnerable User field page carries a table of the 2025 review results. The rows mean:

  • Acceptable – the field holds a specific value and nothing in the report contradicts it. Records where the narrative simply does not mention the field are counted here; silence was never treated as an error.
  • Inconsistent with narrative – the narrative clearly describes something different from the code. For example, the narrative describes a bicycle and the code says Pedestrian.
  • Not coded – the field says None or Other, but the narrative gives a specific answer that one of the listed codes covers.
  • Unknown – the officer selected Unknown.
  • Empty – the field was left blank or Not Reported.
  • Invalid – the stored value is not one of the allowed codes for the field.

How the narrative comparison worked

Reading 3,778 narratives by hand was not practical, so the comparison was automated. Each narrative was read by an AI model running inside the UMassSafe data warehouse, which extracted what the narrative indicated about the vulnerable user. Those readings were then compared with the coded fields using a fixed set of rules. The narratives were processed in place and no crash narrative left the warehouse.

Before these results were published the comparison was validated against hand review, and four out of five of the records it flagged were confirmed as genuine coding errors. The method is deliberately conservative: where a narrative simply does not mention a field, the record is counted as acceptable. The Inconsistent and Not coded rows therefore point to real, reviewable problems rather than to statistical noise. The Empty, Unknown, and Invalid counts come straight from the data.

What it is for

The point is not to grade officers. It is to find the places where the form, the training, or this manual can be clearer, so that the crash data that reaches engineers and safety planners describes what actually happened. Where the audit found a recurring mix-up, the relevant field page now carries a specific accuracy check or FAQ entry addressing it.