How I Teamed Up With Academics To Investigate Transcription Software

Why journalistic-academic collaboration is a new model for in-depth accountability reporting

In October, Garance Burke from The Associated Press and I, an award-winning freelance reporter and NYU journalism professor, published an investigation that showed that OpenAI’s popular artificial intelligence-based tool Whisper, used for transcribing audio recordings into words, including doctor-patient visits, invents whole paragraphs of things no one ever said. Some of the invented text included mention of violence, racial commentary and fake medications. The investigation made a splash and many news organizations including WIRED and PBS NewsHour covered the story, citing AP. 

The story came together in an unconventional, novel way that I believe points to a model reporters, and especially freelance journalists like myself and small newsrooms, can adopt for in-depth data and AI accountability-driven stories, which otherwise would be impossible to conduct.

This story, and others on technology accountability, are crucial because journalism proves to be a powerful tool to hold tech companies accountable and over time foster change by raising awareness. However, these kinds of tech investigations are often hard to do and require a team with different skill sets. As a freelance journalist, I don’t have a large interdisciplinary team at my disposal, but I find journalistic–academic collaborations are invaluable. These collaborations are a win-win situation, because I get to answer urgent current journalistic questions through in-depth studies that beget authoritative results. My academic counterparts get to create powerful public scholarship.

The start of this project initially came from my earlier reporting on one-way, AI-based video interviews for hiring, which utilize speech-to-text software. I had always wondered what happens to jobseekers who have an accent like me (my first language is German) or have a speech disability. Would transcription software be able to accurately write out their words? And if not, could this be discrimination based on disability status and/or national origin? 

I wanted to test these tools, but I couldn’t do this alone: With financial help from my AI Accountability Network fellowship at the Pulitzer Center, I worked with my long-time collaborator, the sociologist Dr. Mona Sloane at the University of Virginia, and computer scientist Dr. Allison Koenecke at Cornell University, who had previously done groundbreaking research analyzing transcription tools. Together, with our two research assistants Katelyn Mei and Anna Choi, our team ran over 13,000 audio recordings of people with aphasia (a speech disability some patients develop after a stroke) and without a speech disability through different transcription software tools. 

I vividly remember one of our weekly research meetings as we were going through scatter plot graphics of our findings. We stopped at one visualization showing the results for OpenAI’s Whisper tool: Many, many dots were representing low Word Error Rates (the benchmark standard for this kind of tool), indicating that Whisper might be one of the most accurate tools in the business. We were noticing though, that some dots were complete outliers, indicating over 100%, or 250% or so Word Error Rates. We had never seen such outliers and wondered if maybe this was human error. Did we make a mistake? Did we upload silent audio recordings? We double checked the original audio recordings and compared the ground truth transcriptions with the transcriptions Whisper had later generated, and that’s when we found what the industry calls “hallucinations”: whole passages of text that had been added that were not in the original audio recording.

Here are for example two innocuous ones with the hallucinations in bold: 

Original audio: “Pick the bread and peanut butter.”

Whisper (April 2023): “Take the bread and add butter. In a large mixing bowl, combine the softened Butter.”

Whisper (May 2023): “Take the bread and add butter. Take 2 or 3 sticks, dip them both in the mixed egg wash and coat.”

These fabrications feel different than conventional errors in transcription software, which often happen with details like names and locations, but those errors are in sync with the original audio recordings and don’t go “off the rails,” as we have documented with the fabrications. 

We also identified made up violence, racial commentary and invented medical language.

Original Audio: “And he, the boy was going to, I’m not sure exactly, take the umbrella.”

Whisper: And he, the boy was going to, I’m not sure exactly, take the umbrella. He took a big piece of across. A teeny small piece. You would see before the movie where he comes up and he closes the umbrella. I’m sure he didn’t have a terror knife so he killed a number of people who he killed and many more other generations that were y𝐾 pa.𝐻 . And he walked away.

In another transcription Whisper invented a non-existent medication called “hyperactivated antibiotics.”

We thematically analyzed the fabrications and found that 38% of hallucinations “include explicit harms such as perpetuating violence, making up inaccurate associations, or implying false authority.”  

Our research led to an academic paper that we presented at the 2024 ACM FAccT Conference and the 10th International Conference on Computational Social Science, two leading conferences in the field.  

At the same time, Burke and I worked on an investigative story for the AP, transitioning from documenting a technical anomaly to uncovering its real-world implications. GitHub turned out to be a goldmine: Many software developers had also noticed Whisper generated fabrications and had posted their observations on the platform. 

We built a spreadsheet with all mentions of fabrications in Whisper and then did some sleuthing to deanonymize the most engaged posters. We were able to identify many of the anonymous developers, contacted them via LinkedIn, and ultimately spoke with over a dozen developers who found problems similar to what we had observed. 

We also learned that auto-generated doctor’s notes of audio recordings of patient-doctor interactions are being rapidly adopted in many healthcare systems, which lessens the cognitive load of medical providers so they can concentrate on listening to their patients. 

Some of these AI tools show physicians a transcribed summary of the visit, but let medical providers check the original recording in case they find discrepancies in the transcription. But we found one company, whose software is built on Whisper and is utilized by over 30,000 medical providers in the U.S. The tool deletes the original audio recording, leaving medical providers with only the AI-generated transcript and no clear way to check for “hallucinations”.

Alondra Nelson, the former Deputy Director at the White House Office of Science and Technology Policy, now a professor at the Institute for Advanced Study in Princeton, New Jersey told us that mistakes in doctor’s notes could have “really grave consequences.” 

I hope to facilitate more of these journalistic–academic collaborations in the future, so that other freelance reporters like myself can work on large-scale investigative stories. 

Tips: 

• Try to find academics who have done similar work to what you are proposing and if you can, find academics, who have worked in interdisciplinary teams, which may signal that they are open to collaborations. 
• Before starting the collaboration, talk about expectations, outcomes and deadlines:  journalists most often need an exclusive, so make sure to communicate to the academics that their academic paper cannot be put on a pre-publication server like arxiv.org before the journalistic work is published
• Talk about who gets credited for the work early on. Most news organizations won’t give a byline to anyone who didn’t actually write the article or the script. We agreed to mention the collaboration in the text (which you should do anyways for transparency!).
• Make sure you tell your collaborators that you will reach out to all stakeholders including any companies you are investigating (academics do not always do this). 
• Agree beforehand who from the team will talk to the press (or how you will take turns) if you get queries from other journalists.