This letter features reporting from "Researchers Say an AI-Powered Transcription Tool Used in Hospitals Invents Things No One Ever Said" by Garance Burke and Hilke Schellmann
Dear Virginia Department of Health Chief Information Officer, Mr. Soundararajan,
My name is Aarya Karmarkar, and I am a student interested in health sciences writing to address the rise of Artificial Intelligence in medicine. In the medical sphere, where detail can mean the difference between life and death, certain AI tools have been employed to process and transcribe patient accounts, according to the AP News and Pulitzer Center-supported article, “Researchers Say an AI-Powered Transcription Tool Used in Hospitals Invents Things No One Ever Said” by Garance Burke and Hilke Schellmann.
With the fast-paced life of health professionals, these technologies—such as Open AI's public transcription service, Whisper—have been sought out to increase efficiency in patient consultations, despite presenting concerns in safety and viability. Extending beyond minor discrepancies, Whisper has been known to twist true statements and make up or “hallucinate” entire passages of text in its transcripts, which, according to the article, included misrepresentation of patients in what was described as “racial commentary, violent rhetoric and even imagined medical treatments.” Such fabrications could further social biases that lead to disparities in healthcare at large. Moreover, Whisper leaves open the possibility of misdiagnosis; it has a record of inaccuracies like no other AI model, with thousands of distorted recordings among millions of transcripts. In certain instances, it can be especially difficult to distinguish the discrepancies, especially without an accurate record as a basis for reference. Even though Open AI advises against the model’s operation in what are described as high-risk situations, the extent of its use in hospitals within the past year alone brings ethical concerns to the future of medicine, and the cost of a slip-up in text processing on human lives.
As an aspiring medical student who will be navigating the field in a technologically accelerating world, this topic is particularly interesting in seeing the potential and drawbacks of AI solutions in determining the trajectory of healthcare within the next decade and beyond. Because of Whisper’s hallucinations, researchers have had to investigate inaccuracies and recode the model, which not only deems having the AI tool as counterproductive but also prompts a thorough understanding of the A.I. model and computer programming among medical professionals. In an age where these technologies could be regularly integrated during patient simulations and actual accounts, doctors would have to be well versed in using the A.I. models and correcting them if hallucinations arise. Concerns exist for patients especially, those who risk sharing their audio and forsaking privacy, a fundamental principle in healthcare, to technology companies. If such practices are propagated in the healthcare system, there would be grave consequences on the security of medical history records and patient safety, while sparking ethical dilemmas in medical practice, which revolves around doing no harm. On a global scale, AI technologies have been found to be biased towards cohorts used in the models’ training, which explains the biases towards minorities or data-poor regions, and technology-driven high income countries such as the United States and China running most of the datasets, according to a paper published by PLOS Digital Health Journal. Therefore, before implementing them in hospitals, a greater global understanding of and contribution to Artificial Intelligence methods is needed.
With your experience in developing effective technological solutions as the CIO of the Virginia Department of Health, you can help lead training programs with AI models, and by utilizing the impact of technology on bridging geographical boundaries, we can work with software engineers familiar with the framework of tools such as Whisper to host training sessions with people across the state. This will both inform the public of the tendencies of the training bias and help reduce it by taking data from a wide audience of participants, and not simply the developers. Such sessions can then be extended to the national and perhaps even international level and can take the form of mock patient simulations to increase the accuracy and compatibility with actual situations in health care. If extended to the global level, pooling can be conducted from audiences in developing countries or underrepresented areas in order to bridge the gap of data-poor regions, and ultimately, increase the understanding and effectiveness of these technologies amidst an ever-evolving world.
Thank you for your consideration towards potentially implementing these solutions in order to utilize technology to enhance, and not detract from, the integrity of healthcare.
Sincerely,
Aarya Karmarkar
Aarya Karmarkar is a freshman attending Maggie L. Walker Governor’s School for Government and International Studies in Richmond, Virginia. A student researcher interested in health sciences and medicine, Aarya has been recognized for her behavioral science and neuroscience studies by symposiums such as the Virginia Junior Academy of Science (VJAS). She is the founder and co-president of her school's rising Society for Scientific Research program, where she seeks to expand involvement in scientific research and academia across multiple disciplines. Throughout her free time, Aarya enjoys badminton, debate, and writing articles on topics of interest.
Recently, Aarya has explored the potential of digital technology in health sciences, through her discussions as a presenter at the University of Virginia’s national Youth-Nex Conference and her subsequent articles for UVA. In her second letter for the Pulitzer Center, she addresses current Artificial Intelligence usage within healthcare, particularly its limitations, with regard for the progress needed to advance such technologies in the years to come
Read more winning entries from the 2024 Local Letters for Global Change contest!