The Pitt is covering this, and in an early episode from this season they had one of the doctors point out that the LLM transcription incorrectly labeled a medication.
Medicine has a very low tolerance for errors. If I ask ChatGPT what episode of Downton Abbey shows lord whatshisface vomiting blood and it tells me that episode was the Red Wedding, worst case scenario is I look dumb. If Claude tells a doctor “this patient doesn’t have any existing medications that are contraindicated for propofol,” and it’s wrong, that patient may die on the table.