Kaiser Mental Health Staff Raises Concerns About AI Record Tool – CalMatters

from Roxy LynnCalMatters

This story was originally published by CalMatters. Sign up for their newsletters.

In 2024, Kaiser Permanente announced the launch of Abridge. Described in a press release as “environmental listening technology,” the AI-powered scribe is designed to help clinicians, including mental health providers, securely record clinical notes during patient visits.

But what the description fails to mention is that the tool records entire medical encounters, including deeply personal mental health sessions.

During these sessions, mental health professionals are required to obtain patients’ consent before using the tool. However, as shared by multiple providers, this consent process does not include explanations of how the information is processed. It does not say how long or where the records are stored, nor who has access to the data.

This happens in part because this information has not been shared with suppliers, despite their attempts to obtain it.

“Empty assurances”

Ilana Marcucci-Morris chose not to use the platform with her patients. She is a licensed clinical social worker with Kaiser Psychiatry in Oakland. She is also a member of the bargaining committee. In this role, she regularly meets with various Kaiser representatives, including the director of the Northern California Department of Mental Health.

Marcucci-Morris describes how during those meetings, she and other committee members asked questions about protecting patient privacy, HIPAA compliance and safeguards for using these technologies.

According to her, the response from management was often empty reassurances: “We’re compliant. That’s it. That’s all you need to know. We’re checking the technology, therapist. Don’t worry. That’s not your job. We have technical experts. That’s their job,” Marcucci-Morris said in an interview with American Community Media.

“They’re not going to show us, are they? And my feeling is that if you have nothing to hide and you’re doing it completely (…) ethically, then you’re going to show us, you’re going to prove it. They can’t and won’t and they refused when we asked.”

Lygia Pacheco is a psychiatric social worker who provides teletherapy services for Kaiser patients in Southern California. She said Kaiser also refused her requests for further explanations.

In an interview with American Community Media, Pacheco recalled how a colleague once raised concerns with a supervisor. The answer: that “it’s unprofessional to bring your personal beliefs about AI into our work environment.”

For Pacheco, “this only leads to low morale, no place to protect patients. We have to be the voice of the patients who come in their most vulnerable state. And we can’t even be that voice for them, so we feel discouraged.”

“Patient after patient after patient‘

In recent years, providers have been required to accept more patients. This creates intense pressure to maintain documentation and workloads, Marcucci-Morris emphasized.

“It’s like you’re seeing patient after patient after patient after patient with barely enough time to go to the bathroom, have a snack (…) get some fresh air,” she said.

According to Marcucci-Morris, refusing to manage the increased patient volume can be treated as a failure to meet job expectations. This may also lead to disciplinary action.

As a union steward, she said she often represents colleagues during workplace investigations related to delayed paperwork or difficulty managing difficult cases. In those situations, she said, management often recommends using Abridge to save time and avoid additional discipline.

According to her, the vendors she knows who use the technology don’t do so because they support or trust it. Rather, it is because they feel compelled to protect their jobs and comply with the demands of the workplace.

“I think it’s coercion because you’re putting someone in a position to either lose their job or use the software. It’s another choice that’s under coercion,” she explained.

Provider, patient concerns

Brian Hoberman is the Chief Information Officer of The Permanente Medical Group. In a Kaiser press releasehe said, “Abridge’s advanced technology supports the well-being of our physicians by reducing the burden of paperwork.”

He added, “We implemented this new technology after careful review and diligent testing and found it to be well received by patients and physicians…”

For at least one patient interviewed for this story, such assurances were insufficient.

“I’m afraid that this kind of information that is being recorded now could end up in the wrong hands,” said the patient, who requested anonymity for privacy reasons. “I may not want my employers, I may not want my family members, I may not want people to know some of these very intimate conversations and deep conversations that I have with my doctor(s) and with my mental health provider.”

Adriana Webb is a social worker at Kaiser Panorama City in Los Angeles. “I work with patients who have sensitive medical diagnoses like (…) HIV and AIDS, and many times my patients don’t even want that on their chart.”

A Kaiser Permanente spokesperson insisted in an email response to American Community Media that clinicians are required to obtain patient consent before using Abridge. “No one is recorded without their knowledge and consent,” the statement said.

He added that records are kept for no more than 14 days and that data processing meets all HIPAA requirements as well as Kaiser Permanente’s own privacy and security standards.

“Abridge helps clinicians spend more time with patients and less time on administrative tasks,” it said.

Using mental health data as a weapon

According to Nicole Alvarez, senior technology policy analyst at the Center for American Progress, “a recording of someone’s worst moments can be used against them in ways that, you know, (…) a high blood pressure reading can’t.”

She said mental health data can be particularly sensitive because of the stigma surrounding mental illness. For patients, this stigma carries real consequences in areas such as employment, child custody cases, immigration issues, and security clearances. She emphasized that, like other forms of personal data, mental health information can be used as a weapon against individuals.

Agreements between health systems and AI vendors can vary widely, she said. This includes terms related to whether audio recordings or transcripts can be used to train AI models, whether patient data is de-identified, how long the data is kept, whether it can be shared with other customers, and what happens to the information after the contract ends.

Kaiser insists that any data it collects is not used to train AI models.

Yet, in Alvarez’s experience, patients often have little visibility into these arrangements. She argued that health systems have a responsibility to clearly disclose how patient information is handled and used.

Alvarez also stressed that in most cases, patients have the right to refuse records. But, she said, the opt-out process isn’t always clearly laid out. Consent options can range from direct questions on admission to language buried in intake documents, she said, making it important for patients to review forms and disclosures carefully.

She said meaningful consent requires patients not only to know they are being recorded and that they can opt out, but also to understand how their information may be stored, shared or used afterward.

Forced consent

Pacheco experienced this during a personal meeting at Kaiser. Her doctor did not ask for permission to use the app and instead informed her that it would be used. After a while, she decided to refuse to use the platform. Although the doctor stopped the recording, she felt a noticeable discomfort in the doctor’s behavior afterwards.

She later decided to change doctors.

Situations like this are a concern for Marcucci-Morris, who said the company’s approach to obtaining consent to use Abridge during meetings can feel manipulative and coercive. According to her, providers are trained to present the tool in a way that pits the needs of patients and doctors against each other.

She explained that patients are often told the system will help doctors with paperwork, reduce burnout and allow them to spend more time with their families. As a result, patients may feel guilty refusing to use the tool because they don’t want to make their provider’s job more difficult.

She believes this framework pressures patients into consent rather than allowing them to make a completely comfortable and independent decision.

According to Kaiser, Abridge is available in “40 hospitals and more than 600 medical offices in eight states and the District of Columbia,” part of greater adoption of AI technology from the healthcare industry. Abridge works in more than 14 languages.

American Community Media has repeatedly reached out to Abridge AI Inc. for comment but did not receive a response. According to company websiteAbridge describes itself as a “Business Associate” to suppliers. Patients are advised to consult their providers’ privacy policies for information on how their data is protected.

“Therapy is most effective in private life and when trust is achieved through two human beings,” Marcucci-Morris said. For her, “healing happens when human empathy is genuinely offered as part of any kind of mental health treatment relationship.”

She added: “I believe that recording a therapy session changes human behavior. It changes the behavior of the patient.”

Roxy Lynn is a California Local News Contributor with the UC Berkeley Graduate School of Journalism.

This article was originally published on CalMatters and is republished under Creative Commons Attribution-NonCommercial-No Derivatives license.