AI Denials vs. AI Resistance: Combatting Health Insurance Cases

 

Recently, we assisted some NGOs (non-profit) with denied coverage by health insurance providers for disadvantaged people they were helping due to the high cost of legal advice. During our initial research inquiries for possibilities, we became aware of a San Francisco software engineer, Holden Karau, who approached and developed an AI (LLM), an open-source platform model that she called Fight Health Insurance to “fight back.” Her motivation came when Karau was denied coverage by her health insurance provider nearly 40 times and refused to give up. She discovered that most denials can be overturned with persistent appeals.

Our article highlights a broader reality of the modern U.S. health insurance system: data analytics and artificial intelligence (AI) have become embedded in insurers' evaluation and management of coverage. We also discuss how Karau's platform model works, its success, the insurance landscape, what is behind the denials, and possible future advocacy through artificial intelligence.

The Fight Health Insurance Process

The Landscape

Data analytics and artificial intelligence (AI) have become deeply embedded in insurers' evaluation and management of coverage. Insurers say evidence-based guidelines and predictive modeling reduce fraudulent claims and keep premiums affordable. However, patients often see an opaque set of approval protocols—algorithms that can inadvertently deny procedures or medications doctors deem necessary.

Utilization management processes such as prior authorization or step therapy rely heavily on data-driven tools. High-tech or “experimental” treatments—like certain gender-affirming procedures—can be flagged for denial. Adding to the complexity is a system in which insurers leverage machine learning to detect fraud, assess outlier claims, and streamline operations. But those same models can seem like blunt instruments to legitimate patients seeking coverage, especially when they must devote hours (or months) unraveling how to access an appeals mechanism.

Actuarial calculations also shape plan benefits, determining which services are included in coverage tiers and how cost-sharing is structured. While analytics-based decision-making frequently optimizes insurer risk and reduces waste, critics argue that it creates unnecessary hurdles for people who want the care they’re paying for.

Could some hallucination, as in other instances with LLM models, be the cause?

AI hallucinations occur when an AI system generates factually incorrect outputs that are nonsensical or irrelevant to the given input. This can happen due to various factors, including biases in the training data, limitations in the AI model, or unexpected inputs.

While there are limited documented cases of patients being denied procedures or medications specifically due to AI hallucinations, there are instances where AI algorithms were implicated in potentially inappropriate denials:

• UnitedHealth Group and NaviHealth: A class-action lawsuit alleged that UnitedHealth Group and its subsidiary, NaviHealth, used a faulty AI algorithm to deny coverage for extended care to elderly Medicare Advantage patients, overriding their physicians' recommendations. The lawsuit claimed the algorithm had a 90% error rate, resulting in patients being prematurely discharged from care facilities or forced to pay out-of-pocket for necessary medical care. (See: Class action complaint against UnitedHealth Group for wrongfully denying care AI model with a high error rate. CASE 0:23-cv-03514 Doc).
• Humana: Another class-action lawsuit accused Humana of using an AI model called nHPredict to deny medically necessary care for elderly and disabled patients covered under Medicare Advantage. The lawsuit alleged that the AI model increased denials and premature terminations of coverage for services that would have been previously approved.
• Senate Investigation: A Senate investigation criticized UnitedHealthcare, Humana, and CVS for using AI to deny prior authorization requests for Medicare Advantage plan holders who require post-acute care. The investigation found that denial rates for post-acute care were significantly higher than for other care types, raising concerns that AI algorithms prioritized cost savings over patient needs.

These cases highlight the potential for AI algorithms to contribute to inappropriate care denials, even if they are not directly caused by hallucinations. They underscore the need for greater transparency, oversight, and accountability in using AI in healthcare insurance.

Nevertheless, it is important to note that AI systems in insurance are often subject to human oversight, and human reviewers typically make the final decisions. However, the increasing complexity of AI algorithms and the pressure to automate processes may limit the effectiveness of human oversight, leading to denials. Or is there some percentage by insurance companies' designs?

Behind the Denials

As the KFF graph at the beginning of the article suggests, only 0.1% of Affordable Care Act rejections are appealed. The cumbersome process deters most consumers. Healthcare advocates confirm that most denials could be overturned if policyholders were willing to fight.

“Very few people know about the process, and even fewer take advantage of it because it’s rather cumbersome, arcane, and confusing—by design,” says Dr. Harley Schultz, a patient advocate in the San Francisco Bay Area. Physicians typically don’t have time to file appeals for each patient, leaving coverage disputes unresolved. Karau’s platform could return the power to the patient, automating some of the most time-consuming aspects.

Karau's Approach

At first, Karau’s battles with her insurer were personal. Facing a denial, she’d spend days gathering documentation, writing multi-page appeal letters, and referencing insurer guidelines. The payoff: She consistently won.

Eventually, she started writing appeals for friends, and her hobby morphed into a question typical of Bay Area engineers: “Can we automate this?” Fight Health Insurance, a gift for self-advocacy with cutting-edge AI.

Future Advocacy

In our opinion, Karau’s story represents a shift toward consumer empowerment in health insurance. By blending AI with open-source collaboration, she shows how everyday patients can leverage technology to challenge data-driven insurer protocols. We are confident that similar platforms for the same purpose and more complex in the near future will emerge.

For insurers, this movement could force internal reevaluations. If denial overturns rates soar, the algorithms and policies that flag claims might need retooling to avoid endless appeals. On the consumer end, increased transparency and faster appeals open the door for fairer, more patient-centric coverage decisions.

Still, algorithmic bias remains a concern. For example, insurers’ predictive models may systematically disadvantage certain groups. Without clear standards and frequent auditing, AI can accidentally codify discriminatory practices.

Where Karau Goes From Here

Karau does not plan to quit her day job anytime soon, but she hopes Fight Health Insurance will grow into a self-sustaining platform. She has already invested thousands of dollars in development, and each new user gives her feedback to refine the appeal generation process and sharpen the LLM’s language.

“The best-case scenario—admittedly unlikely—is that this tool makes health insurance companies think twice before rejecting legitimate claims. If it causes them to stop being ‘dicks about small things,’ that’s a step in the right direction.

Meanwhile, insurers continue to embrace data analytics to manage risk, reduce costs, and detect fraud. The collision of these two approaches—insurer AI vs. patient AI—might reshape the relationship between health plans and their members. It could mean fewer blind denials and a more even playing field. Alternatively, it could push insurers to develop even more sophisticated coverage systems.

Karau believes in “making the world suck a little less.” For patients overwhelmed by paperwork, this is a glimmer of hope that they can challenge—and often reverse—insurance denials.

Finally, coverage decisions and AI-driven analytics have undeniably transformed the health insurance landscape—but personal accounts like Holden Karau’s shed light on what happens when the system denies valid care. Insurers cite data-based methods to ensure cost-effectiveness, while patients often experience those same methods as opaque barriers. Karau’s success exemplifies a new wave of consumer advocacy, armed with open-source AI, pushing back against under-explained denials.

In the bigger picture, this tension highlights AI's dual nature in health insurance: It can enable faster, more equitable claims processing or act as another gatekeeper. The public perception of insurers will likely hinge on how transparently and ethically they deploy data analytics and on whether platforms like Fight Health Insurance become mainstream enough to hold them accountable. For now, Karau’s motto, “Make your health insurance company cry too,” captures the grit and creativity driving this new era of patient empowerment.

References

‘Make your health insurance company cry’: One woman’s fight to turn the tables on insurer, By Jillian D’Onfro, Published Aug. 23, 2024, The San Francisco Standard.

Claims Denials and Appeals in ACA Marketplace Plans in 2021, Karen Pollitz, Justin Lo, Rayna Wallace, and Salem Mengistu. Published: Feb 09, 2023, KFF.

A Patient’s Guide to Navigating the Insurance Appeals Process, By Patient Advocate Foundation.

MITIGATING AI/ML BIAS IN CONTEXT Establishing Practices for Testing, Evaluation, Verification, and Validation of AI Systems Apostol Vassilev Harold Booth Murugiah Souppaya National Institute of Standards and Technology November 2022 ai-bias@nist.gov

The Role of AI in Insurance: From Buzzword to Bottom Line, By Shift Technology. October 24, 2024

Data Analytics in the Insurance Industry | The Ultimate Guide Chandan Gaur | 12 November 2024.