Researchers warn a core AI flaw may never be fully patched

The News

A pair of researchers has shown that today's most widely used AI models share a design weakness that lets attackers slip past their safety controls, and they argue it may be impossible to fully fix. The finding was presented at the International Conference on Machine Learning (ICML) in July 2026 and reported by MIT Technology Review.

The attack, which they call "chain-of-thought forgery", exploits how large language models tell their internal roles apart. Rather than relying on rigid markers such as XML tags, a model infers whether text is a system instruction, a user message or its own private reasoning largely from style and wording. Independent researchers Charles Ye and Jasmine Cui found that text imitating a model's reasoning trace can trick it into treating an attacker's words as trusted internal thought.

The two tested it on OpenAI's gpt-oss-20b and GPT-5, plus models from Anthropic, Alibaba and DeepSeek. The forged reasoning coaxed models into producing instructions they are built to refuse, from drug synthesis steps to aircraft-sabotage methods. Swapping the XML tags meant to separate roles made little difference. An early version of the technique won OpenAI's red-teaming hackathon in August 2025.

Why It Matters

Ye's assessment, that the weakness may be "fundamentally unsolvable", cuts against the industry's reassurance that every jailbreak can eventually be patched. If roles are inferred from style, and style can be copied, the vulnerability sits in the architecture rather than in a filter a vendor can tighten.

There is precedent. When developer Simon Willison named prompt injection in September 2022, vendors called it an edge case; three years on it remains unsolved and now sits under agentic systems that touch real money and data. This work pushes the same problem into the reasoning layer the latest "thinking" models depend on.

For a market racing to wire language models into customer service, coding and finance, the message is stark: capability is outpacing containment, and no amount of prompt engineering closes a structural gap.

Indian Angle

The timing matters for India, where enterprise adoption is running fast. Banks and fintechs, from large private lenders to firms such as Razorpay, are embedding model-driven agents into support, underwriting and fraud checks. A flaw that lets forged reasoning override guardrails stops being academic when the model can move money or wave through a loan. The Reserve Bank's FREE-AI committee, which reported in 2025 on AI in finance, stressed human oversight for exactly this risk.

Cost sharpens the exposure. Many Indian startups lean on cheaper or open models such as DeepSeek and gpt-oss because rupee budgets are tight, and those were among the vulnerable systems. Home-grown efforts such as Sarvam and Krutrim, built on the same foundations, inherit the weakness rather than escaping it.

Regulators are watching. MeitY's AI governance work and SEBI's scrutiny of AI in the markets both assume these systems can be made auditable and controllable. If a core flaw resists patching, Indian firms deploying them in banking, healthcare and public services will need to treat every model output as untrusted by default, wrapping the model in external checks rather than trusting its refusals.

FAQ

What is chain-of-thought forgery?

It is an attack that imitates the style of a model's own reasoning so the model treats an attacker's text as its private thought. Because models infer roles from wording rather than fixed tags, forged reasoning can override safety instructions the model would normally follow.

Which AI models are affected?

The researchers demonstrated it on OpenAI's gpt-oss-20b and GPT-5 and on models from Anthropic, Alibaba and DeepSeek. Because the weakness stems from how transformer-based systems handle roles, the authors suggest most current large language models share it.

Why do researchers call it unsolvable?

Roles such as user, system and assistant are inferred from style, which is central to how these models work. Since any style can be imitated, Charles Ye argues there is a real chance the problem is fundamentally unsolvable rather than a patchable bug.

What should Indian enterprises do?

Treat model outputs as untrusted. Firms deploying these systems in finance, healthcare or government should add external validation, tight permissions and human sign-off for high-stakes actions, in line with existing RBI and MeitY guidance.

Where can I read the original announcement?

MIT Technology Review reported the findings, presented at ICML in July 2026. The link sits in the attribution paragraph below.

This story was reported by MIT Technology Review. Read the full original coverage at MIT Technology Review.