New studies published on ArXiv reveal that Large Language Models (LLMs) struggle to maintain consistent security and privacy policies in prolonged conversational contexts, raising crucial questions about their reliability in critical applications and the necessity of robust ethical AI.
What happened
Research titled "Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents" ArXiv cs.AI identified a phenomenon called Security-Recall Divergence (SRD). This study, based on 4,416 trials conducted across 12 LLM models and 8 different providers, demonstrated that AI agents tend to forget omission instructions (e.g., prohibitions on credential disclosure or data exfiltration) when subjected to prolonged context pressure. Conversely, commission instructions (those requiring a specific action) tend to persist. This asymmetry creates a significant risk for operational security in production environments.
In parallel, another study, "Reinforcing privacy reasoning in LLMs via normative simulacra from fiction" ArXiv cs.AI, highlighted how LLM agents' information handling practices are often misaligned with users' contextual privacy expectations. Researchers propose improving privacy reasoning in LLMs by extracting "normative simulacra" (structured representations of norms and information flows) from fiction novels, rather than relying on expensive supervisor-assistant architectures or narrow, task-specific fine-tuning data. The goal is to align LLMs with the principles of Contextual Integrity (CI), a framework defining privacy as the appropriate flow of information within context-relative norms.
These challenges emerge as research continues to explore new architectures to enhance LLM capabilities. For instance, the paper "Absorber LLM: Harnessing Causal Synchronization for Test-Time Training" ArXiv cs.AI proposes a new model to address the high computational cost and memory consumption of Transformers in handling long sequences, aiming to preserve the causal effect of context. However, even with technical improvements, the persistence of security and privacy policies remains an open question. The impact of these limitations is particularly relevant in sectors where reliability is critical, as shown by studies on orchestrating biomedical workflows with PoSyMed ArXiv cs.AI and planning planetary exploration with software-defined systems ArXiv cs.AI.
Why it matters
Security-Recall Divergence and the misalignment with privacy expectations are not purely academic problems; they have direct and significant implications for AI adoption and trust. In corporate and public contexts, an LLM agent that forgets a security directive or violates privacy can cause reputational, financial, and legal damage. This is particularly concerning in an era where regulations like the EU AI Act are imposing stringent requirements on the transparency, security, and reliability of AI systems.
For businesses, managing these risks requires not only technological updates but also a review of operational processes and staff training. Developers and operators will need to be aware of these inherent LLM vulnerabilities and implement robust mitigation strategies. An AI system's ability to consistently adhere to user-defined policies is fundamental for its responsible integration into any critical decision-making process or interaction with sensitive data.
The HDAI perspective
These studies underscore that LLM reliability is not merely a technical problem of architecture or optimization, but an issue intrinsically linked to ethical AI and governance. The challenge is not just to build bigger or faster models, but systems that are predictable, secure, and aligned with human values and societal expectations. An AI truly driven by human needs must guarantee the persistence of security policies and respect for privacy, even under contextual pressure.
The philosophy of Human Driven AI promotes an approach where technology is designed to be controllable and accountable. Topics such as policy persistence, privacy management, and the need to instill normative reasoning in models will be central to the discussions and workshops we will address at the HDAI Summit 2026 in Pompeii. It is crucial that the AI community focuses not only on capabilities but also on limitations and ways to make AI more reliable and trustworthy.
What to watch
Future research must focus on developing LLM architectures that are intrinsically more robust to Security-Recall Divergence and on innovative methods for instilling normative reasoning and privacy understanding. The development of standards and best practices for continuous evaluation of AI agents' security and compliance will also be crucial, alongside the evolution of regulations to address these new and complex challenges posed by generative artificial intelligence.