How memory tools can make AI models worse

One of the biggest selling points of modern AI systems is their ability to adapt to users. Every time the AI ​​assistant takes on a task for you, it also adapts to your style and preferences, which are integrated as context for future tasks. With more context and a better understanding of the user, the model can get better every time you use it – or at least that’s the theory.

New research suggests that the adaptive capabilities of models may be both a blessing and a curse. Wednesday, Researchers at the artificial intelligence company Writer published two papers It shows how common memory systems can make models worse, pushing them toward user-provided misconceptions or misunderstandings. When user input fills more of the form’s context window, the form becomes flatter—and less committed to accuracy.

“We wanted to be able to quantify how often a model would usefully consider a user’s preferences versus giving a potentially wrong answer,” said Dan Bickel, head of AI at Writer, who worked on the papers. As Bickel told TechCrunch, “With every additional storage and retrieval of user preferences, you run an increased risk.”

In one form, researchers tested AI models by recording that a user’s favorite book was Station Eleven, and then asking the model to name the best-selling dystopian book. Models became more likely to name Station Eleven in their answers, even though the question was not related to the user’s favorite book. The trend increases when using memory compression tools such as Note0 and Zeb.

As the paper says, “Fundamentally all memory systems struggle to distinguish between relevant context and irrelevant anchors, which severely undermines diversity and creativity and introduces unintended pathways to bias that can limit the usefulness of the system.”

The second paper shows how the same dynamic can effectively degrade performance, introducing the user to misconceptions about finance and thus challenging the model for analyzing company performance. The more context a model has, the worse it performs.

“With no memory or allocation, the AI ​​model correctly assesses that the company is a capital-intensive business and experiences high customer churn,” the post said. “But with these features turned on, it will happily change its answer to acknowledge the user’s error or provide them with an incorrect answer based on their assessment of their previous preferences.”

Notably, the research did not look at Anthropic’s recent Opus 4.8 model, which was They are trained to effectively address input errors Like that introduction. The patterns the researchers discovered were true across different models. It’s a testament to how delicate the balance is in the context of AI, and how useful tools can have unintended consequences if they upset that balance.