The AI that remembers every failure it fixes, and gets better with each one

When a cloud service goes down, an engineer's brain does something no current AI diagnostic system has managed to replicate: it holds the current theory of the failure in working memory while simultaneously drawing on years of past experience with similar symptoms. That seamless dance between the immediate and the accumulated has now been formalized into a software framework called OpsMem. See also: the VitaBench 2.0 benchmark that proves how easily agents forget user preferences.

Two memories, one goal

OpsMem, detailed in a preprint by researchers from Nankai University, Huawei, and other Chinese institutions, proposes a dual-memory architecture for failure diagnosis. The framework maintains a short-term memory (STM), a graph that captures symptoms, evidence, and candidate hypotheses for the incident at hand, and a long-term memory (LTM), a separate graph that organizes cross-incident operational experience into patterns, cases, and procedures.

This is not merely a retrieval-augmented generation (RAG) system with a better index. The key innovation is a process the authors call cross-memory resonance (CMR), which runs at each diagnostic round. CMR first extracts symptom and evidence nodes from the STM, normalizes them into query signals through an LLM, then matches those signals against the signals embedded in LTM patterns. Patterns that score above a threshold are activated, and activation propagates to associated cases and procedures, producing a tailored LTM subgraph that conditions the next round of multi-agent diagnosis. This approach aligns with broader research into making AI agents collaborate more effectively, as seen in RecursiveMAS, where agents exchange unspoken thoughts through latent space.

The numbers that matter

Tested on 120 real-world failure incidents from Huawei's production microservice systems, OpsMem achieved a Match score (exact root-cause match) of 78.33% and a Relevant score (at least partial relevance) of 85.83% when using Qwen3.5-27B as the seed LLM. The strongest baseline, GoS combined with LinearRAG, reached 53.33% and 72.50% respectively. That translates to a 46.88% improvement in exact-match accuracy and an 18.39% improvement in relevance over the best competitor.

The improvement held across three different seed LLMs, Qwen3.5-27B, Gemma-4-31B, and GLM-4-32B, suggesting the benefit comes from the architecture itself, not from a particular backbone. For context on how different models perform in engineering tasks, see MiniMax's M2.7 matching Claude Opus on an engineering benchmark.

What the ablation reveals

The paper's ablation study unpacks exactly where the gains come from. Removing the LTM entirely caused the largest performance drop, from 78.33% Match down to 30.83%, confirming that operational experience is the single most important factor. Removing the STM dropped Match to 45.00%, showing that explicit state tracking is necessary for long-horizon diagnosis. Without CMR, i.e., using static retrieval instead of state-aligned activation, Match fell to 56.67%. And without LTM consolidation (the mechanism that distills solved incidents back into LTM), Match dropped to 70.83%, a smaller but still meaningful decline that suggests the system can indeed improve over time as it accumulates resolved cases.

Why current methods fall short

Existing approaches fall into two camps: agentic-reasoning methods (like ReAct and GoS) that organize the diagnostic trajectory but lack operational experience, and knowledge-augmented methods (like VectorRAG and GraphRAG) that retrieve external knowledge but fail to align it with the evolving diagnostic state.

"In real-world operations, engineers diagnose failures through iterative evidence collection, observation analysis, and hypothesis refinement guided by operational experience," the authors write. "Agentic-reasoning methods emphasize diagnostic state, while knowledge-augmented methods emphasize operational experience. However, effective diagnosis requires the two to be tightly coupled."

That coupling is precisely what CMR achieves. When the STM contains evidence of sleeping database connections, for example, CMR activates LTM patterns related to idle-slot occupation rather than generic overload, the same associative leap an experienced engineer would make. This echoes the philosophy behind Alibaba's Qoder, which gives developers visibility into what the AI is thinking.

Self-evolution: learning from solved incidents

Perhaps the most intriguing finding is the system's ability to improve over time. The researchers processed the 120 incidents sequentially, allowing OpsMem to consolidate validated experience into LTM after each diagnosis. Compared with a variant that kept the initial LTM fixed, OpsMem correctly diagnosed additional incidents in every consecutive window of 30 incidents, with the largest exact-match gain appearing in the last window.

This suggests that LTM consolidation is not a gimmick but a genuine capability: each solved incident enriches the LTM with new patterns, cases, or procedures, making the system incrementally better at future diagnoses. For a deeper look at how agents can learn from experience, see VitaBench 2.0's test of continuous user preference extraction.

Limitations and open questions

Several questions remain unanswered. The experiments rely on a single dataset from Huawei, and the LTM was initially constructed from interviews, questionnaires, and operational documents, a labor-intensive process that may be hard to replicate at scale. The threshold settings (0.6 for both signal and pattern activation, top-3 retention for patterns, cases, and procedures, and a maximum of 3 diagnostic rounds) were fixed in the experiments; sensitivity analysis across different environments is not provided.

Furthermore, the evaluation uses LLM-as-a-Judge with Qwen3.5-27B and self-consistency voting, which, while validated by expert sampling, still inherits the judge model's own blind spots. And the paper does not report end-to-end latency or compute cost, which could be significant given the multi-agent loop and the CMR step that runs at every diagnostic round.

What this means for the field

OpsMem enters a crowded space. LLM-based failure diagnosis has become one of the most active areas in AI engineering, with papers like ReAct, GoS, RCACopilot, and FlowXpert all proposing different architectures. What OpsMem contributes is a principled answer to a question none of them fully addressed: how to keep the diagnostic state and the operational knowledge base mutually aligned as the diagnosis itself changes the state. This challenge is part of a larger industry trend, as explored in the gap between prototype and production in vibe coding.

The dual-memory idea is not new in cognitive science, it is essentially a computational take on the distinction between working memory and long-term memory that psychologists have studied for decades. But applying it to AI-driven failure diagnosis with a concrete resonance mechanism that bridges the two is novel, and the empirical results are strong enough to warrant serious attention from both the research community and operations teams at large-scale cloud providers.

The code and prompts are publicly available on GitHub, which should accelerate independent verification. For now, OpsMem stands as one of the most promising recent approaches to the stubborn problem of making AI diagnostics that learn from every failure they encounter.