Bottom Line: LLMs can be forced to leak data through targeted prompt attacks, but they disclose training data only with low probability in everyday usage scenarios.
Large language models can reproduce training data, but under normal usage this happens rarely. Researchers demonstrate a significant gap between worst-case scenarios and realistic propensity.
A research team conducted a comprehensive evaluation to distinguish between two types of memorization: forced extraction (capability) and natural leakage (propensity). The study evaluated two fully open models – Comma and DFM Decoder – on two datasets and two languages. For this purpose, PropMe was introduced, a framework that contrasts classical capability tests with more realistic scenarios.
The central findings point to a stable gap between theoretical extraction and practical risk: prefix-based attacks elicit significantly stronger memorization signals than generic or dataset-specific prompts. However, propensity scores remain overall low. This means: the models can reveal training data when specifically queried for it – but do so only rarely in typical, non-adversarial usage situations.
DFM Decoder, which was continuously pretrained from Comma, showed reduced memorization and memorization propensity for the Common Pile dataset. This demonstrates that memorization capabilities can decrease through late-stage training runs that emphasize partially different data. The researchers also investigate mechanisms such as SimpleTrace, a lightweight tracing procedure based on Infini-Gram, which traces outputs back to large-scale training corpora and computes verbatim, near-verbatim, and propensity-transformed metrics.
For CTOs and security professionals, this implies: memorization audits should report both worst-case scenarios and actual leak propensity to obtain a complete risk picture. A model that would rarely disclose training data under normal usage but is vulnerable to attacks requires different protective measures than one with a high spontaneous leak rate.
Source: arxiv.org · Published June 3, 2026
Lumi AI News — AI-assisted curation pursuant to Article 50 EU AI Act. Paraphrase and classification via Lumi News Pipeline v1.6.0.