Chatbot privacy fails by default, not by users

68 percent of German chatbot users have already discussed a sensitive topic with an AI, according to a survey by the software provider Proton: mental health, finances, relationships, sexuality, family conflicts. At the same time, 40.5 percent of respondents don't trust providers to adequately protect their data. This contradiction is not a failure of the users. It points to a market failure: there is no provider that delivers on privacy technically instead of just promising it.

What the survey shows

Proton surveyed just over 4,000 active chatbot users in Germany, France, the United Kingdom and the United States between July 22 and August 2, 2026. Alongside health and finances, respondents also named secrets and work problems as topics they confide in an AI. At the same time, 51 percent say they fear the creation of advertising profiles from these conversations, and 43 percent worry their statements will feed into training future models. 71 percent would prefer a chatbot that explicitly commits to privacy, and 44 percent said they'd share more intimate details if their input were guaranteed never to be used for training.

These numbers don't describe careless users, but people weighing a trade-off. The benefit of a conversation that's instantly available and never judges outweighs, for them, the diffuse risk that a statement ends up somewhere they'd rather have kept to themselves. Convenience doesn't win because users don't care about privacy, but because there's barely an alternative that offers both. Anyone using a chatbot in Germany, France, the United Kingdom or the United States currently has almost no choice between a fast option and a privacy-conscious one. There is effectively only the one option, and the survey shows that people use it despite their concerns, because a second one is missing.

What the industry delivers instead

OpenAI introduced MentalHealthBench the same week, a benchmark that tests how helpfully and safely a model responds to realistic conversations about mental health. That's an answer to a real problem, since a model that responds wrongly during a crisis can cause harm. Reports on new metrics for answer quality like this keep piling up in the Artificial Intelligence section.

But a benchmark for answer quality doesn't answer the question that 40.5 percent of Proton's respondents are asking: what happens to what I just said, once the answer has been given. Is it stored, for how long, does it feed into training, who sees it internally. MentalHealthBench measures whether the answer is good. It doesn't measure where the question goes afterward.

How the default happens without anyone deciding it

I once built a chat-voice interface for a company's own website, purely informational, so users could ask questions about the company and its services, nothing intimate. Even so, user input ended up with the API provider, not on a server the client or I controlled. The client himself never asked afterward where that data went.

That's exactly the mechanism behind the Proton survey, just one level earlier. It takes no negligence and no ill will for data to end up with a provider. It only takes nobody on the project asking the question before the chatbot goes live. A data processing agreement with clear deletion periods, a self-hosted model, or at least an explicit commitment against using data for training, could have been settled at the start of the project just as easily as any other technical decision. Instead, the provider's default decided, because it was the most convenient option and nobody objected. For an informational bot, that may stay inconsequential. The moment a chatbot on a client's site actually receives intimate or business-critical input, that same unexamined default suddenly becomes a privacy problem nobody planned, but everyone helped build.

The strongest objection

One could object that MentalHealthBench does help privacy indirectly. A model that handles sensitive topics more safely might also handle the data behind them more carefully, for instance by repeating conversation content less often in a later answer or processing it less inappropriately. Answer quality and data handling wouldn't be separate issues then, but two symptoms of the same model architecture.

That doesn't convince me. Storage, training use and access rights are decisions made by the data pipeline and the contracts, not by the model that phrases an answer. My own project shows this in the simplest possible form. The chatbot had no problem with risky answers, the topic was harmless, yet the input still ended up with the provider, because that was simply the default setting. Better response behavior wouldn't have changed that. Quality and data retention are two separate problems, and solving one doesn't touch the other.

What this means for running your own systems

Anyone who builds chat or voice interfaces for clients can't rely on the client asking where the data goes. In my own case, he didn't, and I suspect that's the rule rather than the exception. The question of where input lands, how long it stays, and whether it feeds into a provider's training therefore belongs in project planning, regardless of whether the chatbot's topic seems harmless. Harmless, at the start of a project, is often only an assumption about usage, not a reliable property.

Personally, I apply the same caution to private data elsewhere. I'm careful with sensitive information even with providers whose technology I can assess, because the type of information ends up being secondary. That caution translates to client projects by making the question of storage location and training use part of the architecture from the start, not because every chatbot has to expect intimate confessions, but because nobody can reliably know beforehand what users will actually use it for.

Concretely, that now means: before a project goes live, it's settled where input is stored, for how long, and whether a data processing agreement with the provider covers it. These questions cost little time during planning. Clarifying afterward what happened to data already collected costs considerably more, and it's exactly that after-the-fact clarification that the Proton survey makes visible among users.