Fireworks AI security breach exposed Notion AI chats: what leaked and what to rotate

Notion has told the owners of fewer than 1% of its workspaces that some Notion AI chat transcripts may have been taken in a security incident. The breach did not happen at Notion. It happened at Fireworks AI, an inference company Notion used to run part of its AI features, and the exposed data, according to Notion's Help Center notice, is a set of chat transcripts used by a feature called Insights.

Those transcripts carried no label saying who wrote them or which workspace they came from, and they contained no tool call results. Notion calls them “unattributable”. In the same notice it tells affected customers to treat the data as exposed anyway, and to rotate any password, API key or credential someone pasted into Notion AI. Data that is anonymous in structure but not necessarily in content: that tension is the part worth understanding.

Notion describes Fireworks as one of the most-used AI infrastructure platforms in the world, powering tens of thousands of customers and used by hundreds of thousands of developers. That scale makes this a supply-chain story as much as a Notion one. When a product's AI runs on someone else's GPUs, the prompts go there too.

What leaked, and what did not

The dataset that Fireworks confirmed to Notion contained the words of Notion AI conversations: what people asked and what came back. It did not contain the fields that would normally tie a conversation to its owner. There was no author, no workspace identifier and no company name attached. Notion says that beyond what is in the transcript itself, nothing links a chat to a workspace, user or company, and no other workspace data is affected.

The exclusion of tool call results matters more than it sounds. When Notion AI answers a question, it often pulls in Notion page content or external data. Notion gives exactly those two examples of what tool call results contain, and says none of it was in the transcripts. So a chat that summarised a confidential roadmap page leaked the request and the reply, but not the page the tool fetched to write that reply.

Why Notion's chats were at Fireworks at all

Notion used Fireworks for two jobs. The first was classification: Fireworks ran jobs that sorted Notion AI conversations so that customers could understand their own AI usage through a feature called Insights. The second was general inference, serving open-weight models inside the Notion product rather than calling a proprietary model vendor.

The exposed dataset came from the first job. Notion's analysis found that it held the unattributable transcripts used for Insights. That is a useful detail for anyone designing an AI feature: an analytics layer on top of AI usage needs the conversations themselves as input, which means a copy of user prompts living at a second company for a reason most users never think about.

Both uses are now off. Insights is disabled while the investigation continues, and Notion says it is no longer serving open-weight models through Fireworks.

How the response unfolded

Notion first heard from Fireworks about potential incidents earlier in the month and shut down every connection to Fireworks' systems as a precaution. The sharper turn came later, when Fireworks reported that attackers may have reached customer data and then confirmed a new dataset. Notion's earlier disconnection did not prevent the exposure, because the data had already been sitting at Fireworks; what it did was stop anything new flowing there.

1. Before Nov 3

   First warning, connections cut

   Fireworks tells Notion about potential incidents. Notion shuts down all connections to Fireworks' systems out of caution.

2. Nov 3

   Full disconnection

   By this date Notion has disconnected from Fireworks, deleted all of its data in the Fireworks system, stopped sending data and traffic there, and halted all training runs.

3. Nov 6

   Possible access to customer data

   Fireworks notifies Notion that malicious actors may have accessed potential customer data. Notion opens another internal investigation.

4. Nov 8

   Dataset confirmed

   Fireworks confirms the new dataset. Notion's analysis finds unattributable Notion AI chat transcripts used for Insights.

‘Unattributable’ is not the same as harmless

Stripping identifiers before sending data to a vendor is sound practice, and here it limited the damage. An attacker holding the dataset has no column to sort by company and no user ID to look up. Notion's wording is careful about this: there is nothing tying a chat to a workspace, user or company “beyond what's in the chat transcript itself.” That qualifier is the whole risk.

People write identifying things into AI chats all the time. A request to draft a reply to a named client, an email signature pasted for formatting, a project codename, a database connection string pasted for debugging: each turns an unlabelled transcript into a labelled one. Notion's own advice to treat the data as exposed is an acknowledgement that the content can carry what the metadata did not.

There is a second subtlety. Attribution exists, just not in the leaked copy. Notion can produce, for Enterprise owners, a version of the affected data annotated with the user behind each transcript. So the mapping lives on Notion's side, and the exposure was the de-identified copy at Fireworks. That is the design working as intended, and also a reminder that de-identification protects only as far as the text inside it allows.

Checking whether your workspace is affected

Notion has contacted the owners of every affected workspace. If neither you nor your workspace owner has heard from Notion, your workspace is not affected and nothing needs doing now. If you have been contacted, the review lives inside Notion itself, not in an email attachment, and it needs a desktop browser.

1. Update Notion and open it on desktop

   Use a desktop browser with the latest version of Notion. The affected data cannot be viewed on mobile.

2. Open the incident review

   In the workspace sidebar, click ? and then Security incident review. Only workspace owners can see this.

3. Download or share

   Enterprise owners can download the CSVs directly. Owners on other plans get a page with a link to share with affected members.

4. Read the transcripts for secrets

   Search the exported text for passwords, API keys, tokens and anything else that grants access, and rotate whatever you find.

Enterprise owners see more than everyone else

What an owner can retrieve depends on the plan. Enterprise workspaces get two files, one of which adds back the attribution the attackers never had, so that a security team can run its own impact analysis. Other plans get a member-by-member approach, where each affected person sees only their own conversations.

What to rotate, and why the advice is broad

Notion's recommendation is blunt: if anyone on the team pasted passwords, API keys or other credentials into Notion AI during this period, rotate them right away. The reasoning follows from everything above. A credential in a chat transcript is usable by whoever reads it, with or without knowing whose it is. An unlabelled AWS key still opens the AWS account.

For engineering teams, the practical list is the usual one: cloud access keys, database passwords and connection strings, CI/CD tokens, third-party API keys, OAuth client secrets and personal access tokens. Revoke and reissue rather than merely changing a password where the service supports it, and check the access logs of each service for use of the old credential. Owners on non-Enterprise plans should push the member links out quickly, because only each member can see what they themselves pasted.

If you cannot tell exactly which window matters, rotate anyway. The cost of reissuing a key is minutes; the cost of guessing wrong is an attacker with working access.

Open questions and what to watch

Notion's notice explains what it shared with Fireworks and what it found in the confirmed dataset. It does not explain how attackers got into Fireworks, or what else they reached there. Notion points readers to Fireworks' own security updates for that, and any team that uses Fireworks directly should read those rather than infer its exposure from Notion's account.

Several things on Notion's side are also still in motion. Insights is disabled pending the investigation, with no stated return. Open-weight models are no longer served through Fireworks, and the notice does not say what, if anything, replaces it. The line that Notion has halted all training runs is given without further detail. Notion says it continues to monitor the incident, so the Help Center page is the place to check for changes.

The wider question is one every AI product team now faces. Classification for analytics features quietly creates extra copies of user prompts at extra companies. Notion's de-identification kept this incident smaller than it could have been; whether teams start sending less to vendors in the first place is the thing to watch.

This article draws on Transport security has blocked a cleartext HTTP and answers to it (1, 2), on the Stack Exchange network, licensed under CC BY-SA 4.0.