Would This Leak? Shadow AI Prompt Challenge

Question 1

Would this leak sensitive data: Drafting a reply to an angry customer?

Accepted Answer

Yes, this would leak. Name, email address, partial card number, order and account identifiers all go to the model provider. Under the Australian Privacy Act and GDPR this is a disclosure to a third party, and most public AI terms allow retention for abuse monitoring even when training is disabled.

Question 2

Would this leak sensitive data: Understanding a regular expression a colleague wrote?

Accepted Answer

No, this is safe. A regex pattern with no real data in it. This is exactly the kind of use that a sensible AI policy should permit. Blocking it is how you push people onto their personal accounts for everything else.

Question 3

Would this leak sensitive data: Debugging a production error?

Accepted Answer

Yes, this would leak. A live service-account password and an internal hostname. Credentials pasted into a chat window should be treated as compromised and rotated. This is one of the most common real incidents, because stack traces and config lines feel like 'just error text'.

Question 4

Would this leak sensitive data: Rewriting clinical notes into plain English for a patient?

Accepted Answer

Yes, this would leak. Age, sex, diagnoses and a discharge date are health information about an identifiable individual. Under HIPAA this needs a Business Associate Agreement with the provider, and public AI tools do not offer one. Australian Privacy Act sensitive-information rules apply too.

Question 5

Would this leak sensitive data: Tightening up a job advertisement before posting?

Accepted Answer

No, this is safe. Content that is about to be published publicly anyway. There is no confidentiality left to protect. This is a good example of work that should be actively encouraged on AI tools.

Question 6

Would this leak sensitive data: Summarising the board pack before a meeting?

Accepted Answer

Yes, this would leak. Unreleased financial performance, churn and restructuring plans. For a listed company this is material non-public information with disclosure obligations. For a private one it is still the single most damaging category if it surfaces.

Question 7

Would this leak sensitive data: Refactoring a function that handles pricing?

Accepted Answer

Yes, this would leak. Proprietary pricing logic, including the uplift multiplier and rebate structure. Code is trade-secret material, and pricing logic in particular is competitively sensitive. Several public AI providers retain conversations for a period regardless of training settings.

Question 8

Would this leak sensitive data: Finding pay anomalies before a remuneration review?

Accepted Answer

Yes, this would leak. Named individuals linked to salary is personal information, and in most jurisdictions it is treated as particularly sensitive employment data. Uploading it to a public tool is a disclosure the employees never consented to.

Question 9

Would this leak sensitive data: Getting up to speed before a compliance meeting?

Accepted Answer

No, this is safe. A general knowledge question containing nothing about your organisation. This is the overwhelming majority of day-to-day AI use, and it is why blanket blocking backfires: staff who cannot ask harmless questions at work will ask them on their phone.

Question 10

Would this leak sensitive data: Extracting obligations from a signed customer contract?

Accepted Answer

Yes, this would leak. Executed contracts almost always carry a confidentiality clause covering the agreement itself. Pasting it discloses your counterparty's name and negotiated terms, which is usually a breach of the very agreement you are reading.

Would this leak?

What counts as a leak

Related resources

The CISO's AI Security Policy Checklist

AI Data Classification Framework Template

Enterprise AI Acceptable Use Policy Template

Areebi vs Nightfall AI

AI Governance for Internal Knowledge Management

Shadow AI Discovery & Remediation Playbook