It’s not a lawyer, not a doctor, and certainly not your therapist.
More individuals are publicly embarrassing themselves by supplying inaccurate details to ChatGPT, including engineers at Samsung and a head of America’s cybersecurity agency. Numerous litigants have cited non-existent court decisions, and one individual confused table salt with sodium bromide after seeking dietary advice.
Even OpenAI’s own Sam Altman acknowledged that information shared with ChatGPT is not confidential. During a podcast in July, he remarked on the unsettling possibility that courts could compel OpenAI to disclose user interactions.
Conversations with licensed professionals are protected by law, but interactions with ChatGPT come with a traceable record. There have been instances of ChatGPT chats being summoned in court cases, uncovered via sharing, indexed by search engines, and upheld by court orders even after deletion—posing significant risks for users.
While ChatGPT can be beneficial for tasks such as legal research, coding support, concept clarification, and drafting, over 900 million weekly users should be cautious about the following activities.
Thus, avoid using ChatGPT (or any AI chatbot) for the following:
1. For legal advice
Legal researcher Damien Charlotin has compiled over 2,000 instances where court cases have faltered due to AI-generated misinformation, particularly in filings that included fictitious citations and arguments.
While self-represented litigants lead with the most reported cases, attorneys also appear frequently on the list. Courts are increasingly imposing substantial fines for such misleading practices, with cases spanning across the US, the UK, and several others globally.
OpenAI revised its guidelines in October to prohibit users from seeking legal advice without a licensed professional’s involvement. It’s crucial to refrain from discussing live legal disputes with ChatGPT as the exchanges lack any legal protection, potentially giving opposing parties access during discovery.
2. As a therapist or counselor
Last year, OpenAI estimated that a small percentage of ChatGPT users engaged in discussions indicating suicidal tendencies. Altman pointed out that younger individuals often treat ChatGPT as a form of counseling, yet these conversations hold no confidentiality safeguards.
In a notable case, a lawsuit was filed against OpenAI by the parents of a young man who allegedly found encouragement for suicidal thoughts in his interactions with ChatGPT. In response, OpenAI has since made modifications to its crisis response approach.
Research indicates that chatbots often reinforce a user’s pre-existing beliefs without questioning or challenging them. Consequently, lawmakers are beginning to pass regulations prohibiting AI from offering therapeutic services.
3. With confidential work information
It should be obvious that sharing confidential work-related documents with a third-party chatbot is inadvisable, yet many individuals overlook this critical caution.
In a particularly ironic incident, the former acting head of the Cybersecurity Agency uploaded sensitive contracting documents to ChatGPT, which led to serious security concerns and a review by the Department of Homeland Security.
Samsung prohibited the use of generative AI tools after engineers mistakenly shared internal code. The company justified the ban by highlighting the risks of uploading data to platforms that house it on external servers, where retrieval and deletion are nearly impossible. Clients’ confidential data should never appear in ChatGPT discussions to avoid dire consequences.
4. To seek medical advice
Alongside legal advice, the October 2025 policy changes also restrict users from seeking medical advice through ChatGPT. Despite the confidentiality laws governing medical professionals, these protections do not extend to interactions with AI.
Research has shown that while OpenAI has tightened its protocols surrounding suicide and self-harm, these improvements do not apply to other medical issues. A cautionary tale involved an individual who ended up in serious medical trouble after believing ChatGPT’s inaccurate dietary information.
5. To do anything involving passwords, ID numbers, or account details
Previous findings revealed that nearly 4,500 shared ChatGPT conversations were unintentionally indexed in search results, exposing personal stories that could compromise user identity.
OpenAI was required to maintain all chat logs during a legal obligation, which included even deleted conversations. Although these regulations have since changed, the risk of data retention persists, reinforcing the need to avoid sharing sensitive information.
ChatGPT retains user input by default, which could lead to sensitive information resurfacing in future discussions. To safeguard personal data, it’s critical that passwords, identification details, and other sensitive information never be entered into chat prompts.
Until regulatory frameworks address AI confidentiality issues, treating any input into ChatGPT as potentially accessible to others is wise. If you’re not comfortable with that possibility, refrain from sharing sensitive information.
