---
title: "Why a Chatbot Can Be Confidently Wrong"
description: "A sentence that sounds likely and a sentence that is true are not the same thing, and the model has no way to tell them apart. Invented facts, names and references come out in exactly the tone of corr"
canonical: https://lightmysky.com/learn/computing/why-a-chatbot-can-be-confidently-wrong-mt_TMHNkprrks
source: https://lightmysky.com/learn/computing/why-a-chatbot-can-be-confidently-wrong-mt_TMHNkprrks.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Why a Chatbot Can Be Confidently Wrong

A sentence that sounds likely and a sentence that is true are not the same thing, and the model has no way to tell them apart. Invented facts, names and references come out in exactly the tone of correct ones.

Subject: Computing · Area: Artificial Intelligence · Ages 12 to 13
Page: https://lightmysky.com/learn/computing/why-a-chatbot-can-be-confidently-wrong-mt_TMHNkprrks

## Ready when they can

- Give an example of an answer that reads well and is wrong, and say what would have caught it
- Say why the confidence of an answer carries no information about whether it is right
- Check one claim from a model's answer against a source that is not the model

## Lesson: Confident words are not proof

A sentence that sounds likely and a sentence that is true are different things. The model picks likely words, and it has no way to feel the difference between them.

**Example.** Ask for a book reference and you may get a real-sounding title, author and year that never existed. It arrives in exactly the tone of a correct answer, calm and detailed.

Confidence carries no information about truth. A smooth paragraph can be fully invented, so ask what would have caught the error before you trust a single claim in it.

**Tip.** Pick one claim from any model answer and check it against a source that is not the model. If that claim fails, check the rest the same way.

**Recap.** Likely wording is not truth, so every claim worth using gets checked outside the model.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [How a Chatbot Picks Its Next Word](https://lightmysky.com/learn/computing/how-a-chatbot-picks-its-next-word-mt_1Zbhamhzai)
- [Searching and Evaluating Online](https://lightmysky.com/learn/computing/searching-and-evaluating-online-mt_S-XboPwA8k)
- [Online Identity and Misinformation](https://lightmysky.com/learn/personal-and-social-development/online-identity-and-misinformation-mt_WtO50EZQkf)

## Opens up

- [Using an AI Tool on Schoolwork Without Cheating Yourself](https://lightmysky.com/learn/computing/using-an-ai-tool-on-schoolwork-without-cheating-yourself-mt_fjlXx-hkp_)