Why a Chatbot Can Be Confidently Wrong
A sentence that sounds likely and a sentence that is true are not the same thing, and the model has no way to tell them apart. Invented facts, names and references come out in exactly the tone of correct ones.
What a learner can do afterwards
- Give an example of an answer that reads well and is wrong, and say what would have caught it
- Say why the confidence of an answer carries no information about whether it is right
- Check one claim from a model's answer against a source that is not the model
1 · Read
A sentence that sounds likely and a sentence that is true are different things. The model picks likely words, and it has no way to feel the difference between them.
Ask for a book reference and you may get a real-sounding title, author and year that never existed. It arrives in exactly the tone of a correct answer, calm and detailed.
Confidence carries no information about truth. A smooth paragraph can be fully invented, so ask what would have caught the error before you trust a single claim in it.
Pick one claim from any model answer and check it against a source that is not the model. If that claim fails, check the rest the same way.
Likely wording is not truth, so every claim worth using gets checked outside the model.
2 · Watch
Take it off screen
Where it sits
Where this leads
Jobs that lean on this skill. Follow one to see everything it is built on.
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.