Fluent answers can still fail

A fluent answer can be wrong. Pretraining rewards prediction of the observed text, while post-training can reward instruction-following, preferred responses, and correctness on particular tasks. None of those objectives guarantees that every generated claim is true.

Five failure patterns

These written examples illustrate different failures. They are not live outputs or measurements of how often each failure occurs.

Hallucination. An answer presents a false or unsupported claim as fact—for example, a plausible paper title that does not correspond to a real paper. Check the claim against evidence rather than treating fluent wording as evidence.

Sycophancy. The model agrees with a user’s mistaken premise or changes its answer to match the user’s preference without adequate reason. Feedback that favors agreeable responses can contribute, but agreement is not the sole cause of every such error.

Representational harm. An answer reproduces a stereotype, excludes a group, or describes it unfairly. Evaluate concrete outputs and relevant groups rather than assuming one overall accuracy score covers this behavior.

Privacy leakage. Some training passages can be memorized and later reproduced. The possibility and frequency depend on the data, model, and prompts; repetition does not imply a simple threshold above which every passage will be extractable.

Over-trust. A confident explanation can encourage reliance beyond what its evidence supports. A useful response should make the supporting information inspectable when the task requires it.

What can improve reliability?

Better data, training feedback, retrieval, tools, and evaluation address different parts of the problem. These changes can affect the model itself as well as the surrounding application. Instruction tuning with human feedback, for example, improved measured truthfulness in the reported experiments while leaving mistakes.

Retrieval provides external evidence, but a citation can be irrelevant or misrepresented. Running code tests checks the tested behavior, not every possible behavior. A generated explanation can be useful without being a faithful account of the model’s internal computation.

For a concrete answer, separate three questions: What is the claim? What evidence supports it? What would show it is wrong? That is more useful than treating the model as either always trustworthy or incapable of useful work.

Return to the AI overview or explore post-training.

Sources · 6

1. Cheng, M., Yu, S., Lee, C., Khadpe, P., Ibrahim, L., & Jurafsky, D. (2025). ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs. arXiv:2505.13995.
2. Nadeem, M., Bethke, A., & Reddy, S. (2021). StereoSet: Measuring Stereotypical Bias in Pretrained Language Models. In Proceedings of ACL-IJCNLP 2021 (Vol. 1: Long Papers, pp. 5356–5371). arXiv:2004.09456.
3. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650. arXiv:2012.07805.
4. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 7 (Large Language Models); Ch. 10 (Masked Language Models).
5. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.11401.
6. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., et al. (2021). Ethical and social risks of harm from Language Models. DeepMind. arXiv:2112.04359.

Full bibliography →