A language model can give a useful explanation of a subject and then cite a paper that doesn’t exist. For example, you might ask about a research method and get a clear description of how it works, followed by a reference with a title, authors, and journal. When you look it up, you find that the journal and some of the authors are real, but they don’t belong to a paper with that title.
Errors like this are usually called AI hallucinations. The term includes false factual claims and statements that are unsupported by the material the model was asked to use. A summary that adds a finding that wasn’t in the original report is another example. In both cases, the answer can be written clearly enough that the error is difficult to notice without checking the source.
There are several reasons these errors happen. The model may have learned incorrect information, encountered too little information about a subject, or misinterpreted a document supplied with the question. Its training also affects whether it answers when it is uncertain. Understanding these processes helps explain why a model can be useful on one question and unreliable on a closely related one, and why adding a citation or asking it to double-check doesn’t always resolve the problem.
How language models generate text
Many chatbots use language models whose initial training, called pretraining, involves repeatedly predicting the next piece of text. These pieces are tokens, which can be words, word fragments, or punctuation. At each position, the model assigns probabilities to possible next tokens and compares those predictions with the token that actually appears in the training example. Training adjusts its internal numerical settings, or parameters, to make the observed text more likely.
To do this well, the model needs to learn relationships in the text. A sentence about a country’s capital is easier to complete correctly if the model has learned which city is associated with that country. In a longer explanation, it also needs to keep track of the subject and how the ideas relate to one another. Across many examples, training teaches factual associations along with grammar and patterns of explanation. Neural language models use these learned relationships to produce sentences they never encountered verbatim, which allows them to explain a familiar topic in response to a new question.
When you ask a question, the model uses its parameters and the text available in the conversation to generate a response. It selects a token, adds it to the answer, and predicts again with that addition included in the context. Unless a separate search or retrieval tool is involved, the answer may have no corresponding source passage that you can open and inspect. Tracing generated answers to their sources is one of the problems researchers have tried to address by combining language models with document retrieval.
For a journal reference, the model may have learned the names of researchers who work on a topic, the kinds of titles their papers have, and the journals that publish related work. It can use those associations to generate a reference even when it has not reliably learned the specific paper being requested. Each part can look reasonable on its own. The error is in combining them into a record of a publication that never existed.
As the response continues, the generated text becomes part of the context for later predictions. If the model has already supplied an incorrect title, it may go on to produce a journal name or summary that fits that title. To verify the reference, we would need to locate a publication whose title, authors, and other details match.
The model can combine what it has learned into an accurate explanation even if that wording never appeared in its training data. This is useful when you ask it to explain a method in terms you understand or connect it to something you already know. A request for a reference requires additional facts about a particular publication, including its authors and findings. The model may have learned the method well enough to explain it without reliably learning those details.
What the model learns about facts
The training examples contain human writing, including errors, fiction, outdated information, and repeated misconceptions. During pretraining, the model is evaluated on how well it predicts that text. If a false statement appears in the examples, predicting it can improve performance on the training task. The probability assigned to a token therefore describes how likely that continuation is in context. It doesn’t directly tell us how likely the completed claim is to be true.
Researchers tested one consequence of this in the original TruthfulQA study. They asked questions designed to elicit common human misconceptions and found that the models often reproduced them. The larger models tested were not consistently more truthful. This study evaluated particular models and questions in 2022, so its results should be interpreted in that setting. It provides an example of models learning false answers that are common in human writing.
The amount of relevant information in the training data also affects factual performance. Kandpal and colleagues examined this by counting training documents that mentioned both the subject of a factual question and its answer. Across the settings they studied, questions with fewer relevant training documents were harder to answer accurately. Larger models did better at learning these uncommon facts, although the facts remained difficult when there was little information about them in the training data.
This helps explain why a model might give a good overview of a research area and then make mistakes in a specific reference. The main concepts may appear across textbooks, lectures, and articles. The combination of an author’s name, a paper title, and a publication year may occur much less often. Asking for those details tests information that the broad explanation may never have required. We should check them even when the earlier explanation was accurate.
Information can also become outdated after training. For example, a model may have learned the requirements in an old policy and continue to describe them after the policy changes. Giving it today’s date doesn’t provide the revised requirements. To answer a question about the current policy, it needs access to the updated information and must use that information correctly. This is one reason it matters whether a response was generated from training alone or with access to current sources.
A model’s wording gives us limited help in identifying these problems. It can use the same writing style for a well-learned fact and a detail it gets wrong. Ideally, the system would recognize when it lacks enough information and respond by asking a question, searching for a source, or explaining what it cannot establish. That behavior depends partly on training beyond the initial prediction task.
How training affects uncertain answers
Chatbots typically receive additional training after pretraining. Examples and human feedback can shape how they follow instructions and respond to requests, including how they handle uncertainty. The model needs to learn when an answer is supported, when it needs more information, and when it should decline to make a claim. Teaching it to write “I don’t know” provides a possible response, but it still has to use that response appropriately.
The way answers are scored can lead developers to favor models that guess. Imagine a test that gives one point for a correct answer and zero for both a wrong answer and “I don’t know.” When a model attempts an uncertain question, it has some chance of receiving a point. Declining the question always receives zero. Over many questions, attempting uncertain answers can improve the total score while also increasing the number of errors. If developers use that score to choose between models or judge improvements, they can end up favoring a model that guesses more often. OpenAI’s analysis of hallucinations describes this incentive and argues for evaluations that treat an appropriate admission of uncertainty differently from an incorrect answer.
Uncertainty can concern just one part of a response. A model might be able to explain a research method while being unable to identify the first paper that used it. In that situation, it could give the explanation, state that it hasn’t verified the attribution, and search for a source if needed. This would let the reader use the explanation while knowing which detail still needs checking. An invented citation makes that distinction difficult to see because it presents the unverified detail as if it were established.
Sometimes the question itself includes an assumption that needs checking. Suppose you ask, “What year did this company win the award?” Before supplying a date, the system needs to establish that the company won. If it accepts the question’s premise without checking, it may generate an answer about an event that never happened. Searching an award record could resolve this. The useful response might then explain that no matching award was found, or ask whether you meant a different company or award.
Instructions such as “be accurate” or “don’t make anything up” can request careful behavior, but they don’t provide missing facts. They also don’t guarantee that the model will recognize when more information is needed. The response has to depend on the particular question: an ambiguous request may need clarification, while an obscure fact may need a source. Even after a search, some questions will remain unanswered by the available evidence.
Using sources to answer a question
In retrieval-augmented generation, or RAG, a search process finds relevant material and supplies it to the model as context for its response. That material might be a webpage, an internal policy, or a database record. The model can use it to answer questions about information it did not reliably learn during training. It can also use revised documents to answer questions about changes that happened later.
For a question about a paper, retrieval can supply the publication itself. The title, authors, and publication details can be taken from that record, and the abstract or full text can be used to explain what the researchers did. This gives both the model and the reader a document to consult. It also makes specific checks possible, such as comparing a claim about the study’s results with the results the paper actually reports.
The quality of the answer still depends on which document was found and how the model uses it. Search might return an older policy, a paper with a similar title, or a passage that discusses the topic without answering the question. Once the material is supplied, the model must identify the relevant parts and represent them accurately. The RAGTruth study documents unsupported claims and contradictions in responses generated with retrieved material, so access to a source alone doesn’t establish that an answer is supported by it.
For example, suppose a travel policy allows reimbursement only when a trip was approved in advance. A response that says “Travel is reimbursable” leaves out a condition that affects whether someone can receive reimbursement. It could link to the correct policy and still give incomplete advice. To answer accurately, the model would need to include the approval requirement. If the question concerned a particular trip, it would also need to establish whether that requirement had been met.
Researchers use factuality to describe whether a claim is true and faithfulness to describe whether an answer accurately represents its source. Maynez and colleagues examined these issues in generated summaries. A summary could accurately describe an outdated policy and still be wrong about what is required today. It could also add a detail that happens to be true but that the supplied document never established.
Suppose a report describes a treatment but gives no information about the outcome. A summary that adds “the patient improved” introduces a claim that cannot be supported from that report. Even if the patient did improve, someone reading the summary could reasonably assume that the outcome was documented in the source. A faithful summary would describe the treatment and state that the report did not give an outcome. This preserves the information available to the reader without filling in an unknown detail.
Assessing uncertainty and errors
Researchers also study whether a model’s responses can help identify answers that need checking. One approach compares several answers to the same question. If the model gives different dates for the same event, the disagreement is a reason to investigate further. We still need another source to establish which date is correct.
Comparing the wording alone would count many harmless differences. “It happened in 1998” and “The event took place in 1998” give the same answer. “It happened in 2003” gives a different one. Semantic entropy accounts for this by measuring uncertainty across meanings. Farquhar and colleagues developed a method that generates separate answers to the same question, groups them by meaning, and combines the probabilities of answers within each group. The uncertainty measure is low when the model assigns most of the probability to one meaning, and higher when that probability is spread across several different answers. It helped detect a subset of hallucinations in which answers vary arbitrarily, and could be used to flag unreliable responses or withhold an answer.
The method has limits because some errors are consistent. A model may repeat a misconception each time it is asked, producing little disagreement despite being wrong. Asking the model to check its previous response may also reproduce the original error. Checking an original document or repeating a calculation separately gives us additional evidence to compare with the model’s answer.
To assess whether withholding uncertain answers improves reliability, we also need to know how many questions the system declined. If it declines difficult questions, its accuracy on the questions it does answer may rise. To understand that improvement, we need to know both how often the system answers and how often those answers are correct. Otherwise, we could mistake a system that answers only a small set of easy questions for one that performs equally well across the full set.
A reported hallucination rate also depends on what the evaluation counts as an error. A brief factual answer may contain one checkable claim, while a research summary contains many. FActScore breaks long answers into individual factual claims and assesses their support. The resulting score can show how much of an answer is supported. For a hypothetical paragraph with nine supported claims and one invented result, a claim-based measure would record nine supported claims out of ten. A response-based measure might mark the entire paragraph wrong because it contains an error. The reported rate needs to specify which of these is being measured.
These methods help researchers compare systems and identify certain kinds of unreliable answers. When using a response yourself, you will usually need a more direct check of the claims relevant to your work. A low overall error rate cannot verify a particular citation or establish that a summary accurately describes the document you supplied.
Checking an answer before using it
For a journal reference, start by finding the work independently. Search for the title and authors, then confirm that the publication details match. If you cannot locate the paper, treat the citation as unverified. Asking the model for more details may help you search, but any new details need checking too. They may have been generated from the same incorrect information as the original reference.
Once you have the paper, read the passage that is supposed to support the answer. Compare what the model said with what the researchers reported. Check who was studied, what was measured, and what conclusions the authors drew. An answer may describe most of these details correctly while overstating one result. In that case, you can correct the conclusion and keep the supported information, with a citation that lets another reader check it.
The same process applies to other kinds of answers. Reproduce a calculation with a calculator or code, and check current factual claims against dated sources. When reviewing a summary, pay attention to causes, outcomes, quantities, and exceptions that the model adds or leaves out. Asking it to identify supporting passages can help organize the comparison, but you still need to open the source and check those passages yourself.
Some questions remain unresolved after checking. A paper may describe a method without evaluating the outcome you asked about, or a policy may leave a particular situation unspecified. The answer should explain what the source establishes and what remains unresolved. If you need the missing information to make a decision, you will need another source or further investigation before you can use the answer.