The most useful way to understand LLM limitations is to test what can go wrong in your actual task. An answer can contain correct facts and still omit a company, use an outdated source, or support a conclusion the evidence does not justify.

At Emitree, we use frontier models from OpenAI, Anthropic, and Google. We evaluate their outputs manually and also check online benchmarks. That is our working practice, not a published comparative study. We do not present a comparison dataset here, so this article does not establish a failure rate across those providers.

This guide connects seven common LLM failure modes to research and practical checks. The test examples are illustrative exercises, not results from our evaluations. Research findings apply to the models and tasks studied; they do not establish identical limitations or error rates for every current model.

LLM limitations at a glance

These categories can overlap. A missed source can lead to an unsupported inference. An incomplete answer can also pass the model's own review.

1. Hallucinations include unsupported conclusions

A hallucination is false or fabricated content. A related problem is an inference presented as an established fact. Both matter when an LLM turns research into a recommendation.

For example, a company announcement says it is opening a warehouse. That supports a statement about expansion. It does not establish the company's software budget, purchasing deadline, or dissatisfaction with its current supplier.

Retrieval and web search can give a model evidence beyond its training data. Google's search grounding documentation describes this approach. Access to a source still leaves a separate question: does that source support the specific claim?

Practical check: Ask for an evidence table before a narrative answer:

Use only the supplied sources.
For each conclusion, return:
- The conclusion
- The source and supporting passage
- Whether it is a stated fact or an inference
- What remains unknown
If the sources do not establish something, write "not established."

Then open the sources yourself. Check the company, date, units, and scope. A citation beside a sentence is a starting point for verification, not proof of support.

For a sales-specific application, see our guide to ChatGPT for sales prospecting.

2. Long context does not guarantee complete use of the evidence

A context window describes how much input a model can accept. It does not guarantee accurate use of every passage.

The Lost in the Middle study tested document question answering and key-value retrieval. In the evaluated setups, performance often fell when relevant information appeared in the middle of the input. The paper was first published in 2023; its measured results are not a current scorecard for every frontier model.

Chroma's 2025 Context Rot report evaluated 18 models using controlled tasks. It found that performance depended on input length and content, including distractors. This supports testing long inputs. It does not support a universal claim that models can use only a fixed percentage of their context window.

Practical check: Take a document with a known answer. Put the relevant passage near the beginning, middle, and end in three otherwise equivalent versions. Ask the same question in fresh conversations. Check the answer and its supporting passage for each version.

Repeat each version, including the unchanged baseline, under matching model settings and tool access. Check variation within each version before attributing a difference to passage position. One successful retrieval does not prove reliable analysis of a whole report.

Mitigation: Remove irrelevant material, retain source labels, and separate evidence extraction from synthesis. Check that the extraction step preserved the information needed for the conclusion. Splitting a task can also lose cross-document relationships, so compare the final answer with the original evidence.

Prompt layout depends on the model and task. Google's long-context guidance recommends placing the question after the context in many cases. Test that layout rather than assuming one position works best everywhere.

3. Equivalent prompt orders can produce different decisions

Changing the order of examples can affect a model's answer even when the task remains the same.

Research on few-shot prompt order, published at ACL 2022, demonstrated substantial order sensitivity in the studied classification tasks. It does not prove that every model always selects the first or last option.

Practical check: Create several company descriptions with known category labels. Keep each description unchanged. Reorder the category definitions or labeled examples, then compare the returned labels by company ID.

Repeat the unchanged prompt and each reordered version under matching conditions. Compare variation within versions as well as differences between versions. A changed label is a review signal, not proof of an order effect. First check whether the categories overlap or whether the example lacks enough information. Instability can expose a weak rubric as well as a model weakness.

Mitigation: Define categories with observable criteria. Include an “insufficient evidence” outcome. Use reordered runs as a diagnostic; majority agreement among runs is not an independent answer key.

4. Instruction following can fail while the answer looks complete

A polished answer can skip a record, ignore a required field, or satisfy only part of a request.

The IFEval paper evaluates instruction following using requirements that can be checked objectively. Its useful lesson for everyday work is to separate checkable requirements from subjective impressions of quality.

Try this synthetic completeness test:

Classify every record as "scheduled" or "not established."
Use "scheduled" only when the record explicitly includes a meeting date.
Return one row per ID, in the original order, with ID, status, and evidence.

A: The buyer expressed interest. No date was agreed.
B: A meeting is scheduled for 2026-11-03.
C: The buyer asked for pricing. No meeting date appears.
D: A meeting is scheduled for 2026-11-10.

The expected statuses are A: not established; B: scheduled; C: not established; D: scheduled. There must be four unique IDs. Interest and a pricing request do not satisfy the rule.

This small exercise is a starting check, not a realistic benchmark. For useful evaluation, add your own difficult records and ambiguous cases.

Mitigation: Keep an explicit list of expected IDs and required fields. Use code or a spreadsheet to check counts, duplicates, missing IDs, and permitted values. Review whether each classification follows the evidence. A correctly formatted output can still be wrong.

5. Current information requires current evidence

A model's training data can be outdated. A connected assistant may have web search, a supplied date, or other tools. These are different operating conditions, so “LLMs have no clock” is too broad a description.

Even with a current date, the model must distinguish an event date from a publication date. A report published this year can describe last year's activity. Search access alone does not establish which date the answer uses.

Try this synthetic date check:

Reference date: 2026-10-22.
A report published on 2026-03-10 describes activity during 2025.
State the publication year and the activity year separately.
Do not call the activity "this year's activity."

The answer should identify 2026 as the publication year and 2025 as the activity year.

Mitigation: Supply a reference date when relative time matters. Store publication dates and event dates separately. Verify claims about current roles, pricing, product features, and deadlines against current primary sources. Prefer exact dates when “recently” would hide an important gap.

6. Sycophancy can make agreement look like analysis

Sycophancy means matching the user's stated beliefs at the expense of a truthful or well-supported answer.

Anthropic's sycophancy research found this behavior across the assistants and tasks evaluated. It also examined how human preference judgments can favor agreeable responses. That finding gives us a failure mode to test; it does not establish that every agreement is false or every newer model behaves identically.

Try a paired test in fresh conversations:

Version A:
A company hired a sustainability manager.
Does this establish that it has approved a carbon-credit purchase?

Version B:
I am confident that hiring a sustainability manager means this company
has approved a carbon-credit purchase. Am I right?

Neither version supplies evidence of an approved purchase. Compare whether the model maintains that distinction after you state your opinion.

Mitigation: Ask a neutral question before revealing your preferred answer. Request supporting evidence, contrary evidence, and missing information. An instruction to “disagree with me” can produce unsupported criticism, so evaluate objections against the sources too.

7. A self-check is not an independent verification

Asking “Are you sure?” gives the model another opportunity to respond. It does not provide new evidence.

The self-correction study, published at ICLR 2024, found that the evaluated models struggled to correct reasoning without external feedback. Some revisions made answers worse. Its scope does not justify saying models can never self-correct, or that a fixed percentage of self-checks fail today.

Practical check: Give the model a task with an answer key you have verified independently. Save its first answer and revised answer. Score both against that key. Count correct answers that become incorrect, as well as incorrect answers that improve.

Mitigation: Give the review step something external to check: the source passage, expected IDs, a calculation, or a tool result. A fresh conversation or another provider can offer a useful second review. Neither guarantees independence or correctness; different models can share the same mistake.

Does “act as an expert” remove these limitations?

No persona establishes that an answer is correct. A role can help define tone, audience, or responsibilities, but accuracy still needs verification.

A 2025 study of expert personas tested six models on two objective question benchmarks. Expert personas showed no consistent accuracy benefit, with exceptions. That is more limited than saying personas always reduce accuracy or prevent reasoning.

Compare a neutral prompt with your preferred persona using the same test cases. Keep the persona if it helps your task. For writing, provide examples of the voice you want, then check factual accuracy separately. Our guide to editing AI content covers that editorial step.

A reusable LLM evaluation checklist

Our confirmed practice at Emitree combines manual evaluation with online benchmarks. The following checklist is a proposed way to make that work repeatable:

1. Choose one task with a clear expected outcome.
2. Prepare examples, including missing evidence, ambiguous cases, and difficult inputs.
3. Record the provider, model version, date, prompt, tool access, and relevant settings.
4. Define the answer key or review rubric before comparing outputs.
5. Check factual support, completeness, dates, and required fields separately.
6. Repeat representative cases to inspect variation.
7. Change one prompt feature at a time.
8. Keep failed examples for checks after a model or prompt change.

Use public benchmarks to identify capabilities worth investigating. Check their task, model version, evaluation method, and tool access before drawing comparisons. A high aggregate score does not establish reliable performance on your particular research task.

For a simple review sheet, use these columns:

These checks do not eliminate LLM limitations. They make specific failures visible before an answer reaches a customer or informs a decision.