Critical Thinking When AI Has an Answer

An AI answer can arrive just before a meeting, written in a tone that sounds settled and complete. You might use it to recommend a supplier, explain a change in customer behaviour, or prepare a staffing proposal. The risk is not only that a detail is wrong. The answer can mix facts, explanations, assumptions, and recommendations so smoothly that you cannot see which parts the available information actually supports. This chapter gives you a practical way to inspect that mixture before it shapes a work decision. You will learn to separate claims, reasons, evidence, and conclusions, then judge whether evidence is relevant and sufficient for the decision at hand. You will practise finding assumptions hidden in fluent answers and recording unanswered questions that could change your recommendation. The examples focus on professional decisions with real trade-offs, where speed matters but so do the people affected, the quality of the evidence, and the cost of being wrong. The aim is not to reject AI answers automatically. It is to use them with judgment. By the end, you can identify what an answer asserts, what supports those assertions, what remains uncertain, and what you need to check before acting.

Separate the parts before judging the answer

A claim is a statement someone wants you to accept. It can describe what is happening, explain why it is happening, or predict what will happen. A reason is offered to make a claim more credible. Evidence is information that can be checked and that bears on the claim. A conclusion is the decision or judgment drawn from the claims and reasons. These parts often appear together in one fluent paragraph. For example: “Customer complaints rose after the support team changed its schedule, so the new schedule is driving dissatisfaction. Restore the old schedule.” The first sentence contains a claim about a change in complaints and a causal explanation. The second gives a recommendation. But no evidence is shown for the size or timing of the change, and the answer does not rule out other causes. A reason is not automatically evidence. “Customers value fast replies” might be a sensible reason to care about response time. It does not prove that a particular schedule caused a particular rise in complaints. Evidence could include complaint records over time, matched with schedule changes and other relevant events. Even then, the evidence might support only a narrower claim, such as a rise in complaints about delayed replies.

Sort the answer into a chain

A useful way to inspect an answer is to lay out its reasoning as a chain. Start with the conclusion, because that is often what will influence your action. Then ask which claims must be true for that conclusion to make sense. For each claim, identify the reason offered and the evidence that could be checked. Mark explanations that have not yet been tested as explanations, not facts. The chain is not always linear. An answer may use several claims to support one recommendation, or one piece of evidence to support several claims. Your job is not to force it into a tidy argument. Your job is to make the connections visible and notice where a link is missing. Keep the wording close to the answer when you record a claim. If the answer says “costs may fall,” do not rewrite it as “costs will fall.” That small change turns a qualified possibility into a prediction. If the answer says “the survey suggests,” preserve that qualification rather than reporting that customers have decided.

Worked example: a service schedule proposal

Imagine a manager at a large service organization asks an AI tool whether a proposed change to support-team hours will reduce operating costs without harming customers. The answer says: “The change should reduce costs because fewer staff will be scheduled during quiet periods. Customer demand is usually lower then, so service quality should remain stable. Adopt the schedule for all teams.” The answer contains three distinct claims: the change will reduce costs, demand will be lower during the affected hours, and service quality will remain stable. Its reason for the first claim is that fewer staff will be scheduled. Its reason for the second appears to be a general pattern about quiet periods, but the answer provides no local demand evidence. Its third claim depends on the second and on an unstated belief that the remaining staff can handle the work. The conclusion is to adopt the schedule for all teams. An ambitious professional preparing a recommendation would not treat the recommendation as established by the confident tone. They would ask for local staffing costs, demand by hour, wait times, customer outcomes, and differences between teams. They would also ask whether the proposed schedule shifts work to other hours or creates pressure elsewhere.

Apply the sort in your own work

• Write down the action the answer recommends, if it recommends one.
• Underline each statement that could be true or false. Treat each as a separate claim.
• For each claim, note the reason the answer gives.
• Identify the specific information that could verify or challenge the claim.
• Draw a line from each supported claim to the conclusion it helps justify.
• Mark any jump in reasoning, such as moving from a general pattern to a local prediction.

This takes only a few minutes for a short answer. For a long answer, start with the claims that matter most to the decision. A minor background statement may not need the same scrutiny as a prediction that would change staffing, customer commitments, or investment.

Mistakes that blur the reasoning

A common mistake is to treat a plausible explanation as evidence. Another is to accept a list of reasons as proof even when none of the reasons is supported by checkable information. Watch for a conclusion that is broader than the claims beneath it. Evidence about one team, for example, does not by itself justify a decision for every team. Do not assume that a source named by an AI answer supports the exact sentence beside it. Check what the source actually says, when the information was collected, and whether it applies to the decision you face. Also resist polishing an uncertain claim into a firm statement when you brief colleagues. Your summary should preserve the answer’s limits, not hide them.

Evidence must fit the question

Evidence is relevant when it bears on the specific claim or decision you are evaluating. Information can be accurate and still be irrelevant. A broad industry article about customer preferences may not answer whether customers at your organization will accept a particular service change. A total number of complaints may not explain whether a change affected complaint rates if customer volume also changed. Sufficiency is a separate question. Evidence is sufficient when it provides enough support for the particular decision, given its consequences and uncertainty. There is no single amount of evidence that works for every decision. A reversible pilot with limited impact may reasonably require less support than a decision that would affect many people, be hard to undo, or create substantial cost. The relevant standard is not “Do I have a lot of information?” It is “Could this information reasonably support this claim and this level of action?” Strong-looking evidence can still be a poor fit. A detailed spreadsheet may use the wrong population, measure the wrong outcome, or cover a period that does not reflect current conditions. A small number of direct customer accounts may reveal a problem worth investigating, but may not show how common it is. Ask what the evidence can establish, and what it cannot.

Test evidence against the claim

Use four practical questions. First, does the information measure the thing named in the claim? If the claim concerns customer retention, evidence about satisfaction may be related but is not the same measure. Second, does it come from the people, teams, locations, or time period the decision concerns? Third, is the information credible and traceable enough for the stakes? Fourth, does it distinguish the proposed explanation from plausible alternatives? The final question matters when an answer claims that one event caused another. A change happened, and an outcome changed afterward. That sequence may be worth investigating, but other events could also explain the outcome. Check whether the answer considers changes in demand, staffing, measurement, or the mix of cases being handled. Where a cause cannot be isolated, state the conclusion at the level the information supports. “Errors fell after the guide launched” is more defensible than “The guide caused errors to fall” if other explanations remain open.

Worked example: deciding whether to expand a training pilot

Imagine a learning and development lead at a large organization asks an AI tool whether to expand a new training programme. The answer cites positive comments from participants and says the programme improved job performance. It recommends making the programme standard across teams. The comments are relevant to participants’ experience. They do not directly show that performance improved. To support that claim, the lead would need an appropriate measure of the work the training was meant to affect. They would also need to know whether the people who provided comments represent the people who will receive the programme more broadly. If the pilot included volunteers who were already engaged, their reaction might not predict the experience of colleagues with different roles or constraints. A leader facing this choice has to weigh the cost of delay against the risk of overextending a programme whose value is not yet clear. A measured decision could be to continue the pilot while collecting job-related evidence and checking whether teams with different work patterns can use the material. That is not indecision. It is a choice to match the scale of action to the strength and reach of the evidence.

Check relevance, then judge sufficiency

• Restate the claim in precise terms. Avoid vague claims such as “the change worked.”
• Name the outcome that would show whether the claim is true.
• Check whether the evidence measures that outcome directly or only a related signal.
• Check whether the evidence represents the people and conditions covered by the decision.
• Look for missing comparisons, changes in measurement, or plausible alternative explanations.
• Match the strength of your conclusion and the scale of your action to what the evidence can support.

If evidence is relevant but too limited, record what would strengthen it. That might mean checking another source, comparing outcomes across groups, clarifying how a measure was collected, or testing a change on a limited basis. The question is not whether the evidence is perfect. It is whether its limits are acceptable for the decision you are considering.

Common evidence traps

Do not confuse volume with quality. Many weak sources can repeat the same unsupported claim. Do not confuse an example with a pattern. One customer story may show a real experience, but not how often it occurs. Do not confuse a change in a measure with a change in the underlying outcome. A new way of recording incidents could increase recorded incidents even if the actual rate stayed steady. Another trap is demanding certainty before any action. Work decisions often happen under uncertainty. The disciplined response is to make the uncertainty visible, choose an action proportionate to the risk, and decide what evidence would prompt you to change course. That approach protects both momentum and judgment.

Fluency can conceal missing links

An assumption is something an argument treats as true without establishing it. Assumptions are not always unreasonable. Every decision depends on some things we do not know. The risk comes when an important assumption stays invisible and is presented as if the evidence had already settled it. An AI answer may state a result and a recommendation while skipping the conditions between them. “This option costs less, so the organization should choose it” assumes that the cost comparison includes all relevant costs, that the options provide acceptable quality, and that the lower-cost option does not create a larger burden elsewhere. “The customer feedback is positive, so the change will succeed” assumes that the feedback represents the affected customers and that stated preference predicts later behaviour. Look especially for assumptions about cause, representativeness, continuity, and capacity. Cause assumptions treat a sequence as proof that one event produced another. Representativeness assumptions treat a small or selected group as if it reflects a broader population. Continuity assumptions expect a past pattern to continue under changed conditions. Capacity assumptions presume people, systems, or suppliers can handle the recommendation. These are useful prompts for inspection, not formal categories you must force onto every answer.

Find what has to be true

A practical test is to ask, “What has to be true for this recommendation to work?” Start with the proposed action, then work backward. If the recommendation is to move a service to a new channel, what must be true about access, customer needs, staff capability, and the cost of supporting that channel? If any condition fails, would the recommendation still hold? Next ask, “What would change my mind?” This pushes you beyond collecting supportive information. A useful answer should identify evidence that could weaken the claim, reveal a different cause, or show that the proposed action has uneven effects across groups. If no possible evidence seems capable of changing the recommendation, the conclusion may be functioning as a preference rather than a reasoned decision. Finally, rank assumptions by consequence. Some are minor. Others sit directly beneath the decision. You do not need to investigate every possible unknown. Focus on assumptions that are both uncertain and important: if they prove false, the decision, its timing, or its reach would change.

Worked example: choosing a new supplier

Imagine a procurement lead at a large organization asks an AI tool to compare two suppliers. The answer recommends the lower-priced option. It says the supplier offers equivalent service and can meet the organization’s needs, but it gives no direct evidence for those statements. The recommendation may depend on several assumptions. The quoted prices may cover the same scope. The service may be equivalent in the areas that matter to users. The supplier may have enough capacity during busy periods. A change may not create significant transition work for the organization. The quoted terms may remain available through the decision and implementation period. The trade-off is not simply “save money” versus “spend more.” The leader must consider service quality, continuity, staff workload, customer effects, and the credibility of the comparison. A careful professional would confirm that the quotes cover comparable services, ask for relevant performance information, and involve the teams who would manage the change. If evidence about capacity is unavailable, that uncertainty might justify a limited commitment or a clear contingency rather than a broad, irreversible switch. The appropriate response depends on the consequences and what can be verified.

Surface and test the assumptions

• Circle words that imply certainty, such as “will,” “equivalent,” “typical,” or “all.”
• Ask what must be true for each important claim and recommendation to hold.
• Separate assumptions supported by evidence from assumptions that are merely plausible.
• Consider who or what is missing from the information, including affected teams and customers.
• Ask what observation or source could disconfirm the answer.
• Prioritize assumptions whose failure would change the decision, its scale, or its timing.

Turn the most important assumptions into questions you can assign or investigate. “Can the supplier meet demand?” is more useful than “Is the supplier good?” Ask what evidence would answer the question and who can obtain it. If you cannot resolve an assumption in time, state it openly and consider how to limit the consequences of being wrong.

Mistakes to avoid

Do not treat every assumption as a fatal flaw. A decision can be sensible even when some uncertainty remains. The goal is to distinguish manageable uncertainty from a hidden condition that could undermine the recommendation. Do not look only for assumptions in claims you dislike. The option you favour also rests on beliefs about costs, outcomes, people, and timing. Apply the same questions to every option. And do not confuse a question with a challenge to someone’s competence. Asking what must be true is a way to improve the decision, not to catch a colleague out.

Make the review useful to the decision

A claim and evidence review is a short working note, not a formal report. Its purpose is to prevent important distinctions from disappearing between reading an AI answer and discussing a recommendation. Keep it brief enough to use under time pressure, but specific enough that a colleague can see what you checked. Start with the decision, not with every sentence in the answer. What choice is in front of you? What would change if the answer were right or wrong? Identify the claims that bear directly on that choice. Then record the source or information offered for each claim, whether it has been checked, and what it actually supports. Mark assumptions and write unanswered questions in plain language. A useful note also records the limits of the evidence. A source can support a narrow claim without justifying the wider conclusion. For instance, a short pilot may show that a process can work for one team under certain conditions. It may not establish that the process will work across departments. Recording that boundary makes it harder for a careful finding to grow into an unjustified promise as it passes through meetings.

Use a short review sequence

• Decision: State the choice and who or what it affects.
• Claims: Write the answer’s most decision-relevant claims in precise, neutral language.
• Support: Record the evidence offered and where it came from. Note whether you checked the original source.
• Fit: Say what each piece of evidence measures, who it covers, and what it does not establish.
• Assumptions: List conditions the answer depends on but does not demonstrate.
• Open questions: Record what you still need to learn, especially anything that could change the choice.
• Judgment: State what action the evidence supports now, with any limit, condition, or review point.

This sequence is a practical aid, not a scoring system. Avoid assigning a numerical grade to evidence unless your organization has a sound reason and a clear method for doing so. A label such as “checked,” “uncertain,” or “not supported here” is often more informative than a number that suggests precision the review cannot provide.

Worked example: a proposed shift in recruitment

Imagine a people leader at a large organization is considering whether to use a new recruitment channel. An AI answer says the channel will produce a broader candidate pool at lower cost. It points to examples from other employers and recommends changing the organization’s approach. A review note might say: Decision: Whether to include the channel in recruitment for selected roles. Claims: The channel can reach candidates not reached through current sources. It will cost less per suitable candidate. The examples cited show this can happen elsewhere. Support and fit: The examples are relevant as possibilities, but they concern other employers. The answer provides no local comparison of cost, candidate suitability, or outcomes. The word “suitable” needs a clear definition for the roles in question. Assumptions: Candidates reached through the channel will meet role requirements. The channel’s results will transfer to the organization. Lower advertising cost will not be offset by additional screening work. Open questions: What does a comparable local test show? How will candidate suitability be assessed? Which roles would make a fair initial comparison? Judgment: The answer supports investigating the channel, but not claiming that it will lower costs or improve results locally. A limited test could be considered if the organization can define the comparison and review the result. This note gives leaders something better than an unqualified yes or no. It distinguishes what the answer suggests from what the evidence establishes. It also identifies a proportionate way to learn while limiting the reach of an uncertain recommendation.

Write questions that could change the decision

An unanswered question earns space in your note when a credible answer could alter the recommendation, its scope, its timing, or the safeguards around it. “Can the proposed channel reach qualified candidates for these roles?” could change whether it is adopted. “What colour is the vendor’s logo?” probably would not, unless a specific decision makes that relevant. Make questions answerable. Replace “Is this a good idea?” with questions about outcomes, costs, coverage, quality, or implementation conditions. Identify the information source and, where practical, the person best placed to obtain it. If a question cannot be answered before a decision is due, record the uncertainty and make its consequence explicit.

Mistakes that make a review less useful

A long list of every possible doubt can bury the two questions that matter. Keep the review focused on the decision. A second mistake is recording a source name without recording what it supports. A third is treating an unanswered question as a reason to stop automatically. Some decisions can proceed with limits or a way to revisit them. Others should wait because the unresolved point carries too much risk. Be equally careful with the final judgment. “The evidence is mixed” is not enough on its own. Say which evidence supports which claim, where it is limited, and what action that combination justifies. If different stakeholders face different consequences, name that trade-off instead of implying that one answer serves everyone equally.

A senior team is considering changing how it handles customer requests. An AI answer says the change will reduce costs and improve response times. It cites a general article about automation, notes that another organization used a similar approach, and recommends adopting the change across all service teams. The answer does not show local demand, service quality, implementation cost, or effects on staff workload. The decision has competing consequences. Delaying could leave a possible improvement unexplored. Acting broadly could disrupt service or shift work onto teams that are not prepared. Decide what the answer supports, what remains uncertain, and how a leader could respond without presenting an assumption as a fact.

What good looks like

• You identify the recommendation and the claims that support it.
• You distinguish checkable evidence from reasons, explanations, and assumptions.
• You ask whether the evidence fits the people, place, outcome, and decision at hand.
• You judge sufficiency in light of the consequences, not the answer’s confident tone.
• You record unanswered questions that could change the decision, its scope, or its timing.