The Machine Said So

A model's answer looks equally confident either way: when it is right and when it is invented. The tone does not change, and no caveats appear.

People trust answers like that. A prompt from software is accepted faster than one's own doubt. This was noticed long before chat services: an operator agrees with the instrument even while seeing the opposite.

What it looks like at work

You ask for an analysis of a contract, get the analysis, and never look at the contract again. The summary has been read, the feeling of knowing is there, and the text itself you have not seen.

You take a number from the answer into a report because it sits next to the correct ones. Proximity to what is right lends weight to what is not.

You send an email without rereading it, because the last 10 emails were fine. Verification turns into a ritual and gradually dies off.

A summary does not replace reading

A summary answers the question "what is this document about." You make the decision on something else: which conditions exactly, in which clause, with which exceptions.

The working order that follows is simple: read the summary first, and open the passages the decision depends on in the document yourself.

What helps

Ask where the answer might be wrong. The direct question "what here is worth checking" brings back caveats that were not in the first answer.

Verify selectively but seriously. Not every claim, but two or three key ones, and against the original source rather than against another answer from the model.

Be suspicious of what is convenient. An answer that matched your expectations gets checked more carefully than the rest: it does not make you want to argue.