Over the past few years there’s been a lot of focus on prompt engineering. How to write better prompts, how to structure your questions, how to get more useful responses. And while that’s worth understanding, it misses something more fundamental. The bigger skill isn’t learning how to ask better questions. It’s understanding what kind of answer you’re actually getting back.
Most people use AI tools as though they’re retrieving information from a reliable source, the way you might query a database or pull a report from your CRM. That’s not what’s happening. What’s actually happening is that the model is predicting the most statistically probable next sequence of text, given the prompt you’ve provided and the patterns encoded in its training data. There’s no lookup happening, no verification step, no confidence score attached to the output. The model produces what is most likely to be a coherent and contextually appropriate response, and it does that whether the underlying information is solid or not.
That distinction matters a lot in practice, and it’s one we don’t think enough organisations have really internalised yet.
Why the output can sound right without being right
The technical term for when a model generates something that sounds plausible but is factually incorrect is hallucination. It’s a useful word because it captures something important. The model isn’t lying, and it isn’t malfunctioning. It’s doing exactly what it’s designed to do, which is generate a fluent, coherent response. The problem is that fluency and accuracy are two separate things, and the model optimises for the former without guaranteeing the latter.
There are a few conditions that make this more likely, and they’re worth understanding clearly.
Training cutoffs are the most straightforward. Every frontier model has a date after which its knowledge becomes unreliable or absent. A model with a cutoff of mid-2024 has no reliable knowledge of anything that changed after that point. It may have picked up some information through additional fine-tuning, but that coverage is inconsistent. This is something that comes up constantly in Salesforce work. A number of products have been renamed and repositioned in the past year or so, and most frontier models are still working from the old naming conventions because the changes happened after their training data was locked. Ask one of these models about current Salesforce product architecture and you may get a very confident answer that is simply out of date.
Context gaps are subtler but equally consequential. The model can only reason from what it has been trained on and what you’ve provided in the current session. If you haven’t told it who your customers are, what your business constraints look like or what the specific situation requires, it will fill those gaps with statistically likely approximations. Those approximations can be coherent and well-structured while being completely misaligned with your actual circumstances.
Weak source material is harder to detect. The model’s training data includes a vast range of content, not all of it reliable or well-reasoned. When a response draws on lower-quality patterns in that data, the output can still read as authoritative. There’s nothing in the surface presentation of the text that signals the difference.
What this means in practice
The appropriate level of scrutiny depends on what the output is being used for.
For exploratory or generative work, for example brainstorming, drafting, reshaping language, the review bar is relatively low. You can assess the output on its merits, apply your own judgement and move on. The model is functioning as a capable thinking partner and the consequences of an imperfect response are manageable.
For work that informs a business recommendation, shapes a customer-facing decision or affects an outcome involving people, the bar needs to be higher. Not because the tool is categorically unreliable in those contexts, but because the model’s confidence is not a proxy for accuracy, and in high-stakes situations that distinction matters quite a bit.
A useful habit is to ask explicitly where the model’s information is likely to be coming from. Is this general knowledge that would have been well-represented in pre-cutoff training data? Is this something that depends on specific business context, and if so, has that context actually been provided clearly enough? Is this a topic where recent developments could affect the answer, and is the model’s training recent enough to cover them? Those questions take seconds to run through and they significantly change how you read what comes back.
Context is the real lever
Better prompts help at the margin. The more meaningful shift comes from treating context as a primary input rather than an afterthought. Your business goals, your constraints, your audience, the specific data or material the model should be reasoning from rather than approximating from general patterns.
When you provide that properly, the model isn’t operating on generic assumptions. It’s reasoning from your situation, and the output reflects that. In our experience, the quality of what comes back is much less about the cleverness of the prompt and much more about the quality and specificity of what you’ve given it to work with.
It’s also worth being deliberate about which model you’re using and what information you’re sharing with it. The licence and account type you’re operating under determines what the vendor can do with that data. The more context you provide, the more useful the model becomes. That context should only go into a platform you’ve verified is appropriate for the sensitivity of the information involved, something worth checking before you start, not after.
The skill that actually matters
Writing a good prompt is a useful habit. Knowing how to read the output critically is a more important one.
AI becomes more useful when you understand how the information is being created, because that tells you how to review it, where to push back and when your own judgement still needs to lead. The goal isn’t to trust AI less. It’s to use it in a way that’s actually worth trusting.
If you’d like to understand what AI use cases could apply in your business, let’s talk.
