Separate problem solving, understanding and consciousness before deciding what a convincing chatbot has proved.
A chatbot explains a joke, notices a contradiction, and solves a puzzle you had to read twice. Then it confidently invents the name of a book. It is tempting to respond with one of two verdicts: “It thinks like us,” or “It is only guessing words.”
Neither verdict settles much on its own. “Thinking” can mean solving a problem, maintaining a goal, understanding a situation, or having a conscious experience. Those are different claims. We make the conversation clearer by separating them.
This is less a courtroom trial of AI than a guide to asking better questions. What does a system actually do? How reliably? Under what conditions? And what can we reasonably infer from watching it?
Prediction can require useful structure
A language model is trained, in part, to predict text. That objective sounds small until you consider the material it must model: dialogue, explanations, code, recipes, arguments, and descriptions of the physical world.
Imagine completing the sentence “Rita put the ice cream in the cupboard, so an hour later…” A good continuation draws on relationships involving temperature, time and food. Statistical learning can capture useful structure even when the training target is the next piece of text.
Language Models are Few-Shot Learners documented varied capabilities obtained from a broad language-training objective. It did not establish that the system has a human inner life. “Predicting words” names a mechanism and training target; by itself, it does not tell us the full range or reliability of the resulting abilities.
Solving a task is an observable claim
Suppose a model correctly sorts several delivery routes by travel time. We can check the answers, change the conditions, and compare its performance with a simple program. That is a tractable evaluation.
Now suppose it writes, “I carefully visualized the roads.” That sentence is another output. It is not a direct window into the computations that produced the route choice. A system can generate plausible explanations of itself without those explanations being faithful records of its internal process.
A calculator does arithmetic without understanding a shopping trip. A person can understand a shopping trip while making an arithmetic mistake. These examples do not prove AI lacks understanding; they show why one performance test cannot answer every question we might care about.
Consciousness is a separate, unsettled issue
Conscious experience concerns whether there is something it feels like to be the system. A fluent first-person answer is insufficient evidence, because producing that answer is exactly the kind of behaviour language training can encourage.
A 2023 interdisciplinary report on AI consciousness proposed examining systems using indicators informed by scientific theories of consciousness. It argued for a more grounded assessment than simply asking a chatbot whether it feels anything. These proposals are not a universally accepted consciousness detector.
We should avoid pretending the philosophical question has been solved in either direction. There is room for uncertainty without treating every convincing conversation as proof of awareness. Saying “we do not yet have decisive evidence” is a more informative answer than a dramatic yes or no.
Understanding is easier to examine in pieces
Consider a model translating a polite refusal between languages. It may preserve tone in common examples but miss a local social convention. Did it understand? The broad question hides several narrower ones: vocabulary, context, social meaning and adaptation to an unfamiliar setting.
A useful test changes one feature at a time. Replace a familiar name with a rare one. Reverse who gave an object to whom. Add a detail that should change the answer. If the response stays polished but ignores the change, you have learned something specific about its limits.
Try this with an invented example rather than a famous puzzle it may have encountered before. Write three short rules for a fictional board game, ask who wins a position, and then change one rule. You still cannot rule out every kind of pattern matching, but you can check whether the system follows the information you supplied.
Why eloquence can mislead us
People naturally read social signals into language. A warm apology sounds like remorse. A confident sentence sounds like knowledge. Those reactions are understandable; most of our experience with conversation involves people.
Training can also encourage agreeable, helpful-sounding responses. The InstructGPT study showed that human feedback could change instruction-following behaviour. A preference for a response is valuable evidence about user experience, but it does not turn style into a truth test.
Here’s the awkward bit, an answer can be both impressively structured and wrong. Instead of asking whether the system “seems smart,” inspect the part you plan to rely on. Does the quotation exist? Do the numbers add up? Does the explanation survive a counterexample?
Use a claim ladder
When discussing AI, move gradually from easier claims to harder ones:
| Claim | Useful evidence |
|---|---|
| It completed this task | Checkable output and clear scoring |
| It handles similar tasks reliably | Varied, independent tests |
| It adapts to new conditions | Controlled changes and unfamiliar cases |
| It has conscious experience | A scientific account beyond conversation alone |
The evidence in one row does not automatically establish the next. This approach lets us acknowledge real achievements without smuggling in conclusions about minds.
It also helps in everyday decisions. A student may benefit from a model that offers alternative explanations, even while checking those explanations against course material. A business should require stronger evaluation before letting the same model decide which customer deserves a refund.
Questions that keep coming up
Is “just autocomplete” a fair description?
It captures the sequential prediction idea but often understates the complexity of learned behaviour. It also does not evaluate correctness or consciousness.
Can an AI lie?
It can produce false or misleading statements. Whether a particular case involves anything comparable to human intention is a further question that the text alone may not resolve.
Should I be polite to a chatbot?
You can be, if that suits how you like to communicate. Politeness is not evidence that the system has feelings or a requirement for evaluating its answers.
Leave room for two ideas
AI can display useful problem-solving behaviour, and its language can lead us to overestimate what has been established. Holding both ideas is not indecision. It is a practical habit: admire the capability, test the claim, and be precise about the much larger questions that remain open.