How can AI developers tell whether an AI is actually thinking critically?

Many professors and other people who regularly work with AI have pointed out that AI can produce a fast and convincing answer without necessarily demonstrating much critical thinking. A model can explain an argument and list counterarguments. It spits out a confident conclusion within seconds. But producing an answer quickly isn't the same thing as carefully evaluating information. That raises a bigger question: how can developers tell whether an AI is actually improving its reasoning or simply learning what a "critical thinking" answer is supposed to sound like? For example, a model could be trained to question an argument and provide an opposing viewpoint. That doesn't necessarily mean it understands whether the evidence supports either position. It could just be following a learned pattern. A useful test might require an AI to deal with incomplete or conflicting information, identify which evidence is reliable, explain why it reached a conclusion, and then change that conclusion when new evidence contradicts it. The difficult part is creating a benchmark for this. What would a genuinely good test of AI critical thinking look like? How could developers separate actual reasoning from an AI that has simply become better at producing convincing explanations?