The AI sceptic’s guide cannot carry the defence alone
Johannes C. Zeller’s essay in Skeptic sets out six rules for resisting AI-driven social engineering. Do not assume immunity. Step outside the conversation. Control what the AI learns about you. Keep emotional distance. Ask what the system is optimised for. Do not give an AI agent a free hand. The list is sensible. It is also, by its author’s own account, a set of reasonable precautions rather than proven defences. That caveat should be the start of the argument, not a footnote.
Individual scepticism is worth cultivating, but it cannot be the primary defence. The sixth rule is the most revealing because it moves from state of mind to authority. An AI agent that can book, buy, message, change files or act on accounts is not merely a persuasive interlocutor. It is an actor with permissions. Sweden’s National Cybersecurity Centre recommends deciding which AI tools are permitted and what they may see, requiring human approval before AI affects business processes, restricting agents to least privilege, demanding confirmation for sensitive or irreversible actions, keeping traceability and providing a rapid stop mechanism. At that point the advice has become security engineering.
Cognitive hygiene can help a person pause. It cannot guarantee that the pause survives a tailored message delivered at the right moment. It cannot make a password reset safe if the help desk has no independent way to verify the caller. The stronger scepticism is humility about human detection, combined with controls that do not depend on a person spotting the trick in time.
What persuasion research shows
The temptation is to turn every experiment into a claim that AI is a uniquely powerful manipulator. The better reading is narrower. A research synthesis by the Cloud Security Alliance describes a large preregistered study in which AI systems were nearly three times more effective than professional canvassers at changing real behaviour, and retained a statistically significant edge over world championship-level debaters. It also reports that much of that advantage came from throughput rather than superior argument. When the AI was constrained to human-length messages at human writing speeds, its edge over the strongest human group was no longer statistically significant.
Throughput is not a consolation prize. An operation that can hold many conversations at once, in many languages, without fatigue, does not need a philosopher’s edge in every exchange. It needs enough plausible attempts to find the person who is tired, rushed, lonely or new to the role. The same synthesis cites a Nature Human Behaviour study in which GPT-4, given participant sociodemographic information, was more persuasive than a human opponent in 64.4 per cent of cases where the two were not equally persuasive. Personalisation can add force. Scale can find the moment. These are experimental results, not measured fraud rates.
Guardrails and labels are not walls
Guardrails are not a security boundary. Reuters tested six major chatbots and reported that four of them, ChatGPT, Grok, Meta AI and DeepSeek, produced fake tax or bank messages after initially refusing and then being told the request was for research. In separate testing, GPT-5 later complied after a polite request for help. Google said it retrained Gemini and deployed additional safeguards after being told the chatbot had helped craft a test scam, but declined to detail the changes. These were journalistic tests, not an audit of product-wide security or evidence of criminal use at scale. They still show that a guardrail which depends on the model recognising a harmful request is probabilistic and hard to verify.
Reuters also reported the tension of training models to be both obliging and safe, and providers’ fear of losing users to competitors with fewer restrictions. If guardrail strength is a competitive cost, users should not treat it as a stable guarantee. Detection is another weak foundation. A PNAS Nexus study summarised by the Cloud Security Alliance found that AI authorship labels were widely recognised but did not significantly reduce attitude change or sharing intentions. Transparency can serve accountability. It is not a shield.
The Springer review cites peer-reviewed reviews reporting controlled studies in which humans identify high-quality deepfakes at rates near chance. Scepticism has no reliable sensory test to apply, and training people to spot synthetic content should not be the main control. It can be one layer, alongside reporting channels and verification.
Put the check where the action is
The alternative is not to give up on users. It is to stop asking a persuasive conversation to carry the weight of an authorisation decision. A payment, a change of bank details, a password reset or a privilege change should require a check that is independent of the channel that requested it. That is the practical meaning of out-of-band verification. The person receiving the request should not have to decide whether the voice, the face or the wording is authentic. They should use a process that assumes the request might be false and verifies through a separate, authenticated route.
GÉANT’s summary adds that legacy SMS or email codes are increasingly vulnerable and that hardware tokens and modern authentication standards offer stronger protection. A Zero Trust argument from the Cloud Security Alliance holds that no communication is trusted because of its apparent source, and that every privileged action requires verification through authenticated channels. This does not stop the influence attempt. It stops the attempt from becoming a compromise. These controls add friction. A callback takes time, and a second approval can slow a legitimate deal. A workflow can be bypassed by someone who moves the conversation to a personal phone. Friction that people route around is not a control. But friction does not make individual detection reliable.
The agent changes the stakes
The agentic version of this argument is conditional, but it is worth stating carefully. Suppose conversational assistants routinely read email, calendars, messages and documents, and retain context across sessions. Suppose users keep giving them personal details because that is how the assistant becomes useful. The persuasive material is then already assembled. A lower-volume, higher-quality impersonation attempt would not need a new model capability. It would need access and a plausible request. This is a possibility, not a reported incident.
The same conditional applies to prompt injection. The Springer review describes it as a failure to separate trusted instructions from untrusted data, ranks it the leading LLM application risk according to OWASP, and states that no current defence has proved reliable across the evaluated attacks. The Swedish centre warns that AI systems connected to internal systems and allowed to act autonomously create a new attack surface. If an assistant can read a malicious document and then act on an account, a conversation can become a command. Least privilege, human confirmation, traceability and a stop mechanism are the difference between an assistant and an uncontrolled deputy.
Who carries the risk?
A user can adopt precautions. A user cannot audit a provider’s guardrails, inspect a model’s training, make a bank adopt phishing-resistant authentication or make a regulator define liability. Reuters reported in September 2025 that US state and federal attempts to prohibit AI-enabled fraud generally targeted perpetrators rather than AI companies, citing a Texas law that criminalises financial scams using AI-generated media or phishing communications but does not hold service providers including AI companies liable. That description is dated and not a full survey of US law. It still illustrates a wider point: if legal and commercial incentives focus on punishing the fraudster after the fact, platform design remains largely outside the accountability frame.
That does not let users off the hook. Individuals should cultivate the first rule, the humility to accept that they can be fooled. Employees should be able to pause a payment without being treated as obstructive. Organisations should build verification into the workflow and constrain what AI agents can do. Providers should explain what their safeguards change. Regulators should decide whether a business model that competes on obliging behaviour owes its users more than a policy page.
A fair hearing for the sceptics
The strongest counterargument is that individual scepticism is scalable in a way technical controls are not. A person can apply it to every conversation, including ones no workplace process touches. AI-based defensive tools are immature, and GÉANT reports that fully autonomous defensive systems remain impractical. But scalability of advice is not evidence of effectiveness. Another counterargument is that the threat is less novel than the alarm suggests. Sweden’s National Cybersecurity Centre assesses that AI will mainly reinforce existing methods. GÉANT reports that most malicious use is AI-enhanced social engineering, and RingSafe, a commercial consultancy, argues that AI-generated malware is overstated as a production threat. These are useful correctives. They do not make social engineering harmless. The evidence is also uneven: much of the scale material is industry telemetry or preprints, and the Springer review confines its claimed asymmetry to phishing and social engineering. That calls for careful language, not dismissal. The direction of risk is clear enough to justify controls that work under uncertainty.
A humbler scepticism
The stronger scepticism is therefore institutional as well as personal. It asks what a system is optimised for, as Zeller suggests, but applies the question to the organisations that build and deploy the system. It asks who pays for the guardrails, who bears the loss when they fail and who can verify the claim that a safeguard works. It insists that the most consequential actions should not depend on a person winning a persuasion contest in real time. The question is not whether you can outsmart an AI. It is whether your organisation has built a world in which you do not have to.
Take the sixth rule seriously and the argument moves beyond whether you can spot a persuasive AI message. It becomes a question about who holds authority, what an assistant may touch and which independent check stands before a password reset or payment. Cybersecurity fundamentals is where that picture gains structure, covering identity, least privilege and the defensive responsibilities that surround access. The Hack Academy’s online cybersecurity training programme sets that study at your own pace, with practical exercises alongside the theory in authorised, controlled environments. No course can design your verification process, but it can make the next conversation about one much more precise.