Abstract
Artificial intelligence systems, and large language and generative models in particular, now shape how many people find information, communicate, and make decisions. This review separates two related problems that are often blurred together. The first is reliability: the tendency of these systems to produce confident, fluent, and wrong output even when no one intends harm. The second is deception: the deliberate use of AI to mislead, impersonate, defraud, or manipulate. We survey established concepts behind each problem, including hallucination and confabulation, probability calibration and overconfidence, distribution shift, the limits of benchmark evaluation, and automation bias. On the deception side we examine misinformation produced at scale, synthetic voice and video, impersonation fraud, computational persuasion, and the "liar's dividend," in which the mere existence of fakery lets bad actors dismiss authentic evidence. We aim to be accurate rather than alarming: these tools have genuine value, capability is not the same as widespread harm, and many countermeasures are practical today. We close with concrete guidance for parents, educators, clinicians, and policymakers.
Keywords: artificial intelligence, reliability, calibration, hallucination, automation bias, deepfakes, misinformation, fraud, provenance, media literacy
Introduction
Scope
This paper is a plain-language review for readers who are not machine learning specialists but who are responsible for others: parents, teachers, health and mental health clinicians, and people who write policy or run institutions. We focus on the current generation of generative systems, especially text models and synthetic audio and video, because these are the tools most people now encounter. We do not attempt a technical tutorial on how models are trained. Instead we describe how they fail, how they can be misused, and what a reasonable person can do about it.
Reliability versus deception
Figure 1. Unreliability is an honest error. Deception is purposeful. Both grow from the same root: fluent output disconnected from truth.
The two failure modes in this review differ mainly in intent. A reliability failure is an honest error: the system generates something inaccurate, and no human meant for that to happen. A deception is purposeful: a person uses AI as an instrument to make someone believe a falsehood or to pose as someone they are not. The line matters because the responses differ. Unreliability is addressed through better engineering, honest disclosure of limits, and careful human oversight. Deception is addressed through security practices, verification of identity and provenance, law, and public awareness. A single incident can involve both. A fabricated legal citation produced by a model is an honest error until a person knowingly submits it to deceive a court.
Why it matters
Fluency is persuasive. Human beings tend to treat confident, well-formed language as a signal of competence and truth, a habit that served us reasonably well when fluent speech required understanding. Generative systems break that link. They can produce polished prose, a convincing voice, or a realistic face without any grounding in fact and without any intent behind the words. This decoupling of surface quality from truth is the common thread running through both halves of this review. When the cues we rely on to judge credibility can be manufactured cheaply, individuals and institutions need new habits, and they need them before a crisis rather than during one.
Reliability: How AI Systems Fail On Their Own
Figure 2. Four ways a system fails on its own, and a fifth failure that is really about us.
Hallucination and confabulation
The most discussed reliability problem is the tendency of language models to state things that are not true. Many researchers prefer the word "confabulation," borrowed from clinical psychology, because it captures the mechanism better than "hallucination." A model trained to predict likely text produces output that is plausible given its training, not output that has been checked against the world. When the training data thinly covers a topic, or when a question invites a specific detail such as a citation, a date, or a quotation, the system will often generate something that fits the pattern of a correct answer while being invented. Documented examples include fabricated legal cases that have appeared in real court filings and invented references in academic drafts.
Two points keep this in proportion. First, error rates vary widely by task; models are far more reliable summarizing a provided document than recalling obscure facts from memory. Second, techniques such as retrieval, in which the system is given source material to work from, and requiring citations to checkable sources, reduce but do not eliminate the problem. The safe assumption is that any specific, verifiable claim from a generative model may be wrong and should be treated as a lead, not a fact.
Calibration and overconfidence
Calibration is a precise idea worth understanding. A predictor is well calibrated if, among all the times it says it is ninety percent sure, it is right about ninety percent of the time. Calibration is measured with proper scoring rules, such as the Brier score and logarithmic loss, which reward a system for assigning honest probabilities rather than for sounding certain. The central reliability concern with conversational AI is that its expressed confidence and its actual accuracy are often poorly matched. The prose is uniformly assured whether the underlying answer is solid or invented. Human experts are not perfectly calibrated either, but experienced people usually signal doubt through hedging and tone. Current systems tend to strip those signals out, which makes their mistakes harder to catch. When a tool cannot reliably tell you how much to trust a given answer, the burden of judgment falls back on the user.
Distribution shift and brittleness
Machine learning systems learn patterns from a particular body of data collected at a particular time. They perform best on inputs that resemble that data and can degrade sharply on inputs that do not, a problem called distribution shift. A medical model trained on images from one set of hospitals may falter at another with different equipment. A fraud detector tuned on past behavior can miss new tactics. Related to this is brittleness: small, sometimes deliberate changes to an input can produce large changes in output, and systems can latch onto incidental correlations that do not hold in the wider world. The practical lesson is that strong performance in testing does not guarantee strong performance later, especially once conditions change or once people begin adapting to the system.
Evaluation limits and benchmarks
Public discussion often centers on benchmark scores. Benchmarks are useful but limited. A high score shows competence on one curated test set under specific conditions, not general reliability. Several problems recur. Test material can leak into training data, so a model may have effectively seen the answers, a concern known as contamination. Aggregate scores hide the specific cases that matter most, which are frequently rare and high stakes. Averages also obscure uneven performance across languages, dialects, and groups of people. And benchmarks measure narrow tasks that may not resemble the messy situations of real use. Treat headline numbers as one weak signal among many, and give more weight to careful evaluation on the actual task in front of you.
Automation bias and over-trust
Automation bias is the well-studied human tendency to over-rely on automated systems: to accept their suggestions without enough scrutiny and to overlook problems the machine misses. It has two faces. Errors of commission occur when people follow a wrong recommendation against better evidence. Errors of omission occur when people fail to act because the system did not prompt them. The effect is stronger when people are busy, tired, or lack time to check, exactly the conditions under which AI tools are marketed as helpful. Research in aviation, healthcare, and driving assistance has shown that a capable automated aid can quietly erode the vigilance of the person supervising it. This makes over-trust a central reliability issue rather than a side note. A system that is right most of the time can be more dangerous than one that is often wrong, because it trains its users to stop checking.
Deception: How AI Is Used to Mislead
Misinformation at scale
Generative models lower the cost of producing plausible text to nearly zero. This does not create the human motives behind false information, which are old, but it changes the economics. One person can now generate large volumes of tailored, fluent content: fake reviews, comment-section filler, entire networks of low-quality sites, and messages adapted to a specific audience. The concern is less that any single piece is uniquely convincing and more that sheer volume can crowd out reliable sources, exhaust the attention of moderators, and make coordinated activity look like organic consensus. It is worth being careful here. The evidence that synthetic misinformation changes large numbers of minds is mixed, and researchers continue to debate real-world effects. The clearer risks are pollution of the information supply and the erosion of shared trust, rather than mass persuasion on demand.
Synthetic media: voice and video
Deepfakes are synthetic or manipulated audio and video that depict real people saying or doing things they did not. Voice cloning is now the most immediately dangerous form, because a short sample of someone speaking can be enough to produce a convincing imitation, and because a phone call carries fewer visual cues to inspect. Video synthesis is advancing quickly, though close inspection can still reveal artifacts in some cases. Two honest qualifications belong here. First, the same technology has legitimate uses in film, accessibility, education, and voice restoration for people who have lost the ability to speak. Second, quality varies, and detection is possible in many cases, though detection is an ongoing contest rather than a solved problem. The prudent stance is that audio and video are no longer self-authenticating. Seeing or hearing is no longer sufficient proof.
Impersonation and fraud
Figure 3. Voice-clone fraud follows a chain. One verification step, a callback on a known number, breaks it.
The most concrete harm to ordinary people is fraud, where synthetic media joins long-running scam techniques. Several patterns are well documented in general terms. In romance scams, criminals build a false relationship over weeks or months, and generated photos, chat, and voice make the fictional partner more convincing and easier to run at scale. In what is often called CEO fraud or business email compromise, an attacker poses as an executive or a trusted vendor to authorize an urgent payment; the addition of a cloned voice on a phone call removes a check that once helped catch these schemes. A widely reported category involves the "grandparent" or family-emergency scam, in which a caller imitates a relative in distress to demand money quickly. The common ingredients are urgency, secrecy, an unusual payment method, and pressure that discourages verification. AI does not invent these tactics. It makes them cheaper, more personalized, and harder to detect by ear.
Persuasion and manipulation
Beyond outright fraud lies a subtler concern: systems that influence beliefs and choices through sustained, adaptive interaction. A conversational agent can adjust its approach in real time, remember what a person responds to, and maintain endless patient engagement in a way no human salesperson or propagandist can. This raises worries about manipulation of vulnerable people, including children, isolated adults, and those in mental health crisis, and about the commercial incentive to maximize engagement rather than wellbeing. The strength of these effects is still being studied, and claims of irresistible AI persuasion are not supported by current evidence. But the direction of risk is clear enough to warrant caution, especially for products aimed at or accessible to people who may form attachments to a responsive system.
The liar's dividend
The legal scholars Robert Chesney and Danielle Citron named an effect that may prove as important as any fake itself: the "liar's dividend." Once the public knows that convincing fakes exist, anyone caught on genuine audio or video can claim the evidence was fabricated. The mere possibility of deepfakes gives the dishonest a ready excuse and shifts the burden onto those who would hold them accountable. This is a corrosive second-order effect. The primary danger of synthetic media may not be that people believe false things, but that they lose confidence in true things, and that reliable evidence loses its power to settle disputes. Protecting the credibility of authentic records is therefore as important as detecting fakes.
Practical Guidance
The tone here is deliberately calm. Most people can protect themselves and those they care for with a small number of durable habits, without becoming experts and without treating every message as an attack.
Verification habits for individuals
Treat specific claims from any AI system as unverified until checked against a primary source, especially names, numbers, quotes, citations, medical and legal facts, and anything you would act on. Ask a tool to show its sources, then read those sources; a confident answer with no checkable origin deserves less trust, not more. For anything consequential, consult a second independent source rather than a second AI. Notice your own state: fatigue and time pressure are when automation bias and scams both work best.
Provenance and disclosure
Provenance is the effort to attach verifiable origin information to media. The C2PA standard, known in consumer form as Content Credentials, lets images and video carry a signed record of where they came from and how they were edited. Provenance is not a complete solution, since content can be stripped of its credentials or captured outside the system, but a record of authenticity is easier to build and defend than a perfect detector of fakes. Prefer platforms and organizations that support these standards, and learn to look for the credential indicator where it appears. Detection tools exist and are improving, but they should be treated as fallible aids, not verdicts.
Protecting older relatives and other targets of fraud
Fraud protection is best handled as a family and community matter rather than left to the most-targeted individual. A few measures are effective. Agree on a shared verification step for any urgent request for money or sensitive information: hang up and call the person back on a known number, or ask a question only the real person could answer. Consider a family code word for emergencies, established in advance. Name the specific tactics out loud, since people who know that voice cloning and family-emergency scams exist are far less likely to be caught by them. Reduce the public voice and video samples available of people who may be imitated, where that is practical. Above all, make it easy and shame-free to pause and check; scams depend on urgency and embarrassment, and removing those removes much of their power.
Institutional safeguards
Organizations should treat AI output as a draft requiring human judgment for any decision that affects people, and should say so in policy. Payment and access controls should not rely on a voice or a video call alone; require a second channel and a callback for high-value or unusual requests, and train staff on business email compromise. Where AI assists in medicine, law, hiring, or education, keep a qualified person accountable for the outcome, evaluate the system on your own real cases rather than on vendor benchmarks, and monitor for drift as conditions change. Schools and clinics can teach the verification habits above directly, framed as ordinary information literacy rather than fear. Policymakers can support provenance standards, fund independent evaluation, require clear disclosure of synthetic media in sensitive contexts, and protect the value of authentic evidence, which the liar's dividend puts at risk.
Conclusion
The reliability and deception problems described here share one root: modern AI can produce fluent, realistic output that is disconnected from truth and from any honest intent. That decoupling breaks the ordinary cues people use to judge what to believe. The right response is neither dismissal nor alarm. These tools are genuinely useful, capability is not the same as widespread harm, and the most effective protections are practical and available now. They come down to a habit of verification, support for provenance and honest disclosure, sensible security around identity and payments, and institutions that keep human beings accountable for consequential decisions. The skill worth building, individually and collectively, is calibrated trust: neither believing everything a confident system says nor rejecting everything as possibly fake, but knowing when and how to check. Cultivating that skill, and passing it on, is the surest defense available.
References and Further Reading
- Robert Chesney and Danielle Keats Citron, "Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security," California Law Review (2019). Origin of the "liar's dividend" concept.
- Coalition for Content Provenance and Authenticity (C2PA) technical specification, and the Content Credentials initiative. Provenance standards for media.
- Raja Parasuraman and Dietrich Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors (2010). Review of automation bias.
- Kate Goddard, Abdul Roudsari, and Jeremy Wyatt, "Automation Bias: A Systematic Review," Journal of the American Medical Informatics Association (2012).
- Glenn W. Brier, "Verification of Forecasts Expressed in Terms of Probability," Monthly Weather Review (1950). Foundational work on proper scoring and calibration.
- Tilmann Gneiting and Adrian Raftery, "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association (2007).
- Chuan Guo and colleagues, "On Calibration of Modern Neural Networks," Proceedings of the International Conference on Machine Learning (2017).
- Emily Bender, Timnit Gebru, and colleagues, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (2021).
- United States Federal Trade Commission, consumer guidance on imposter scams, business email compromise, and voice cloning. Practical, regularly updated advice.
- United States Federal Bureau of Investigation, Internet Crime Complaint Center (IC3), annual reports on romance scams and business email compromise.
- Britt Paris and Joan Donovan, "Deepfakes and Cheap Fakes," Data and Society Research Institute (2019). On synthetic media and its social context.
- Renee DiResta and colleagues, Stanford Internet Observatory research on coordinated influence operations and information integrity.