A few years ago, a phishing email was easy to spot. The grammar was off, the greeting was generic, and the “bank” asking for your password couldn’t spell its own name. Those tells are disappearing. Anyone with access to a decent language model can now produce a clean, convincing message in any language in seconds. That’s one small example of what AI is doing to cybersecurity, and it cuts both ways.
Attackers are using AI to write better lures, clone voices, and speed up the boring parts of an intrusion. Defenders are using it to sort through millions of alerts, catch strange behavior, and find software flaws before someone else does. Neither side has a lasting advantage. What matters is who adapts faster and who has the basics in place.
This guide covers how attackers are using AI, how defenders are responding, where AI-powered defense falls short, and what you can do about it whether you run a security team or just want to avoid getting scammed.
Why AI Changes the Cybersecurity Equation
Cybercrime has always been a numbers game, and it has always had friction. Writing a believable email in a language you don’t speak takes effort. Researching a target takes time. Writing and debugging malicious code takes skill. Every one of those steps limited how many attacks one person or group could run.
AI reduces that friction. Generative models write fluent text, summarize large amounts of information, translate, and help with code. Machine learning models are good at spotting patterns in huge datasets. Both capabilities are useful to attackers and defenders, because both are doing the same underlying job: processing a lot of information quickly and making decisions about it.
The asymmetry is worth understanding. An attacker only has to succeed once, while a defender has to succeed constantly. Automation tends to help the side that needs to do many things at scale, and that’s why security teams have been adopting machine learning for years. But it also means AI can multiply the volume of attempts an attacker can make, even if each attempt is only slightly better than before.
It’s also worth being honest about the hype. Plenty of vendor marketing claims AI will fix security entirely, and plenty of headlines suggest AI-powered super-hackers are already roaming free. The truth is less dramatic. Most real-world AI misuse so far looks like existing attacks done faster, cheaper, and more convincingly, rather than entirely new categories of crime. That’s still a serious problem, just a different one than the movies suggest.
How Hackers Use AI
Phishing remains one of the most common ways attackers get in, and it’s the area where AI has the most obvious impact. Language models make it trivial to write messages that read naturally, match a company’s tone, and avoid the spelling mistakes people used to rely on as warning signs.
The more important shift is personalization. Traditionally, a targeted attack (often called spear phishing) required someone to research the victim by hand. Now an attacker can feed a person’s public LinkedIn profile, company announcements, and social media posts into a model and ask for a message that references their actual job, their actual colleagues, and a plausible reason to click. What used to be reserved for high-value targets can now be done for thousands of people.
Translation matters here too. Attackers who once focused on a language they spoke can now target any market with equally polished text. Regions that were relatively protected by language barriers are no longer protected.
This also affects business email compromise, where an attacker impersonates an executive or vendor to trick someone into sending money or changing payment details. These scams depend on tone and context. A model that can mimic the way a CFO writes, based on a few public samples, makes the impersonation more believable.
Deepfakes and voice cloning
Text is only part of the picture. Voice cloning tools can now reproduce a person’s voice from a short audio sample, and video deepfakes have become good enough to fool people in live calls.
The most widely reported example is a 2024 case in Hong Kong, where an employee at the engineering firm Arup was reportedly tricked into transferring roughly $25 million after joining a video call with what appeared to be the company’s CFO and other colleagues. The people on the call were deepfakes. The case matters less for the dollar figure than for what it showed: seeing and hearing someone on a video call is no longer reliable proof of who they are.
Voice cloning is also used in smaller, more personal scams. A common version involves a call that sounds like a family member in distress, asking for money urgently. The panic is the point. Attackers rely on people acting before they think, and a familiar voice makes that far more likely.
Faster reconnaissance
Before attacking an organization, a hacker needs to understand it: what software it runs, who works there, how it’s structured, and where it might be weak. This reconnaissance phase involves a lot of tedious reading and cross-referencing.
AI helps by summarizing and organizing open-source information at speed. Job postings reveal which technologies a company uses. Public code repositories sometimes leak credentials or internal naming conventions. Employee profiles reveal the org chart. A model can sift through this material and produce a usable picture much faster than a person could.
None of this information is secret, which is the point. The value AI adds is speed and synthesis, turning a day of digging into a few minutes.
Lowering the skill barrier
Another concern is that AI makes cybercrime more accessible to people who wouldn’t otherwise have the skills. Mainstream AI products include safeguards intended to refuse clearly malicious requests, and those safeguards do work to a meaningful degree. But they aren’t perfect, and attackers have looked for ways around them.
There’s also an underground market for tools marketed specifically for criminal use. Names like WormGPT and FraudGPT surfaced in 2023 as chatbots sold to cybercriminals with the promise of no restrictions. How capable these tools really were is debated, and many appear to have been wrappers around existing models or outright scams targeting the criminals themselves. Still, the demand they revealed is real, and the market keeps producing new variants.
The practical effect is mostly at the lower end of the skill range. Someone with limited technical ability can get help writing scripts, debugging code, or explaining how a technique works. This doesn’t turn a novice into an elite hacker, but it does let more people attempt more things, and it saves experienced attackers time on routine work.
Finding and exploiting vulnerabilities
Software flaws are what attackers ultimately exploit, and AI is getting better at helping find them. Models can read code, point out suspicious patterns, and help analyze how a piece of software behaves. Used by an attacker, that speeds up the search for weaknesses. Used by a defender, it speeds up the fix, which we’ll get to shortly.
This is one of the areas where the line between offense and defense is thinnest. The same capability, spotting a bug in code, serves whoever is using it. The practical question becomes who finds the flaw first and who acts on it first. This is a big part of why patching quickly matters more than ever.
Attacking AI systems themselves
As organizations add AI tools to their products and workflows, those tools become targets. This is a newer category of risk, and it’s worth understanding because it’s growing along with AI adoption.
Prompt injection is the best-known example. Language models follow instructions written in plain text, and they can struggle to tell the difference between instructions from their owner and instructions hidden inside content they’re asked to process. If an AI assistant reads emails, summarizes web pages, or browses on your behalf, an attacker can plant hidden instructions in that content. The model may then follow them, potentially leaking data or taking actions the user never intended. The risk grows when the assistant has access to sensitive information or can act on its own.
Data poisoning targets the training process. If an attacker can influence the data a model learns from, they may be able to quietly degrade its performance or introduce behavior that triggers under specific conditions.
Model theft and extraction involve using repeated queries to reconstruct a proprietary model or its training data, which can expose intellectual property or private information.
Adversarial inputs are carefully crafted inputs that cause a model to misclassify something. Researchers have shown this against image recognition for years, and the same idea applies to systems like spam filters or malware detectors that rely on machine learning.
The OWASP organization maintains a list of top risks for applications built on large language models, and MITRE maintains a framework called ATLAS that catalogs attack techniques against AI systems. Both are useful starting points if you’re building or deploying these tools.
Agentic and automated attacks
The newest development is AI agents: systems that don’t just answer questions but take a series of actions toward a goal, using tools along the way. For attackers, the appeal is automation across many steps instead of just help with one.
Anthropic reported in late 2025 that it had disrupted a cyber-espionage campaign in which a state-linked group used an AI coding tool to carry out a large share of the work against a set of targets, with humans mostly supervising and approving key decisions. Details of cases like this are still emerging, and AI-driven operations can make mistakes and invent findings, which limits how autonomous they are in practice. But the direction is clear. As these systems improve, the cost of running a multi-stage operation drops, and defenders should expect more of it.
How Defenders Use AI
The good news is that security teams aren’t standing still. In many ways, defenders have been using machine learning longer than attackers have, and the problems it solves are the ones security teams struggle with most: too much data, too few people, and too little time.
Detecting threats and anomalies
Traditional security tools rely heavily on signatures, which are known patterns of malicious files or behavior. That works well against threats that have been seen before and poorly against anything new.
Machine learning approaches the problem differently. Instead of asking “does this match a known bad thing,” it asks “is this unusual compared to what’s normal here.” A system learns what typical activity looks like for a user, a device, or a network, then flags significant deviations. An employee who normally logs in from one city during business hours suddenly accessing large amounts of data from another country at 3 a.m. stands out, even if no known malware is involved.
This behavioral approach is the backbone of user and entity behavior analytics, and of modern endpoint detection and response tools. It’s particularly useful against attacks that use legitimate credentials or built-in system tools, since those don’t trip signature-based alarms.
Catching phishing and fraud
If attackers use AI to write better phishing, defenders use AI to catch it. Email security platforms analyze far more than the text of a message. They look at sender reputation, the relationship between sender and recipient, unusual writing style compared with a sender’s history, link destinations, and sudden changes in behavior like a vendor suddenly requesting new bank details.
Because a model can compare a message against the normal communication patterns of an organization, it can flag a message that looks perfect on the surface but doesn’t fit. A well-written email from a “supplier” who has never contacted you before, asking you to change payment details, is suspicious regardless of grammar.
Banks and payment companies use similar techniques for fraud detection, scoring transactions in real time based on patterns of past behavior. This is one of the oldest and most mature uses of machine learning in security.
Taming alert overload
Security operations centers face a chronic problem: tools generate far more alerts than analysts can investigate. Many are false alarms, and genuine threats can get buried. Analyst burnout and missed incidents follow.
AI helps by triaging. It can group related alerts into a single incident, rank them by likely severity, add context like which asset is involved and whether it’s critical, and sometimes handle the initial investigation automatically. Language models add a further layer: an analyst can ask questions in plain language, get a summary of what happened in an incident, or have a draft report generated from the raw logs.
This doesn’t replace analysts. It changes where their time goes, away from repetitive sorting and toward the judgment calls that need a person. Done well, it also helps less experienced team members work more effectively, since the tool can explain what it found and why it matters.
Finding and fixing vulnerabilities
AI is increasingly used to find flaws in software before attackers do. In 2024, Google’s Big Sleep project, a collaboration between its Project Zero and DeepMind teams, reported finding a previously unknown, exploitable vulnerability in the widely used SQLite database engine. The flaw was reportedly caught before it appeared in an official release. In 2025, DARPA’s AI Cyber Challenge finished at DEF CON, where teams built systems designed to find and patch vulnerabilities automatically in open-source software.
AI coding assistants are also being used to catch security mistakes while code is being written, flag risky patterns in code review, and suggest fixes. That said, AI-generated code can itself contain security flaws, so it needs the same review as human-written code. Treating it as trusted by default is a mistake.
The long-term promise here is significant. Software has always shipped with bugs, and if AI can help find and fix them faster than attackers can exploit them, the overall balance shifts toward defense. We’re not there yet, but the early results are encouraging.
Speeding up incident response and threat intelligence
When something goes wrong, speed matters. AI tools help responders by summarizing logs, reconstructing timelines, identifying affected systems, and suggesting containment steps. Some platforms can take automated actions, like isolating an infected device from the network, within seconds of detection.
On the threat intelligence side, security teams deal with enormous volumes of reports, indicators, and advisories. Language models can read and summarize these, extract the details relevant to a specific organization, and connect dots across sources. Work that would take an analyst hours can be done much faster, leaving more time to act on it.
Strengthening identity and access
Identity is where many breaches begin, since stolen credentials are a favorite way in. AI-driven identity systems evaluate risk continuously instead of just at login. They consider the device, location, timing, and behavior, and can ask for extra verification or block access when something looks off. This supports the broader move toward zero trust, where nothing is assumed safe just because it’s inside the network.
The Limits of Defensive AI
It would be dishonest to present defensive AI as a solved problem. It has real weaknesses worth knowing about.
False positives and false negatives. Anomaly detection flags things that are unusual, and unusual isn’t always malicious. A new employee, a business trip, or a legitimate software update can all look strange. Too many false alarms and analysts stop trusting the tool. Meanwhile, a careful attacker who behaves like a normal user may slip past entirely.
Models can be fooled. As covered earlier, machine learning systems can be manipulated with adversarial inputs, and attackers who understand how a detector works can try to evade it. Defense models are targets too.
Hallucinations and overconfidence. Language models sometimes produce confident answers that are simply wrong. In a security setting, a fabricated finding or an incorrect remediation step can waste time or cause damage. Outputs need to be verified, especially for anything consequential.
Data and privacy concerns. Feeding sensitive logs, code, or customer data into AI tools raises questions about where that data goes and who can access it. Organizations need clear policies on what can be shared with which tools.
Over-reliance. If a team leans too heavily on automation, its own skills can fade, and a blind spot in the tool becomes a blind spot in the organization. AI works best as support for skilled people and not as a replacement for them.
The basics still matter most. Many breaches still come down to unpatched software, weak or reused passwords, missing multi-factor authentication, and poor access controls. No AI product makes up for neglecting these.
Practical Steps for Individuals and Organizations
The defensive advice that follows from all this is less exotic than you might expect. Much of it comes down to strengthening habits and processes so that a convincing fake alone isn’t enough to cause harm.
For individuals:
- Slow down on urgent requests. Pressure and urgency are the common thread in almost every scam. Whether it’s a text, an email, or a phone call, a sense that you must act right now is a reason to pause.
- Verify through a second channel. If someone you know calls asking for money or sensitive information, hang up and call them back on a number you already have. Some families agree on a private code word for emergencies, which a voice clone can’t know.
- Use multi-factor authentication everywhere you can, preferably with an authenticator app or a hardware key rather than text messages. Passkeys are better still where they’re available.
- Use a password manager so every account has a unique password. Password managers also help because they won’t auto-fill credentials on a fake site.
- Keep software updated. Many attacks exploit known flaws that already have patches.
- Be careful what you share publicly. The more personal detail available online, the easier it is to build a convincing targeted message.
- Think before pasting sensitive information into AI tools, and be careful about giving AI assistants broad access to your email, files, or accounts.
For businesses and security teams:
- Build verification into payment and access processes. Any request to move money, change bank details, or reset credentials should require confirmation through a separate, known channel, no matter how senior the requester appears to be. A deepfake can imitate a face, but it can’t get around a process that doesn’t depend on recognizing one.
- Train employees with current examples. Awareness training that still focuses on spotting typos is out of date. Teach people to question context and urgency, and run realistic simulations.
- Get the fundamentals right. Enforce multi-factor authentication, apply least-privilege access, patch promptly, keep tested backups, and monitor your systems.
- Adopt AI-enabled defenses where they fit, such as email security, endpoint detection, and alert triage, but pilot them, measure results, and keep humans in the loop.
- Set a policy for employee AI use. Define which tools are approved, what data can be entered, and who reviews AI-generated output, especially code.
- Secure the AI you deploy. If you build products on language models, treat them as an attack surface. Limit what the model can access and do, treat anything it reads from outside sources as untrusted, require human approval for high-risk actions, and test for prompt injection. The OWASP LLM risk list and NIST’s AI Risk Management Framework are good references.
- Plan for incidents. Have a response plan that includes the possibility of deepfake-driven fraud, and rehearse it.
What Comes Next
Predicting the future of fast-moving technology is risky, but a few trends look likely to continue.
Agentic systems will improve, on both sides. Attackers will automate more of the chain from reconnaissance to exploitation, and defenders will deploy AI that investigates and responds with less human involvement. The security of those agents themselves, what they can access and what they’re allowed to do, will become a central concern.
Proving who or what is real will get harder. Expect more emphasis on content authenticity standards, cryptographic verification, and processes that don’t rely on recognizing a face or voice. Organizations that already verify through independent channels will be better positioned.
Regulation and standards are also taking shape, though unevenly across regions. Frameworks for AI risk management are maturing, and security teams will increasingly be asked to demonstrate how AI systems are governed and tested.
Finally, the skills gap in security may shift rather than close. AI can make junior analysts more effective, but the people who can evaluate AI output, spot its mistakes, and design sound processes around it will be in demand.
Frequently Asked Questions
Can AI really hack on its own?
Not reliably, at least not yet. AI can automate parts of an attack and speed up many steps, and recent reports suggest agent-style tools can carry out a significant portion of some operations. But these systems still make mistakes, need direction, and struggle with unfamiliar situations. The realistic near-term risk is faster, cheaper, more scalable attacks rather than fully independent machine hackers.
How can I tell if a message was written by AI?
Often you can’t, and that’s the point. Instead of judging the writing, judge the request. Is it unexpected? Is it urgent? Does it ask for money, credentials, or a change in routine? Verify anything like that through a separate channel regardless of how polished it looks.
Is AI more helpful to attackers or defenders?
It helps both, and the balance shifts depending on the area. Attackers benefit from AI’s speed in social engineering and scaling. Defenders benefit from AI’s ability to process huge volumes of data and automate routine work. Organizations with good fundamentals and well-integrated tools tend to gain more than those without.
Should my company ban AI tools?
Blanket bans tend to push usage underground, where it’s harder to manage. A better approach is a clear policy: approved tools, rules on sensitive data, and review of AI-generated output.
What’s the single most useful thing I can do?
Turn on multi-factor authentication, ideally with phishing-resistant methods like passkeys or hardware keys, and make a habit of verifying unusual requests through a second channel. Those two steps blunt a large share of AI-enhanced attacks.
What is prompt injection?
It’s an attack where hidden instructions inside content, like a web page or email, trick an AI system into following them instead of its intended instructions. It matters most when the AI has access to private data or can take actions on your behalf.
Conclusion
AI hasn’t rewritten the rules of cybersecurity, but it has changed the speed and scale at which those rules play out. Attackers get polished phishing, cloned voices, and faster research. Defenders get better detection, quicker triage, and new ways to find flaws before they’re exploited.
The most dependable defense isn’t any single tool. It’s a combination of solid security fundamentals, processes that don’t hinge on trusting what you see and hear, and thoughtful use of AI where it genuinely helps. Organizations and individuals who treat AI as one part of a broader security approach, rather than a cure or a catastrophe, will be in the strongest position as both sides keep learning.
