Phishing remains the most pervasive cyber‑attack vector, siphoning credentials, installing malware, and draining corporate budgets. Traditional rule‑based filters are increasingly inadequate because attackers constantly evolve their tactics—crafting emails that look almost identical to legitimate communications. That’s why the industry is turning to AI‑powered tools that can learn, adapt, and spot subtle anomalies in real time. In this guide, we’ll walk you through the full workflow of detecting phishing emails with AI, from setting up the environment to fine‑tuning models and integrating them into your organization’s email flow. Whether you’re a security engineer, a SOC analyst, or a DevSecOps lead, you’ll gain the skills to stay ahead of the threat landscape.
What You’ll Need
- A corporate email gateway (e.g., Microsoft Exchange, Google Workspace, or a third‑party MX record).
- Access to an AI‑powered email analysis service (OpenAI GPT‑4, Anthropic Claude, or proprietary solutions like Darktrace Email AI).
- Python 3.10+ and pip for scripting.
- API keys for your chosen AI provider.
- An email parsing library (e.g.,
email,mailparser, ormailbox). - Optional: a sandbox environment (e.g., PlayPlayground or a local VM) to safely open suspicious attachments.
- Version control (Git) and a CI/CD pipeline for model updates.
Step 1: Capture Raw Email Data
Before AI can do its magic, you need a reliable stream of raw email data. Most modern mail gateways expose an API or a webhook that pushes the entire MIME payload to your ingestion system. Below is a typical setup for Microsoft Exchange using the Graph API:
# Register an Azure AD app with Mail.Read and Mail.ReadWrite scopes
# Then poll the /me/messages endpoint
curl -H "Authorization: Bearer {access_token}"
https://graph.microsoft.com/v1.0/me/messages?$top=100
-o raw_emails.json
For Google Workspace, you can use the Gmail API’s users.messages.list and users.messages.get endpoints:
# OAuth 2.0 flow to get a refresh token
# Fetch raw email
GET https://gmail.googleapis.com/gmail/v1/users/me/messages/{messageId}?format=raw
Store the raw MIME payload in a secure, immutable archive (e.g., S3 with versioning). This ensures you have a forensic trail for every email processed.
Step 2: Parse the Email and Extract Features
AI models thrive on clean, structured input. Use Python’s email library to dissect the MIME parts and pull out key features:
import email
from email import policy
with open('raw_email.eml', 'rb') as fp:
msg = email.message_from_binary_file(fp, policy=policy.default)
subject = msg['subject']
from_addr = msg['from']
to_addr = msg['to']
headers = dict(msg.items())
# Extract body (plain and HTML)
plain_body = None
html_body = None
for part in msg.walk():
if part.get_content_type() == 'text/plain' and not plain_body:
plain_body = part.get_payload(decode=True).decode(part.get_content_charset(), errors='replace')
if part.get_content_type() == 'text/html' and not html_body:
html_body = part.get_payload(decode=True).decode(part.get_content_charset(), errors='replace')
# Attachments
attachments = []
for part in msg.iter_attachments():
attachments.append({
'filename': part.get_filename(),
'content_type': part.get_content_type(),
'payload': part.get_payload(decode=True)
})
Beyond these basics, generate derived features such as:
- URL extraction and domain reputation scores (using services like VirusTotal, Cisco Talos).
- Link count and click‑through patterns.
- Language detection and sentiment analysis.
- Header anomalies (e.g., mismatched
Receivedhops). - Attachment metadata (file size, MIME type, hash values).
Store these features in a JSON object that will be fed to your AI model.
Step 3: Build a Prompt Engineering Pipeline
Large Language Models (LLMs) interpret natural language prompts. Crafting a prompt that balances context and brevity is key. Here’s a template you can adapt:
prompt = f"""
You are a cybersecurity analyst. Determine if the following email is phishing.
Subject: {subject}
From: {from_addr}
To: {to_addr}
Body (plain):
{plain_body}
Body (HTML):
{html_body}
Attachments: {[att['filename'] for att in attachments]}
Headers: {headers}
Answer with a single word: "Phishing" or "Legitimate".
Explain your reasoning in one sentence.
"""
Send the prompt to your chosen LLM endpoint. For OpenAI GPT‑4:
import openai
openai.api_key = 'YOUR_OPENAI_KEY'
response = openai.ChatCompletion.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':prompt}],
temperature=0,
max_tokens=150
)
result = response['choices'][0]['message']['content'].strip()
Parse the response to extract the verdict and the explanation. If you’re using a proprietary AI service, adjust the API call accordingly but keep the prompt structure consistent.
Step 4: Train a Custom Classification Model (Optional but Recommended)
While LLMs provide excellent zero‑shot reasoning, a fine‑tuned classifier can offer faster inference and lower cost. Use the extracted features to train a lightweight model (e.g., XGBoost, LightGBM, or a shallow neural net). Here’s a quick example with Scikit‑Learn:
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import pandas as pd
# Assume df contains feature columns + 'label' (1 for phishing, 0 for legitimate)
X = df.drop(columns=['label'])
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf = RandomForestClassifier(n_estimators=200, max_depth=12, random_state=42)
clf.fit(X_train, y_train)
# Save model
import joblib
joblib.dump(clf, 'phishing_classifier.pkl')
Deploy the model as a microservice (e.g., Flask or FastAPI) and expose a REST endpoint. Your AI‑prompt pipeline can then fallback to the classifier when latency is critical.
Step 5: Integrate with Your Email Gateway
Once you have a reliable verdict, the next step is to enforce it. Most gateways allow custom filtering rules. For example, in Microsoft Exchange Online Protection (EOP), you can create a Transport Rule that references an external script via a webhook:
# In EOP, create a new rule:
# If message header contains X-Phish-Result: Phishing
# Then move to quarantine
In your webhook service, set the X-Phish-Result header based on the AI verdict:
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route('/phish', methods=['POST'])
def phish_check():
email_payload = request.json # raw email JSON
# Parse, prompt, classify
verdict = run_ai_pipeline(email_payload)
response = {
'header': {
'X-Phish-Result': verdict
}
}
return jsonify(response)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8000)
For Google Workspace, you can use the Content Security Policy API to tag emails and set quarantine actions accordingly.
Step 6: Monitor, Log, and Iterate
Phishing tactics evolve; your detection system must adapt. Implement a logging pipeline that records:
- Raw email IDs and timestamps.
- AI verdicts and confidence scores.
- Model version and inference latency.
- Human‑review outcomes (if any).
Use a SIEM (e.g., Splunk, ELK) to surface trends. Set alerts for sudden spikes in false positives or negatives. Schedule quarterly retraining of your custom classifier with fresh data, and review the LLM prompt to incorporate new phishing templates.
Common Mistakes to Avoid
1. Relying solely on the AI verdict – Always cross‑validate with traditional checks (e.g., domain reputation, DKIM/SPF alignment). AI can hallucinate, especially with ambiguous prompts.
2. Ignoring attachment analysis – Phishers embed malicious payloads in PDFs or Office docs. Integrate sandbox scanning (e.g., Cuckoo Sandbox) before the AI step.
3. Over‑fitting your custom model – Use a diverse dataset covering multiple phishing campaigns. Employ cross‑validation and monitor for concept drift.
4. Under‑estimating latency – Large LLM calls can introduce 2–5 s delays. Cache results for identical subjects or use a lightweight fallback model.
5. Failing to secure the pipeline – Protect API keys, encrypt data in transit, and apply least‑privilege IAM roles.
Tips and Tricks
- Prompt Reuse – Store successful prompts in a template repository; update them when new phishing patterns emerge.
- Batch Processing – For high‑volume environments, batch 100–200 emails per API call to reduce overhead.
- Hybrid Scoring – Combine AI confidence with rule‑based scores (e.g., 0.7 * AI + 0.3 * rule) for a more robust verdict.
- Explainability – Use LLM’s explanation field to feed into a knowledge base; this aids analyst training.
- Zero‑Trust Post‑Processing – Even if an email passes the AI check, enforce a sandbox before opening attachments.
Frequently Asked Questions
How often should I retrain my custom classifier?
Ideally, retrain monthly with fresh labeled data. If you observe a significant drop in accuracy or a surge in false negatives, trigger an immediate retraining cycle.
Can I use free LLMs for phishing detection?
Free tiers (e.g., GPT‑3.5) can be used for low‑risk environments, but they have stricter rate limits and lower token capacities. For enterprise use, paid plans offer higher throughput and lower latency.
What if the AI flags a legitimate email as phishing?
Set up a manual review queue for borderline cases. Use the AI’s explanation to guide analysts. Over time, incorporate those false positives into your training set to improve accuracy.
Conclusion
Phishing is a moving target, but with an AI‑driven approach you can stay several steps ahead. By capturing raw emails, extracting structured features, leveraging LLMs for nuanced reasoning, and complementing them with a fine‑tuned classifier, you build a resilient defense that scales with your organization. Remember to monitor performance, iterate on prompts, and maintain a culture of continuous improvement. Armed with these tools and best practices, you’ll transform your email security from reactive to proactive—and keep the bad actors guessing.
Photo by Stephen Phillips – Hostreviews.co.uk on Unsplash
