AI Threat Alert
Attack HIGH
Prompt Injection as Role Confusion
Charles Ye, Jasmine Cui, Dylan Hadfield-Menell
Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models...
AI Security Research
AI Threat Alert indexes 3,397+ peer-reviewed and preprint papers on AI/ML security — covering adversarial attacks, model defenses, red-teaming benchmarks, surveys, and security tooling. Papers are sourced from arXiv, classified by type and by relevance to real-world threats, and cross-referenced with the CVEs and incidents they relate to.
- Adversarial attacks
- Model defenses
- Red-teaming benchmarks
- Surveys
- Security tooling
- Total3,397
- Attack1,314
- Benchmark975
- Defense477
- Tool341
- Survey196
Showing 541–560 of 1,146 papers
Clear filters Attack HIGH
Dark and Bright Side of Participatory Red-Teaming with Targets of Stereotyping for Eliciting Harmful Behaviors from Large Language Models
Sieun Kim, Yeeun Jo, Sungmin Na +5 more
Red-teaming, where adversarial prompts are crafted to expose harmful behaviors and assess risks, offers a dynamic approach to surfacing underlying...
Attack HIGH
When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attacks
Shenyang Chen, Liuwan Zhu
Standard evaluations of backdoor attacks on text-to-image (T2I) models primarily measure trigger activation and visual fidelity. We challenge this...
Attack HIGH
When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models
Zafir Shamsi, Nikhil Chekuru, Zachary Guzman +1 more
Large Language Models (LLMs) are increasingly integrated into high-stakes applications, making robust safety guarantees a central practical and...
Benchmark HIGH
FENCE: A Financial and Multimodal Jailbreak Detection Dataset
Mirae Kim, Seonghun Jeong, Youngjun Kwak
Jailbreaking poses a significant risk to the deployment of Large Language Models (LLMs) and Vision Language Models (VLMs). VLMs are particularly...
Tool HIGH
PenTiDef: Enhancing Privacy and Robustness in Decentralized Federated Intrusion Detection Systems against Poisoning Attacks
Phan The Duy, Nghi Hoang Khoa, Nguyen Tran Anh Quan +3 more
The increasing deployment of Federated Learning (FL) in Intrusion Detection Systems (IDS) introduces new challenges related to data privacy,...
Attack HIGH
TFL: Targeted Bit-Flip Attack on Large Language Model
Jingkai Guo, Chaitali Chakrabarti, Deliang Fan
Large language models (LLMs) are increasingly deployed in safety and security critical applications, raising concerns about their robustness to model...
Attack HIGH
Trojan Horses in Recruiting: A Red-Teaming Case Study on Indirect Prompt Injection in Standard vs. Reasoning Models
Manuel Wirth
As Large Language Models (LLMs) are increasingly integrated into automated decision-making pipelines, specifically within Human Resources (HR), the...
Attack HIGH
Automating Agent Hijacking via Structural Template Injection
Xinhao Deng, Jiaqing Wu, Miao Chen +3 more
Agent hijacking, highlighted by OWASP as a critical threat to the Large Language Model (LLM) ecosystem, enables adversaries to manipulate execution...
Benchmark HIGH
IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages
Priyaranjan Pattnayak, Sanchari Chowdhuri
Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities...
Attack HIGH
Sequential Membership Inference Attacks
Thomas Michel, Debabrota Basu, Emilie Kaufmann
Modern AI models are not static. They go through multiple updates in their lifecycles. Thus, exploiting the model dynamics to create stronger...
Tool HIGH
Recursive language models for jailbreak detection: a procedural defense for tool-augmented agents
Doron Shavit
Jailbreak prompts are a practical and evolving threat to large language models (LLMs), particularly in agentic systems that execute tools over...
Attack HIGH
SRFed: Mitigating Poisoning Attacks in Privacy-Preserving Federated Learning with Heterogeneous Data
Yiwen Lu
Federated Learning (FL) enables collaborative model training without exposing clients' private data, and has been widely adopted in privacy-sensitive...
Attack HIGH
The Vulnerability of LLM Rankers to Prompt Injection Attacks
Yu Yin, Shuai Wang, Bevan Koopman +1 more
Large Language Models (LLMs) have emerged as powerful re-rankers. Recent research has however showed that simple prompt injections embedded within a...
Survey HIGH
Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis
Scott Thornton
AI-assisted code review is widely used to detect vulnerabilities before production release. Prior work shows that adversarial prompt manipulation can...
Attack HIGH
Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections
Xianglin Yang, Yufei He, Shuo Ji +2 more
Self-evolving LLM agents update their internal state across sessions, often by writing and reusing long-term memory. This design improves performance...
Attack HIGH
ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Models
Mitchell Piehl, Zhaohan Xi, Zuobin Xiong +2 more
Large language models (LLMs) are increasingly augmented with long-term memory systems to overcome finite context windows and enable persistent...
Attack HIGH
Boundary Point Jailbreaking of Black-Box LLMs
Xander Davies, Giorgi Giglemiani, Edmund Lau +3 more
Frontier LLMs are safeguarded against attempts to extract harmful information via adversarial prompts known as "jailbreaks". Recently, defenders have...
Attack HIGH
Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks
Lukas Struppek, Adam Gleave, Kellin Pelrine
As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on...
Attack HIGH
Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models
In Chong Choi, Jiacheng Zhang, Feng Liu +1 more
Multi-turn jailbreak attacks are effective against text-only large language models (LLMs) by gradually introducing malicious content across turns....
Frequently asked questions
What is AI security research?
AI security research studies how AI and machine-learning systems can be attacked and defended — covering adversarial examples, prompt injection, model poisoning, training-data extraction, and the mitigations against them. AI Threat Alert curates this research from academic sources so security teams can track the threats behind emerging AI risks.
How many AI security papers does AI Threat Alert track?
AI Threat Alert indexes 3,397+ papers on AI/ML security, classified across attack, defense, benchmark, survey, and tool categories and updated continuously.
Where do the research papers come from?
Papers are sourced from arXiv, then classified by type and by relevance to real-world AI/ML threats, and cross-referenced with the CVEs and incidents they relate to.
What topics does the AI security research cover?
Coverage spans adversarial attacks, model and system defenses, red-teaming benchmarks, literature surveys, and security tooling for LLMs, ML libraries, AI agents, and inference pipelines.
How is this different from a generic paper search?
Every paper is filtered for AI security relevance and linked to the vulnerabilities, vendors, and incidents it relates to, so the research connects directly to operational threat intelligence.
Track AI security vulnerabilities in real time
Get breaking CVE alerts, compliance reports (ISO 42001, EU AI Act), and CISO risk assessments for your AI/ML stack.
Start 14-Day Free Trial