AI Threat AlertBenchmarkMEDIUM
Blue Teaming Function-Calling Agents
Greta Dolcetti, Giulio Zizzo, Sergio Maffeis
We present an experimental evaluation that assesses the robustness of four open source LLMs claiming function-calling capabilities against three...
AI Security Research
AI Threat Alert indexes 4,319+ peer-reviewed and preprint papers on AI/ML security — covering adversarial attacks, model defenses, red-teaming benchmarks, surveys, and security tooling. Papers are sourced from arXiv, classified by type and by relevance to real-world threats, and cross-referenced with the CVEs and incidents they relate to.
- Adversarial attacks
- Model defenses
- Red-teaming benchmarks
- Surveys
- Security tooling
- Total4,319
- Attack1,676
- Benchmark1,229
- Defense610
- Tool432
- Survey252
Showing 3101–3120 of 4,319 papers
AttackMEDIUM
UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning
Feng Zhang, Shijia Li, Chunmao Zhang +7 more
User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and...
DefenseMEDIUM
Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety
Caitlin A. Stamatis, Jonah Meyerhoff, Richard Zhang +3 more
Large language models (LLMs) are increasingly used for mental health support, yet existing safety evaluations rely primarily on small,...
AttackHIGH
KryptoPilot: An Open-World Knowledge-Augmented LLM Agent for Automated Cryptographic Exploitation
Xiaonan Liu, Zhihao Li, Xiao Lan +3 more
Capture-the-Flag (CTF) competitions play a central role in modern cybersecurity as a platform for training practitioners and evaluating offensive and...
AttackHIGH
Too Helpful to Be Safe: User-Mediated Attacks on Planning and Web-Use Agents
Fengchao Chen, Tingmin Wu, Van Nguyen +1 more
Large Language Models (LLMs) have enabled agents to move beyond conversation toward end-to-end task execution and become more helpful. However, this...
BenchmarkHIGH
LLMs in Code Vulnerability Analysis: A Proof of Concept
Shaznin Sultana, Sadia Afreen, Nasir U. Eisty
Context: Traditional software security analysis methods struggle to keep pace with the scale and complexity of modern codebases, requiring...
AttackMEDIUM
SafeRedir: Prompt Embedding Redirection for Robust Unlearning in Image Generation Models
Renyang Liu, Kangjie Chen, Han Qiu +4 more
Image generation models (IGMs), while capable of producing impressive and creative content, often memorize a wide range of undesirable concepts from...
BenchmarkMEDIUM
Robust CAPTCHA Using Audio Illusions in the Era of Large Language Models: from Evaluation to Advances
Ziqi Ding, Yunfeng Wan, Wei Song +7 more
CAPTCHAs are widely used by websites to block bots and spam by presenting challenges that are easy for humans but difficult for automated programs to...
BenchmarkMEDIUM
STAR: Detecting Inference-time Backdoors in LLM Reasoning via State-Transition Amplification Ratio
Seong-Gyu Park, Sohee Park, Jisu Lee +2 more
Recent LLMs increasingly integrate reasoning mechanisms like Chain-of-Thought (CoT). However, this explicit reasoning exposes a new attack surface...
BenchmarkMEDIUM
BenchOverflow: Measuring Overflow in Large Language Models via Plain-Text Prompts
Erin Feiglin, Nir Hutnik, Raz Lapid
We investigate a failure mode of large language models (LLMs) in which plain-text prompts elicit excessive outputs, a phenomenon we term Overflow....
AttackMEDIUM
YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation
Abdelaziz Bounhar, Rania Hossam Elmohamady Elbadry, Hadi Abdine +3 more
Steering Large Language Models (LLMs) through activation interventions has emerged as a lightweight alternative to fine-tuning for alignment and...
DefenseMEDIUM
DNF: Dual-Layer Nested Fingerprinting for Large Language Model Intellectual Property Protection
Zhenhua Xu, Yiran Zhao, Mengting Zhong +4 more
The rapid growth of large language models raises pressing concerns about intellectual property protection under black-box deployment. Existing...
DefenseLOW
Subspace Alignment for Vision-Language Model Test-time Adaptation
Zhichen Zeng, Wenxuan Bao, Xiao Lin +8 more
Vision-language models (VLMs), despite their extraordinary zero-shot capabilities, are vulnerable to distribution shifts. Test-time adaptation (TTA)...
ToolLOW
Towards Verifiably Safe Tool Use for LLM Agents
Aarya Doshi, Yining Hong, Congying Xu +3 more
Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code...
AttackHIGH
SecureCAI: Injection-Resilient LLM Assistants for Cybersecurity Operations
Mohammed Himayath Ali, Mohammed Aqib Abdullah, Mohammed Mudassir Uddin +1 more
Large Language Models have emerged as transformative tools for Security Operations Centers, enabling automated log analysis, phishing triage, and...
BenchmarkMEDIUM
Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation
Dongryeol Lee, Yerin Hwang, Taegwan Kang +3 more
While large language models (LLMs) are increasingly used as automatic judges for question answering (QA) and other reference-conditioned evaluation...
AttackMEDIUM
MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCP
Ruiqi Li, Zhiqiang Wang, Yunhao Yao +1 more
To standardize interactions between LLM-based agents and their environments, the Model Context Protocol (MCP) was proposed and has since been widely...
AttackHIGH
When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agent
Xinyi Wu, Geng Hong, Yueyue Chen +5 more
Web agents, powered by large language models (LLMs), are increasingly deployed to automate complex web interactions. The rise of open-source...
BenchmarkLOW
ActiShade: Activating Overshadowed Knowledge to Guide Multi-Hop Reasoning in Large Language Models
Huipeng Ma, Luan Zhang, Dandan Song +10 more
In multi-hop reasoning, multi-round retrieval-augmented generation (RAG) methods typically rely on LLM-generated content as the retrieval query....
BenchmarkMEDIUM
Small Symbols, Big Risks: Exploring Emoticon Semantic Confusion in Large Language Models
Weipeng Jiang, Xiaoyu Zhang, Juan Zhai +3 more
Emoticons are widely used in digital communication to convey affective intent, yet their safety implications for Large Language Models (LLMs) remain...
Frequently asked questions
What is AI security research?
AI security research studies how AI and machine-learning systems can be attacked and defended — covering adversarial examples, prompt injection, model poisoning, training-data extraction, and the mitigations against them. AI Threat Alert curates this research from academic sources so security teams can track the threats behind emerging AI risks.
How many AI security papers does AI Threat Alert track?
AI Threat Alert indexes 4,319+ papers on AI/ML security, classified across attack, defense, benchmark, survey, and tool categories and updated continuously.
Where do the research papers come from?
Papers are sourced from arXiv, then classified by type and by relevance to real-world AI/ML threats, and cross-referenced with the CVEs and incidents they relate to.
What topics does the AI security research cover?
Coverage spans adversarial attacks, model and system defenses, red-teaming benchmarks, literature surveys, and security tooling for LLMs, ML libraries, AI agents, and inference pipelines.
How is this different from a generic paper search?
Every paper is filtered for AI security relevance and linked to the vulnerabilities, vendors, and incidents it relates to, so the research connects directly to operational threat intelligence.
Track AI security vulnerabilities in real time
Get breaking CVE alerts, compliance reports (ISO 42001, EU AI Act), and CISO risk assessments for your AI/ML stack.
Start 14-Day Free Trial