AI red teaming with open source: Garak, PyRIT, Promptfoo and more

AI red teaming means a team deliberately attacks your AI application before a real attacker does. It tries to slip new instructions to the model, lure out confidential data or misuse tools. Much of this can be automated with free software. The main tools are openly available on GitHub.

The tools at a glance

Garak: the quick baseline check

Garak works like a vulnerability scanner for language models. You give it an endpoint, Garak sends a library of known attacks and delivers a report. Among other things, it checks jailbreaks, prompt injection, leaking of training data and harmful content. Garak is the right start when you want to know quickly where a model stands in general.

The limit: Garak doesn't know your application. It tests the model, not the tools, data and permissions around it.

PyRIT: attacks made to measure

PyRIT comes from Microsoft's AI Red Team. It is a toolkit in Python, not a finished scanner. You use it to build attacks that run over several rounds: an attacker model talks to your application, varies its inputs and rates the answers. This lets you test scenarios that fit your system exactly. For example, whether your support agent reveals customer data after ten friendly messages.

The limit: PyRIT takes developer time. Without a clear goal, you get a lot of effort and little result.

Promptfoo: tests on every change

Promptfoo is a command-line tool that tests prompts and models against each other. It generates attacks that fit the application and runs in the CI pipeline. Every change to a prompt or a model switch is checked automatically. OpenAI acquired Promptfoo in March 2026. According to the founders, the project remains open source and continues to support models from other vendors.

The limit: Promptfoo reliably finds regressions. New attack routes you have to teach it first.

DeepTeam, Giskard and AgentDojo

• DeepTeam from Confident AI covers the risks of the OWASP LLM Top 10 with ready-made tests. Handy for teams that already use DeepEval to measure quality.
• Giskard checks quality as well as security: wrong answers, contradictions, bias. A mode for agents is new.
• AgentDojo comes from research at ETH Zurich. It measures how well agents hold up against hidden instructions while working with email, calendar and banking. Good for comparing different models or protective measures.

How a test works with us

1. Take stock. Which AI systems are running, which data do they read, which actions may they trigger?
2. Baseline scan. Garak against every model in use. This shows the known weaknesses.
3. Targeted scenarios. With PyRIT and our own scripts, we attack the tools and data sources that only your application has.
4. Manual work. A person looks for gaps in permissions and business logic. For example, whether an agent can trigger a payment through a detour.
5. Report with measures. Every finding with an example, the risk and a fix.
6. Regression tests. The attacks we found go into the pipeline as Promptfoo tests. That way a closed gap doesn't quietly reappear.

What no tool replaces

The tools find known patterns. But the most dangerous gaps often aren't in the model. They are in what was built around the model: too many permissions, a missing approval, a data source that strangers can write to. What that looks like is shown in our article on agentjacking. There, a fake error message was enough.

On top of that, language models don't answer the same way every time. An attack that fails nine times can work on the tenth. Good tests repeat every attack several times and measure rates instead of single cases.

Frequently asked questions