Privacy Policy

Last updated 2026-10-05.

AgentProbe is a product of Sampact LLC. On this site, "we" and "us" mean Sampact LLC.

What we collect

• When you sign in with GitHub: your GitHub username, display name, avatar URL, and email address (whatever GitHub's OAuth gives us) -- see github.com/settings/connections to review or revoke this at any time.
• A session cookie that keeps you signed in. It's necessary for the site to work, not used for tracking or advertising.
• Domains you generate -- the business description and tool schemas you submit, and the resulting generated code, so it can be shown back to you under Your Runs.
• Runs you upload or execute -- the scenario id, pass/fail results, cost, and the full rendered report, so it can be shown back to you under Your Runs.
• Baselines -- which of your runs you've marked as the reference for an agent (identified by the endpoint URL you ran it against), so we can compare later runs with it.
• Priority edits -- if you change how an issue is ranked on a report, we store which kind of issue and the High/Medium/Low you chose, with your account, so we can apply it to your future reports.
• A structured summary of each run's results -- for every failure we injected, its type (for example "tool error" or "stale data"), how your agent responded (handled, ignored, misread, overreacted or looped), the step it happened at, and whether the scenario passed. It contains no scenario names, no text from your agent or your business rules, and no model labels you typed. It's kept with your account. It is not used for anything beyond your own runs unless you opt in (see "Anonymized benchmarks" below).
• Your own Anthropic API key, if you use /run -- used only for that one request, sent directly to our run-execution worker over an internal connection, and never written to any database, log, or file. We have no record of it after the request finishes.

Don't want your agent's traffic touching our servers at all?

Using this website's own "Start run" button does send your target agent's real request/response traffic through our run-execution worker (see above -- your API key itself is never stored, but the traffic still passes through our infrastructure to run the test). If that's not acceptable for what your agent handles, you don't need this website at all: pip install agentprobe-testing runs the identical test standalone, in your own process, talking directly to your agent and to Anthropic -- every line of it lands on your machine as plain, readable Python (a real package install, not a compiled binary), so you can verify exactly what it does before trusting it with anything. See the "Run locally instead" panel on any run's configuration page for a ready-to-paste example.

Anonymized benchmarks (optional, off by default)

When you start a run you can tick "Help improve AgentProbe". If you do, the structured summary described above for that run (and only that: injected failure types, how your agent responded, the step number, and pass/fail) may be combined with other opted-in users' results into aggregate statistics, such as how often agents handle each kind of failure. We use these to improve AgentProbe and may publish or share the aggregate numbers. We only report a group of results when it comes from at least three different users, so no figure describes a single user's agent.

We never include your account, GitHub details, agent name or endpoint, business rules, tool schemas, prompts, anything your agent said, your reports, or the model labels you typed. Aggregated, anonymous statistics aren't tied back to you and aren't "selling your data". You can withdraw at any time by emailing us (see the contact on the home page): we'll exclude everything you contributed from future aggregates. Numbers already published as part of an aggregate can't be un-published, since they can't be traced back to any one user. If you use the Python library's upload_run, the same rule applies: nothing is shared unless you pass share_anonymized=True.

What we don't collect

No advertising trackers, no analytics cookies, no third-party tracking scripts. We don't sell or rent your data to anyone.

Who else sees it

Generating a domain sends your business description and tool schemas to Anthropic's API to produce the result -- that's the only third party your data passes through, and only for that purpose. See Anthropic's own privacy policy for how they handle it.

How long we keep it

For as long as your account exists. There's currently no self-serve "delete my account" button -- email us (see the contact on the home page) and we'll delete your account and associated data by hand (including your structured results; anonymous aggregates you opted into can't be taken back out, as described above).

Children's privacy

This service isn't directed at children under 13, and we don't knowingly collect data from them.

Changes

If this policy changes in a way that matters, we'll update the date at the top of this page.