Bot signals · Web crawler

What is Similarweb Site Audit crawler?

Similarweb’s Site Audit tool is built on a website crawler that crawls a site’s pages to collect data for website analytics. Similarweb says it aims to crawl the way Google does.

risk.is_crawler

All web crawlers

01 · Profile

Similarweb Site Audit crawler at a glance.

What to look for in your logs, and what the lookup API returns for its addresses.

Type: Web crawler
robots.txt: User-agent: similarweb
API flag: risk.is_crawler

A user agent alone never sets a flag. Anyone can claim to be Similarweb Site Audit crawler; the flag follows the address the request comes from.

02 · In the response

What a match looks like.

A lookup of a Similarweb Site Audit crawler address sets risk.is_crawler. The flags come with geolocation and network data in the same response, for API keys on every paid plan.

The response says which kind of bot an address belongs to, not which operator. A request that claims to be Similarweb Site Audit crawler while risk.is_crawler is false should not be trusted as Similarweb Site Audit crawler.

GET /api/lookup/{ip}?key=YOUR_API_KEY

{
  // … location and network fields …
  "risk": {
    "is_ai": false,
    "is_crawler": true,
    "is_scanner": false
  }
}

03 · Control

Allow or block Similarweb Site Audit crawler.

robots.txt

User-agent: similarweb
Disallow: /

Similarweb says its crawler obeys robots.txt and will not crawl a site that excludes the similarweb token. It does not support crawl-delay.

To enforce it

robots.txt is a request. Well-behaved bots follow it; impostors using its name don’t.
Firewall, WAF or CDN. Many offer bot rules you can allow or deny by category. Use them to enforce what robots.txt only asks for.
In your app. Look up the client address and allow or deny on risk.is_crawler, alongside the user agent.
Never on the user agent alone. Blocking the name stops only honest clients; allowing it lets anyone in who borrows it.