Bot signals · Web crawler
What is HubSpot Crawler?
HubSpot web crawling egress
HubSpot uses web crawlers to keep the information in its analytics tools up to date and to show live previews of its customers’ pages. HubSpot’s SEO and import tools also crawl pages, including external pages a HubSpot user asks them to scan, with the user agent HubSpot Crawler.
01 · Profile
HubSpot Crawler at a glance.
What to look for in your logs, and what the lookup API returns for its addresses.
- Type
- Web crawler
- User agent
HubSpot Crawler- robots.txt
User-agent: HubSpot Crawler- API flag
risk.is_crawler
A user agent alone never sets a flag. Anyone can claim to be HubSpot Crawler; the flag follows the address the request comes from.
02 · In the response
What a match looks like.
A lookup of a HubSpot Crawler address sets risk.is_crawler. The flags come with geolocation and network data in the same response, for API keys on every paid plan.
The response says which kind of bot an address belongs to, not which operator. A request that claims to be HubSpot Crawler while risk.is_crawler is false should not be trusted as HubSpot Crawler.
{
// … location and network fields …
"risk": {
"is_ai": false,
"is_crawler": true,
"is_scanner": false
}
}
03 · Control
Allow or block HubSpot Crawler.
User-agent: HubSpot Crawler Disallow: /
To enforce it
- robots.txt is a request. Well-behaved bots follow it; impostors using its name don’t.
- Firewall, WAF or CDN. Many offer bot rules you can allow or deny by category. Use them to enforce what robots.txt only asks for.
- In your app. Look up the client address and allow or deny on
risk.is_crawler, alongside the user agent. - Never on the user agent alone. Blocking the name stops only honest clients; allowing it lets anyone in who borrows it.
04 · Questions
About HubSpot Crawler.
What is HubSpot Crawler?
HubSpot uses web crawlers to keep the information in its analytics tools up to date and to show live previews of its customers’ pages. HubSpot’s SEO and import tools also crawl pages, including external pages a HubSpot user asks them to scan, with the user agent HubSpot Crawler.
How do I block HubSpot Crawler?
Add a robots.txt rule for HubSpot Crawler. HubSpot’s SEO tools check robots.txt, and HubSpot asks site owners who want a blocked page scanned to add an exemption for HubSpot Crawler. To enforce a block rather than request one, deny it at your firewall, WAF or CDN.
Can I trust a request that says it is HubSpot Crawler?
Not on the user agent alone: any client can send “HubSpot Crawler” in its user agent. A user agent alone never sets a flag. Check the address with the free bot checker or the lookup API.
Which risk flag does HubSpot Crawler set?
Its addresses set risk.is_crawler in the lookup response. The risk flags come with every paid plan.
From the blog
Go deeper.
Related bots
Recognise HubSpot Crawler on every request.
is_ai, is_crawler and is_scanner in every lookup, on every paid plan, from 9,00 € a month (VAT included).