Bot signals · Web crawler
What is WPMU DEV Broken Link Checker?
Broken Link Checker is WPMU DEV’s WordPress plugin that scans a site for broken links and reports them. Its cloud version runs the scan on WPMU DEV’s servers, and these requests come from that cloud crawler.
01 · Profile
WPMU DEV Broken Link Checker at a glance.
What to look for in your logs, and what the lookup API returns for its addresses.
- Type
- Web crawler
- User agent
WPMU DEV Broken Link Checker Spider- robots.txt
User-agent: WPMU DEV Broken Link Checker Spider- API flag
risk.is_crawler
A user agent alone never sets a flag. Anyone can claim to be WPMU DEV Broken Link Checker; the flag follows the address the request comes from.
02 · In the response
What a match looks like.
A lookup of a WPMU DEV Broken Link Checker address sets risk.is_crawler. The flags come with geolocation and network data in the same response, for API keys on every paid plan.
The response says which kind of bot an address belongs to, not which operator. A request that claims to be WPMU DEV Broken Link Checker while risk.is_crawler is false should not be trusted as WPMU DEV Broken Link Checker.
{
// … location and network fields …
"risk": {
"is_ai": false,
"is_crawler": true,
"is_scanner": false
}
}
03 · Control
Allow or block WPMU DEV Broken Link Checker.
User-agent: WPMU DEV Broken Link Checker Spider Disallow: /
To enforce it
- robots.txt is a request. Well-behaved bots follow it; impostors using its name don’t.
- Firewall, WAF or CDN. Many offer bot rules you can allow or deny by category. Use them to enforce what robots.txt only asks for.
- In your app. Look up the client address and allow or deny on
risk.is_crawler, alongside the user agent. - Never on the user agent alone. Blocking the name stops only honest clients; allowing it lets anyone in who borrows it.
04 · Questions
About WPMU DEV Broken Link Checker.
What is WPMU DEV Broken Link Checker?
Broken Link Checker is WPMU DEV’s WordPress plugin that scans a site for broken links and reports them. Its cloud version runs the scan on WPMU DEV’s servers, and these requests come from that cloud crawler.
How do I block WPMU DEV Broken Link Checker?
Add a robots.txt rule for WPMU DEV Broken Link Checker Spider. WPMU DEV says the cloud Broken Link Checker reads robots.txt and respects rules such as Disallow and Crawl-delay in a group for WPMU DEV Broken Link Checker Spider. To enforce a block rather than request one, deny it at your firewall, WAF or CDN.
Can I trust a request that says it is WPMU DEV Broken Link Checker?
Not on the user agent alone: any client can send “WPMU DEV Broken Link Checker Spider” in its user agent. A user agent alone never sets a flag. Check the address with the free bot checker or the lookup API.
Which risk flag does WPMU DEV Broken Link Checker set?
Its addresses set risk.is_crawler in the lookup response. The risk flags come with every paid plan.
From the blog
Go deeper.
Related bots
Recognise WPMU DEV Broken Link Checker on every request.
is_ai, is_crawler and is_scanner in every lookup, on every paid plan, from 9,00 € a month (VAT included).