Semrush SiteAuditBot
risk.is_crawler
- Method
- Page parser Published ranges, parsed from a documentation page
- Format
- Documentation page, parsed. A parser reads only the scoped address section; if the page structure changes, the update is rejected rather than guessed.
- Refresh
- Every hour. If a refresh fails, entries expire after 24 hours instead of being trusted forever.