Ahrefs

Ahrefs is the crawl I use when the question is off-site: who links here, which keyword is actually contested, what their bot sees after a deploy — not Lighthouse, not your own analytics. Site Audit, Backlink profile, Keywords Explorer, Content Explorer, Rank Tracker. The index is the product.

Paid. Not a library. I keep ahrefs.com next to PostHog on purpose: one is product behavior, the other is how the web sees the site.