Crawlers
ASDK crawler and checker
What the ASDK-Crawler and ASDK-AgentReadinessChecker user agents are, when they visit a website, how they follow robots.txt and how to block them.
On this page
If you found ASDK-Crawler or ASDK-AgentReadinessChecker in your server logs, this page explains who sent the requests, why, and how to stop them if you want to.
The two user agents
ASDK sends requests to other websites for two reasons only, and each has its own user agent so you can tell them apart:
| User agent | What it does | When it visits |
|---|---|---|
ASDK-Crawler/1.0 (+https://asdk.com/bot) | Reads a website's public pages to build that site's knowledge base | Only sites whose owners added them to ASDK |
ASDK-AgentReadinessChecker/1.0 (+https://asdk.com/tools/agent-readiness-checker) | Checks one website for how readable it is for AI agents | Only when someone enters that site's address in our free agent readiness checker |
The product token you can use in robots.txt is the part before the slash: ASDK-Crawler or ASDK-AgentReadinessChecker.
ASDK-Crawler
When a business adds its website to ASDK, the crawler reads that site's public pages so the business's AI agents can answer questions from them. It does not crawl the web at large and does not visit sites that nobody has connected to ASDK. If you see it on your site and did not connect the site yourself, someone with access to it may have; contact us and we will look into it.
The crawler:
- reads robots.txt first and follows the rules for
ASDK-Crawler, or the rules for all user agents (*) when there is no group for it; - honors
Crawl-delayand keeps its request rate low even when none is set; - reads only public pages: it does not log in, submit forms or run purchases.
ASDK-AgentReadinessChecker
The free agent readiness checker looks at a single website on request: someone types an address and the checker fetches a small number of files from that site — robots.txt, the sitemap, llms.txt and the home page — to report how readable the site is for AI agents. It runs only when a person starts a check, never on a schedule, and it follows robots.txt and Crawl-delay in the same way as the crawler.
How to block them
To block either user agent from your whole site, add a group for it to your robots.txt file:
User-agent: ASDK-Crawler
Disallow: /
User-agent: ASDK-AgentReadinessChecker
Disallow: /To keep them out of part of your site only, list those paths instead:
User-agent: ASDK-Crawler
Disallow: /internal/
Disallow: /account/To slow the crawler down, set a delay in seconds between requests:
User-agent: ASDK-Crawler
Crawl-delay: 10Changes take effect the next time the crawler reads your robots.txt. Our article on robots.txt for AI bots explains how robots.txt groups work and which other AI crawlers you may want to allow or block.
Contact
If either user agent causes a problem on your site, or you think a request did not follow your robots.txt, email hello@asdk.com with your domain, the time of the requests and, if you can, a few lines from your server log. To report a security issue, see our security page.