Crawl stuck or blocked

ASDK can't read your site, reads only a few pages or seems stuck? Fixes for robots.txt, firewalls and bot protection, JavaScript-only pages and address errors.

Last updated October 21, 2026

On this page

ASDK's crawler, ASDK-Crawler, reads your public pages the way a search engine does. When a source shows Needs attention, finds far fewer pages than your site has, or keeps reading for a long time, one of the causes below is usually responsible. Each source shows a message that points to it.

Reading takes a long time

A first read of a large site takes time: the crawler keeps its request rate low so it never slows your site down, and it waits longer if your robots.txt sets a Crawl-delay. While it works, the source card shows progress such as "Reading pages: 340 so far". As long as that number grows, the crawl isn't stuck.

You don't need to wait: agents start answering from the first pages as soon as they're read. If the number hasn't changed for a long time, choose Read again on the source.

"Your site's robots.txt doesn't allow ASDK-Crawler to read these pages."

Your robots.txt disallows the crawler, either by name or with a rule for all crawlers (User-agent: *). Open https://yoursite.com/robots.txt and allow it:

Text

User-agent: ASDK-Crawler
Allow: /

A group for ASDK-Crawler takes precedence over the * group, so you can allow ASDK while keeping other rules. Then choose Read again. The crawler page explains how the crawler reads robots.txt.

"Your site refused our crawler (security or bot protection)."

A firewall, CDN or security plugin answered the crawler with a block or a challenge page. Allow the crawler in that tool:

Security plugins such as those for WordPress: look for an allowlist of user agents or "good bots", and add ASDK-Crawler.
Firewalls and CDNs: add a rule that lets requests through whose user agent contains ASDK-Crawler. The full user agent is ASDK-Crawler/1.0 (+https://asdk.com/bot).

On Cloudflare, a WAF custom rule with the Skip action can let a request bypass Super Bot Fight Mode, managed rules and rate limiting. Bot Fight Mode, the free plan's bot protection, can't be skipped by any rule: to let the crawler in, use Super Bot Fight Mode with a skip rule instead, or turn Bot Fight Mode off. Cloudflare's pages on Bot Fight Mode and the Skip action describe both (checked October 2026).

If you can't change the protection, add your content another way: upload key pages as files or write the facts as text snippets.

Single pages can be refused too. They show Couldn't read with "Your site refused our crawler for this page (bot protection)."

Pages built with JavaScript

Some sites send an almost empty page and build the text in the browser with JavaScript. The crawler reads the page your server sends, so on such sites:

pages show as Skipped with "Too little text to be useful.";
the crawl finds only a few pages, because the menu links are built by JavaScript too.

Fixes, from best to simplest:

Render pages on the server, or prerender them, so the HTML contains the text. Most frameworks support this, and search engines and AI assistants read such pages better too.
Add a sitemap, so ASDK knows every page even when it can't follow your menus.
Upload the content as files or text snippets.

Few pages found

If the crawl finishes as Ready with far fewer pages than your site has:

Pages aren't linked: the crawler follows links from your start page. Add a sitemap.
A pattern skips them: check Only pages under and Skip pages under with Edit on the source.
The source starts in a section: a source that starts at /help reads only pages under /help.
A limit stopped it: "Stopped at this source's limit of … pages" or "Page limit of your plan reached". See Page limits.
Duplicates: pages with the same content as another page are kept once and show as Skipped.

Address and server errors

Message	What to do
We couldn't find this website. Check the address for typos.	The domain doesn't resolve. Check the spelling and that the site is online.
This page doesn't exist (it returned "not found"). Check the address.	Fix the start address with a new source.
This address isn't a web page. To add a PDF or document, upload it as a file.	The address points to a file. Upload it instead.
Your site answered with an error (…) for this page.	The site refused the request. Check that the page opens in a browser.
Your site took too long to respond. We'll try again.	Temporary. ASDK retries on its own.
Your site asked us to slow down. We'll try again later.	Temporary. Your server signaled too many requests.
Your site returned a server error (…). We'll try again.	Temporary, unless your site is down.
We couldn't connect to your site. We'll try again.	Temporary network trouble.

Messages that say "We'll try again" need no action: the source retries by itself.

The crawler reads only public pages. It doesn't sign in, fill in forms or pass a password prompt. Staging sites behind a password, customer portals and intranets can't be crawled. Add their content as files or text snippets, or let ASDK read the public site once it launches.

Still stuck?

Write to hello@asdk.com with your site's address, the source's message and your Workspace ID. If you can, add a few lines from your server log showing the crawler's requests and the responses your server sent.