How syncing works

When ASDK reads your website and sitemaps again, what happens to new, changed and deleted pages, what each status means and how to read a source again by hand.

Last updated October 21, 2026

On this page

Your website changes, and your agents should know about it. ASDK reads websites and sitemaps again on a schedule and processes only what changed, so your knowledge stays current without any work from you. This page explains the schedule, the statuses and what to do when you need a change to show up right away.

The schedule

Source	Read again
Website and sitemap sources	Every day; every week on the Free plan
Files	Never on their own: upload a new version to change one
Text snippets	When you save a change
Q&A pairs	When you save a change, within seconds

Each source card shows when it was last updated and when it reads again, such as "updated 3 hours ago · reads again in 21 hours". Sources added at the same time are spread out a little, so they don't all read at once.

What happens during a sync

ASDK reads the pages again

The crawler visits the pages of the source: by following links for a website source, or from the list for a sitemap. It follows your robots.txt every time.

Unchanged pages are skipped

ASDK compares each page's text with the version it already has. If nothing changed, the page stays as it is. Most syncs process only a few pages, which is why they are quick.

Changed and new pages are processed

Changed pages are split into passages again and replace the old version. New pages are added, as long as your plan's page limit leaves room.

Pages that are gone are removed

After a complete sync, pages that no longer exist on your site, that your sitemap no longer lists or that a skip pattern now excludes are removed from your knowledge. A page that answers "not found" is removed too. If a page fails for another reason, such as a timeout, the version ASDK already has is kept.

Read a source again now

To pick up a change without waiting for the schedule, open Knowledge and choose Read again on the source. On the source's own page, the same button is at the top. While it runs, the button shows Reading… and the card shows progress such as "Reading pages: 34 so far, 3 new or changed".

Editing a source's pages or text with Edit also reads it again: the dialog's button is Save and read again.

Statuses

Sources

Status	Meaning
Waiting	The source is queued and starts reading shortly.
Reading	ASDK is reading it now. Progress updates live.
Ready	The last read finished. The card shows how many pages are in your knowledge, and how many were skipped or couldn't be read.
Needs attention	The last read failed. The card says why and what to do.

The status of a site's knowledge, shown on Sites and at the top of Knowledge, sums up its sources: No knowledge yet, Starting, Reading, Ready or Needs attention.

Pages

Open a source with Pages to see every page it found. Filter the list with All, In knowledge, Skipped, Couldn't read and Excluded, or search by address or title.

Status	Meaning
In knowledge	Agents answer from this page. The row shows how many parts it was split into and its language.
Skipped	ASDK chose not to add it. The row says why, for example "Too little text to be useful."
Couldn't read	The page failed, for example with a server error. The row shows the reason.
Excluded	You excluded it, so ASDK doesn't read it. See Exclude pages.
Waiting	Found, not read yet.

When a sync fails

Some problems fix themselves: if your site was slow, briefly offline or asked ASDK to slow down, the source says "We'll try again" and does. Others need you:

"Your site's robots.txt doesn't allow ASDK-Crawler to read these pages."
"Your site refused our crawler (security or bot protection)."
"We couldn't find this website. Check the address for typos."

Crawl stuck or blocked explains each message and how to fix it.

Pausing and limits

A paused site isn't read on schedule. See Add your website.
The page limit applies to new pages. When your workspace is at its limit, pages already in your knowledge stay and are kept up to date, but new pages are skipped with "Page limit of your plan reached".
A source's own maximum stops it at the number you set in Most pages to read, with "Stopped at this source's limit of … pages. Raise the limit to read more."