Exclude pages

Keep pages out of your agents' knowledge: exclude single pages, skip whole sections with path patterns, or remove a page or block the crawler instead.

Last updated October 21, 2026

On this page

Not every page of your website should shape your agents' answers. Old announcements, outdated price pages, internal sections and archive pages can lead to wrong or confusing replies. This page shows the ways to keep pages out, and when to use each one.

Four ways to keep a page out

Method	Scope	Lasts
Exclude a page	One page of one source	Until you undo it
Skip pages under	A section, by path pattern	Until you change the source
Remove a page	One page	Until the next sync adds it back
A robots.txt rule for `ASDK-Crawler`	Any part of the site, for ASDK only	Until you change robots.txt

For most cases, Exclude and Skip pages under are what you want.

Exclude a single page

Open the source's pages

Open Knowledge, find the website or sitemap source, and choose Pages. Search by address or title to find the page.

Exclude it

Choose Exclude on the page and confirm with Exclude page. The page leaves your knowledge right away, its status changes to Excluded, and ASDK won't read it again.

Behind the scenes, excluding adds the page's address to the source's Skip pages under list. To bring the page back, choose Edit on the source, delete that line and choose Save and read again.

A few limits apply:

The home page can't be excluded, because reading starts there. To stop reading a site, delete its source.
Only website and sitemap pages can be excluded. To remove a file or a text snippet, delete its source.
A source can skip up to 50 paths. If you reach that, combine pages into a pattern, such as /blog/** instead of single posts.

Skip whole sections with patterns

Choose Edit on a website or sitemap source. The Choose which pages dialog has a Skip pages under box: one path per line. Saving reads the source again with the new choice.

How patterns match:

Pattern	Matches
`/blog`	`/blog` and everything under it, such as `/blog/2024/launch`
`/blog/*`	Pages one level under `/blog`, such as `/blog/launch`, but not `/blog/2024/launch`
`/**/print`	A `print` page at any depth, such as `/docs/setup/print`
`/de`	The German section of a site that uses language folders

Website sources also have Only pages under: when it lists paths, only those sections are read. For example, /help and /pricing keep a source to the help center and the pricing page. Leave it empty to read everything under the start address.

Sections worth skipping

Text

/tag
/category
/author
/page
/search
/cart
/checkout
/account
/**/print

Adjust the list to your site's addresses. Also consider sections whose content is out of date but has to stay online, such as old press releases.

Remove a page

Remove deletes a page from your knowledge now, without changing what ASDK reads. If the page is still on your site, it comes back with the next sync. This is useful when a page was deleted from your site and you don't want to wait for the next sync, or to test how answers change without it.

Block the crawler with robots.txt

To keep ASDK out of a section for good, whatever sources are set up, add a rule for its crawler to your robots.txt:

Text

User-agent: ASDK-Crawler
Disallow: /internal/

This affects only ASDK, not search engines. It applies to every ASDK source of the site, and pages already in your knowledge are removed after the next complete sync. The crawler page explains the rules. Don't block your whole site: the crawler then can't build your knowledge at all.

What excluding doesn't do

It doesn't hide the page from visitors or search engines. It only keeps it out of ASDK's knowledge.
It doesn't stop questions about the topic. If a visitor asks about something you excluded, the agent answers from what remains, or says it doesn't know. For a topic you'd rather handle in person, add a Q&A pair that points to your contact page.