Use case
Turn a website into a knowledge base.
Point kbDrop at a public site and get a private, queryable knowledge base with cited answers—robots.txt respected, JavaScript rendering optional, recrawl on demand.
How it works
From URL to cited answers
Paste the site URL
Give kbDrop the starting page of a public website. Turn on JavaScript rendering if the site builds its content in the browser.
kbDropBot crawls within limits
The crawler walks reachable same-site pages under strict page, depth, time, and byte limits, honoring robots.txt, sitemaps, and crawl delays as it goes.
Query the site like a document
The crawled pages become a private knowledge base. Ask questions in chat or over the API and get answers that cite the exact pages.
Where this pays off
Sites worth querying instead of browsing
The obvious candidate is your own documentation or marketing site: crawl it once and support, sales, and new teammates can ask it questions instead of hunting through navigation. It works just as well on public material you rely on but do not control—a vendor's docs, a standards body's guidance, a partner's help center—kept as your own private, queryable copy.
Because the crawled site becomes a normal kbDrop knowledge base, everything else applies: answers cite the source pages, access stays private to your account, and the same knowledge base is reachable over the streaming API.
Built to be a good citizen
kbDropBot/1.0 (+https://kbdrop.io/crawler)is recognizable in server logs, and publishers can disallow it in robots.txt.- Robots rules, sitemaps, crawl delays, and HTTP retry guidance are honored; requests are spaced per origin.
- Every navigation, redirect, and subresource is screened and restricted to public HTTP(S) destinations.
- Hard page, depth, time, and byte limits bound every crawl.
Questions, answered
Website crawling FAQ
Can kbDrop crawl any website?
kbDrop crawls reviewed public HTTP or HTTPS destinations only. It follows RFC 9309 robots.txt rules, declared sitemaps, and crawl delays, and it does not bypass authentication, paywalls, CAPTCHAs, or anti-bot controls. If a publisher disallows kbDropBot, the site is not crawled.
Does the crawler execute JavaScript?
Optionally. Enable JavaScript rendering when a site builds its content in the browser, and kbDrop renders pages in an isolated browser where every navigation, redirect, and subresource is independently restricted to screened public destinations.
What happens when the website changes?
Trigger a recrawl and kbDrop fetches the site again to refresh the knowledge base, subject to the same robots.txt rules and crawl limits as the original crawl.
Can I turn my docs site into a chatbot?
Yes. Crawl the docs site into a knowledge base, then use the streaming API with a key scoped to that knowledge base to serve cited answers inside your own product or support surface.
Will the crawler overload the site?
No. kbDropBot identifies itself with a recognizable user agent, spaces requests per origin, respects crawl-delay and HTTP retry guidance, and stops at hard page, depth, time, and byte limits.
Query a site you rely on
Paste a URL, let kbDropBot do the reading, and ask your first question with sources attached.
Create a free account