# Management API The management API lets an authorized agent or automation create a knowledge base, add its source, and wait until it answers questions, over plain HTTPS/JSON. Querying a ready knowledge base uses the existing [messages and search endpoints](https://kbdrop.io/docs/api/reference.md); their contracts are unchanged. | Capability | Status | | --- | --- | | Create from a website, file, archive, or video URL; poll status; recrawl; retry | Live: `/v1/knowledge-bases`, `/v1/uploads`, `/v1/ingestion-jobs` | | Management keys (`kb_mgmt_…`) | Live: create them at [kbdrop.io/management-keys](https://kbdrop.io/management-keys) | | OAuth scopes `knowledge_bases:read` and `knowledge_bases:write` | Live for the `kb-drop-cli` client, only when requested and approved | | `kb-drop knowledge-bases` CLI commands (create, status, list, retry, recrawl) | Live in `@kbdrop/cli` 0.2.0 and later | | Hosted MCP tools for creating knowledge bases | Planned | | Deleting knowledge bases, scheduled recrawls, managing keys over the API | Not in this API; use the web app | Every example uses `https://kbdrop.io`. Keep the credential in a protected `management.headers` file containing one line, `Authorization: Bearer kb_mgmt_...`, so it never appears in a command, prompt, or log. ## Credentials A **management key** belongs to your account, not to one knowledge base. Create it on the Management keys page (account menu → Developers → Management keys). The secret is shown once and is stored only as a hash. **Rotate** there shows a new secret with the same permissions and retires the old one immediately; to rotate without downtime, create a second key, switch your secret manager to it, then revoke the old one. Each key carries only the permissions you tick: | Scope | Allows | | --- | --- | | `knowledge_bases:read` | List knowledge bases; read knowledge base, ingestion job, and upload status | | `knowledge_bases:write` | Create knowledge bases, upload, crawl again, retry, cancel an upload. Uses your plan's ingestion quota | | `knowledge:query` | `POST /v1/knowledge-bases/{id}/messages` and `/search` on any knowledge base in the account | | `knowledge:read` | Read indexed source metadata and text for cited sources | An account can hold ten active management keys. Each key has its own rolling one-minute rate limit; responses carry `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`, and a limited request returns `429` with `Retry-After`. **OAuth** access tokens issued to `kb-drop-cli` work too, but only when the client requested `knowledge_bases:read` or `knowledge_bases:write` and the account owner approved it on the consent screen. Signing in with the default scopes never grants management access. **API keys** (`kb_live_…`) cannot manage knowledge bases. The management endpoints reject them with `403 management_credential_required` without looking them up. ## From the command line The `kb-drop` CLI in `@kbdrop/cli` 0.2.0 wraps this API. It uploads parts in parallel, resumes an interrupted run when the same command runs again, and waits for ingestion: ```sh export KB_DROP_CREATE_ID="$(node -p 'crypto.randomUUID()')" kb-drop knowledge-bases create --idempotency-key "$KB_DROP_CREATE_ID" \ --zip ./product-docs.zip --name "Product docs" --wait kb-drop knowledge-bases create --url https://docs.example.com --name "Public docs" \ --mode site --max-pages 100 --wait kb-drop knowledge-bases status --watch --json ``` Keep the saved key: a rerun with it resumes the same knowledge base at any later time, or fails with `upload_unavailable` if the upload has since expired, failed, or been cancelled. Without `--idempotency-key`, the CLI keeps its own key only for a command that was interrupted or stopped waiting: running it again within 24 hours resumes, unless the interrupted upload has expired, failed, or been cancelled, which starts a new knowledge base. A command that finished forgets its key, including a run without `--wait`, so running it again creates another knowledge base. It reads a management key from `KB_DROP_MANAGEMENT_KEY`, or uses a login saved by `kb-drop auth login --manage`. With `--json`, stdout carries one result object and the exit code says what happened: `0` done, `9` ingestion failed, `10` stopped waiting (run the same command again to continue, or answer the upload's review), `3` credential or permission, `5` or `6` temporary. The [CLI README](https://github.com/CorbinCald/kb-drop-cli#readme) lists every option. `@kbdrop/cli` 0.3.0 adds the `knowledge-bases review` command and the app-export selection flags. With 0.2.0, `--wait`/`--watch` can keep polling a paused review until the timeout: inspect `knowledge-bases status --json` without `--watch`, then answer `next_action: "review"` in the browser at `links.review_page` or through the [HTTP review endpoint](#review-app-exports). ## Conventions - Send `Content-Type: application/json` with every body; anything else returns `415`. - IDs are UUIDs and never change. Responses include absolute `links`, so a client never builds a URL itself. - `201` means a knowledge base was created, with `Location` pointing at it. `202` means work was queued, with `Location` pointing at the ingestion job and `Retry-After` set to the first poll interval. The exception is a `202` from creating a knowledge base, which means another request with the same key is still opening the upload: it carries only `Retry-After`. A repeated request returns `200` and `X-Idempotent-Replay: true`. - Responses use `Cache-Control: no-store`, `Vary: Authorization`, and an `X-Request-ID` to quote to support. - Every error has one shape. `retryable: true` means repeating the same request later can succeed; honor `Retry-After` when present. ```json { "error": { "code": "active_job_quota_exceeded", "message": "You already have 2 uploads being processed. Wait for one to finish, or cancel it, then try again.", "retryable": true, "recovery": "wait" } } ``` ### Idempotency Creating a knowledge base and starting a crawl require an `idempotency_key` (a UUID you generate). Keep it until the request succeeds: retrying with the same key and payload returns the same knowledge base, job, and upload, even after the knowledge base is renamed, and never reserves quota twice. The same key with a different payload returns `409 idempotency_mismatch`; generate a new key for a new knowledge base. Completing an upload and retrying a job are idempotent by design and need no key. ## Create from a website ```bash curl --fail-with-body https://kbdrop.io/v1/knowledge-bases \ --header @./management.headers --header 'Content-Type: application/json' \ --data '{"idempotency_key":"6f1d2c1e-8a8b-4d55-9d8e-0d1f2b3c4d5e", "source":{"type":"web","url":"https://docs.example.com/guide/"}}' ``` Only `type` and `url` are required. The remaining fields match the web app's crawl settings and defaults: | Field | Default | Values | | --- | --- | --- | | `mode` | `site` | `site` follows links; `single_url` indexes one page | | `include_paths`, `exclude_paths` | `[]` | Up to 20 glob path patterns each, such as `/docs/*` | | `query_policy` | `drop_tracking` | `drop_tracking`, `preserve`, or `strip` | | `include_subdomains` | `false` | Follow links to subdomains | | `allow_documents` | `false` | Also index linked PDF files | | `render_mode` | `auto` | `auto`, `always`, or `never` render JavaScript | | `max_pages` | `100` | 1 to the account's plan limit, at most 10,000 | | `max_depth` | `3` | 0–10 links from the start URL | An optional top-level `name` (1–120 characters) overrides the default name. The crawler honors robots.txt, stays on the start URL's site, and refuses private, local, and metadata-network addresses. Unsafe or malformed start URLs fail immediately with `400` (`invalid_web_url`, `private_web_url`, `url_credentials_not_allowed`, `url_port_not_allowed`, `web_url_dns_failed`, or `crawl_trap_url`) and create nothing. A `max_pages` above the account's plan limit fails with `403 crawl_page_limit_exceeded`, whose message names the limit, and creates nothing. The `201` response contains `knowledge_base`, `ingestion_job`, and `upload: null`. ## Create from a video URL ```json { "idempotency_key": "…", "source": { "type": "video_url", "url": "https://www.youtube.com/watch?v=…" } } ``` The URL is a public YouTube video or a direct HTTPS link to an MP4 file. Invalid links fail with `400 invalid_video_url` or `video_url_https_required`. ## Create from a file or archive Files go straight to object storage in 16 MiB parts; kbDrop never proxies the bytes. Upload a ZIP, tar, tar.gz/bz2/xz, or 7z archive for a code or document collection, or a single document, image, audio, or video file. A single file is limited to 1 GiB, and some formats, such as images, have smaller limits of their own. An audio or video recording can be as large as its own limit, 2 GiB, within the largest archive the account's plan allows. An archive can be as large as the plan allows, at most 10 GiB; a larger upload fails with `413 upload_too_large`, whose message names the limit. 1. `POST /v1/knowledge-bases` with an upload source. The response's `upload` lists `part_size_bytes`, `part_count`, and `missing_part_numbers`. A `202` instead of `201` means another request with the same key is still opening the upload; repeat the request after `Retry-After`. 2. `POST {upload.links.part_urls}` with `{}` to sign the next missing parts (up to eight), or `{"part_numbers":[3,4]}` for specific ones. Each signed URL expires 15 minutes after signing. 3. `PUT` exactly `size_bytes` bytes from the part's offset, `(part_number - 1) * part_size_bytes`, to each `url`. Keep the `ETag` response header. 4. `POST {upload.links.parts}` with `{"parts":[{"part_number":1,"etag":"\"…\"","size_bytes":16777216}]}` after each batch. Confirmed parts survive a crashed client. 5. `POST {upload.links.complete}` with `{}`. kbDrop verifies the stored size and queues ingestion, returning `202` and the job. ```json { "idempotency_key": "…", "name": "Handbook", "source": { "type": "upload", "filename": "handbook.pdf", "size_bytes": 482133, "content_type": "application/pdf" } } ``` `content_type` is optional and must match the filename's extension when sent. An optional `fingerprint` (`sha256-tree-v1:<64 hex>`) is stored with the upload and must match on replay. An optional `app_data` answers an app export's review up front, as described under Review app exports. To **resume** after a crash, repeat step 1 with the same idempotency key and body (this needs only `knowledge_bases:write`), or `GET {upload.links.self}`: `missing_part_numbers` lists what is still needed. An upload with no newly confirmed part for `idle_timeout_seconds` (30 minutes by default) can be cancelled to free your quota when you start another upload, so confirm parts as you go. If completion answers `409 upload_parts_rejected`, storage discarded a part: the rejected parts reappear in `missing_part_numbers`; upload them again and complete. `DELETE {upload.links.self}` cancels an unfinished upload and releases everything it reserved. This Node.js 24 script runs the whole flow, resumes on rerun with the same `IDEMPOTENCY_KEY`, and waits for the result: ```js // KB_DROP_MANAGEMENT_KEY=kb_mgmt_… IDEMPOTENCY_KEY=$(uuidgen) node upload.mjs ./docs.zip import { openAsBlob } from "node:fs"; import { basename } from "node:path"; const path = process.argv[2]; const file = await openAsBlob(path); const headers = { Authorization: `Bearer ${process.env.KB_DROP_MANAGEMENT_KEY}`, "Content-Type": "application/json", }; const sleep = (seconds) => new Promise((done) => setTimeout(done, seconds * 1_000)); async function call(url, body) { for (let attempt = 1; ; attempt += 1) { const response = await fetch(url, { method: body ? "POST" : "GET", headers, body: body && JSON.stringify(body), }); const json = await response.json(); if (response.ok) return json; if (!json.error.retryable || attempt === 5) { throw new Error(`${json.error.code}: ${json.error.message}`); } await sleep(Number(response.headers.get("retry-after") ?? 5)); } } const create = { idempotency_key: process.env.IDEMPOTENCY_KEY, source: { type: "upload", filename: basename(path), size_bytes: file.size }, }; let { upload } = await call("https://kbdrop.io/v1/knowledge-bases", create); for (let tries = 0; ["pending", "initiating"].includes(upload.status) && tries < 30; tries += 1) { await sleep(1); // Another request with this key is still opening the upload. ({ upload } = await call("https://kbdrop.io/v1/knowledge-bases", create)); } while (upload.status === "uploading" && upload.missing_part_numbers.length) { const { part_urls } = await call(upload.links.part_urls, {}); const parts = []; for (const part of part_urls) { const start = (part.part_number - 1) * upload.part_size_bytes; const put = await fetch(part.url, { method: "PUT", body: file.slice(start, start + part.size_bytes), }); if (!put.ok) throw new Error(`Part ${part.part_number} failed with ${put.status}.`); parts.push({ part_number: part.part_number, etag: put.headers.get("etag"), size_bytes: part.size_bytes, }); } upload = await call(upload.links.parts, { parts }); } let { ingestion_job: job } = await call(upload.links.complete, {}); // Poll while the job asks you to wait; a finished job or a review stops it. for (let polls = 0; job.next_action === "wait" && polls < 720; polls += 1) { await sleep(job.poll_after_seconds ?? 5); job = await call(job.links.self); } console.log(JSON.stringify({ status: job.status, queryable: job.queryable, next: job.next_action })); ``` ## Wait for ingestion `GET /v1/ingestion-jobs/{id}` (the `Location` from a `202`) is the one resource to poll. Wait `poll_after_seconds` between requests and stop when `terminal` is `true` or `poll_after_seconds` is `null`. A web crawl usually takes minutes; a large archive can take longer. ```text uploading → queued → [crawling] → extracting → parsing → embedding → ready ↘ failed ── POST retry ──→ queued ↘ cancelled extracting → reviewing ── POST review ──→ queued or cancelled crawling → paused ── POST retry ──→ crawling ``` | Field | Meaning | | --- | --- | | `status` | One of the states above. `ready`, `failed`, and `cancelled` are terminal | | `next_action` | `upload_parts`, `complete_upload`, `wait`, `retry`, `review`, `create_new_knowledge_base`, `contact_support`, or `none` | | `poll_after_seconds` | Seconds before the next poll, or `null` when polling cannot change anything | | `progress.files` | `discovered`, `processed`, `skipped`, and `failed` source files | | `progress.upload` | `parts_total`, `parts_confirmed`, `bytes_total`, `bytes_confirmed`, or `null` | | `progress.crawl` | `pages_discovered`, `pages_fetched`, `pages_indexed`, `pages_skipped`, `pages_failed`, or `null` | | `failure` | `null`, or `stage`, `recovery`, `retryable`, and a safe `message` | | `review` | `null`, or what a `reviewing` upload waits on (below) | | `version`, `active_version` | The version this job builds and the version answering queries now | | `queryable` | `true` when search and messages can use the knowledge base | A replacement version never takes a working knowledge base offline: while a recrawl or retry runs, and after one fails, `active_version` keeps answering and `queryable` stays `true`. A crawl that runs out of indexing allowance saves its progress with status `paused`, `next_action: "retry"`, and `poll_after_seconds: null`. The `failure` message explains whether allowance or unresolved charges are holding it up. When allowance is available, send `POST {links.retry}` with `{"attempt": , "resume_key": ""}` to resume the same attempt, frontier, and reservation. Save this key before sending and reuse it after a lost response. Resume returns `202`; insufficient allowance returns `429` and leaves progress saved. It does not charge another ingestion. A paused replacement keeps the previous version queryable. `GET /v1/knowledge-bases/{id}` returns the aggregate: `status`, `queryable`, `source`, `active_version`, `latest_job`, and `links` to `search`, `messages`, and (for websites) `crawls`. `GET /v1/knowledge-bases?limit=20` lists newest first; pass `next_cursor` as `cursor` while `has_more` is `true`. ## Review app exports An upload holding app exports, such as Slack, Microsoft Teams, Datadog, or Grafana, stops before anything is embedded when an export holds private conversations that are left out by default, needs a Datadog site or Grafana address to link records back, or is too large to read whole. The job is `reviewing` with `next_action` `review` and `poll_after_seconds` `null`: it keeps its slot and waits for an answer. `review` holds counts and display names only, never content: | Field | Meaning | | --- | --- | | `exports[].key` | The export an answer names | | `exports[].reasons` | `private`, `link_base`, and `too_large`, as they apply | | `exports[].private` | Conversations left out, as `kinds` (`private_channel`, `direct_message`, …) with counts, and the `records` an opt-in adds; `available` is `false` when they can't be read from this upload | | `exports[].partitions` | Channels, projects, or spaces: `id`, `label`, and `records`, or `bytes` when too large | | `exports[].too_large` | `limit_bytes`, what one upload reads of an export | | `other_files` | Files outside these exports, indexed either way | `POST {links.review}` answers once. `index` returns `202` and the queued job; `{"decision":"cancel"}` ends it and releases what it held. ```json { "decision": "index", "exports": { "": { "include_private": true, "exclude_partitions": [""], "since": "2026-01-01" } }, "link_bases": { "datadog": "app.datadoghq.eu", "grafana": "https://grafana.example.com" } } ``` An export left out of `exports` keeps its defaults, so `{"decision":"index"}` indexes without private content. `since` and `until` are inclusive UTC days; undated records are kept. A Datadog site must be on Datadog's own hosts, and a Grafana address must be HTTPS. An export still too large for its limit is left out whole, and an answer that would index nothing is refused, as is `include_private` for an export whose private content isn't `available`. A job no longer waiting answers `409 review_not_pending`: read it to see which answer took effect. `links.review_page` opens the same review in the browser. To never pause, create the upload with `source.app_data`, which applies to every export in it: `{"include_private":true,"since":"2026-01-01", "until":"2026-06-30","link_bases":{"datadog":"app.datadoghq.com"}}`, or `{}` for the defaults. ## Crawl again and retry - `POST /v1/knowledge-bases/{id}/crawls` with `{"idempotency_key":"…"}` crawls a website knowledge base again with its saved settings, as a new version. A crawl already running returns `409 crawl_already_active`; other source types return `409 knowledge_base_not_crawlable`. - `POST /v1/ingestion-jobs/{id}/retry` with `{"attempt":1}` retries a job whose `next_action` is `retry`. To resume a paused crawl, also send a new UUID in `resume_key`. `attempt` is the attempt you observed, so repeating the request returns the requeued job with `X-Idempotent-Replay: true` instead of retrying twice. A repeated resume key returns the current job without changing it, even after completion, failure, or a later pause. Use a fresh key to resume a later pause. A resume without a key returns `resume_key_required`; a new resume key for a job that is no longer paused returns `ingestion_not_paused`. Neither starts another attempt. - A crawl or retry runs within the account's plan as it is when it starts. An upload larger than that plan takes is not retried and returns `413 upload_too_large`. ## Errors and recovery `recovery` tells an agent what to do without parsing the message: | `recovery` | Do this | Examples | | --- | --- | --- | | `wait` | Repeat after `Retry-After` | `active_job_quota_exceeded`, `crawl_already_active`, `upload_initializing`, `ingestion_in_progress` | | `retry_later` | Repeat after `Retry-After`; back off if it persists | `management_key_rate_limited`, `provider_spend_quota_exceeded`, `management_api_disabled`, `internal_error` | | `fix_request` | Correct the request; the message says what to change | `invalid_request`, `invalid_json`, `request_too_large` (bodies over 256 KiB), `private_web_url`, `unsupported_file_type`, `crawl_page_limit_exceeded` (lower `max_pages`), `upload_completed` (poll the ingestion job instead), `invalid_review_choices`, `review_not_pending` | | `use_new_idempotency_key` | Generate a new key for a different request | `idempotency_mismatch` | | `upload_missing_parts` | Upload `missing_part_numbers`, then complete | `parts_incomplete`, `upload_parts_rejected` | | `create_new_knowledge_base` | Start over with a new idempotency key | `upload_closed`, `file_size_mismatch`, `ingestion_not_retryable` | | `authenticate` | Supply a valid, unrevoked credential | `invalid_management_credential`, `invalid_management_key` | | `use_management_credential` | Use a management key or approved OAuth token | `management_credential_required` | | `request_scope` | Ask the account owner for a key with the missing scope | `management_key_scope_insufficient`, `oauth_scope_insufficient` | | `ask_account_owner` | The account needs more quota or a plan change | `storage_quota_exceeded`, `billing_quota_exceeded`, `payment_required`, `crawl_allowance_pending` | | `check_identifier` | The ID is wrong or belongs to another account | `knowledge_base_not_found`, `ingestion_job_not_found`, `upload_not_found` | | `contact_support` | Stop and report the `X-Request-ID` | `account_suspended`, `ingestion_needs_support` | Resources in other accounts are indistinguishable from missing ones. Error messages and logs never include credentials, signed URLs, local paths, or source content.