Configuration Reference

Everything in Pathfinder is driven by a single pathfinder.yaml file. This is the full annotated reference.

server

Top-level server identity and session management.

• name — Identifies the server in MCP tool descriptions. Required.
• version — Semantic version string. Required.
• max_sessions — Global cap on total concurrent sessions across all IPs. When exceeded, new connections receive a 503 with a descriptive JSON body (error: "capacity_exceeded", reason: "server-capacity", totalSessions, maxSessions, retryAfterSeconds, contact) and a Retry-After header. The session reaper logs a warning while sessions are at or above 80% of this cap. Optional, defaults to 1000.
• max_sessions_per_ip — Per-IP rate limiting. Prevents a single IP from opening too many concurrent sessions. Optional, defaults to 20.
• session_ttl_minutes — Idle timeout for active sessions (sessions that have invoked at least one tool). Sessions with no activity for this many minutes are cleaned up. Optional, defaults to 30.
• session_unused_ttl_minutes — Idle timeout for unused sessions (sessions that connected but never invoked a tool). Designed to shed idle MCP connections quickly while keeping active sessions alive longer. Set to 15 by default to match Railway's 15-minute SSE connection hard limit. Optional.
• allowlist — Array of IPv4/IPv6 addresses or CIDR ranges that bypass max_sessions_per_ip, the per-IP concurrent body-read cap (see below) and the modern per-IP rate limit (modern_rpm_per_ip and modern_burst_per_ip), but not modern_max_inflight. Use for trusted crawlers (e.g. 160.79.106.35 — Anthropic Assistant) or internal health-probe ranges. Optional, defaults to empty. Entries are matched against the per-request client IP — behind a reverse proxy this requires a trust_proxy value other than false or 0, preferably a hop count or a CIDR list (see trust_proxy below); with the default trust_proxy: false the server only sees the proxy's TCP socket address and no upstream client IP will ever match. Directly-exposed servers use the socket peer and need no extra config. Clients that hit the max_sessions_per_ip limit receive a 429 with a Retry-After header and the fields error, reason, limit, currentCount, retryAfterSeconds and contact: on GET /sse they are the top-level fields of the JSON body, and on /mcp the body is a JSON-RPC error (code -32005) that carries them in error.data.
• modern_rpm_per_ip — Steady request rate, in requests per minute, allowed per client IP on 2026-07-28 (stateless) connections to /mcp. These connections hold no session, so max_sessions_per_ip does not apply to them. Entries in allowlist bypass this limit. A request over the limit gets HTTP 429 with a JSON-RPC error (code -32005, the request id echoed, data.retryAfterSeconds) and a Retry-After header with the same number of seconds. A request whose MCP-Protocol-Version header names the 2026-07-28 revision, and whose body is not JSON-parsed (a non-JSON or missing Content-Type), is checked before its body is read: from an IP with no requests left, it gets the same 429 with a null id, and its body is discarded unread. Optional, defaults to 120.
• modern_burst_per_ip — Burst size (token-bucket capacity) per client IP for 2026-07-28 connections. Entries in allowlist bypass this limit. Over the limit, the reply is the same 429 JSON-RPC error (-32005) with Retry-After, as for modern_rpm_per_ip. Optional, defaults to 60.
• modern_max_inflight — Ceiling on 2026-07-28 requests being handled at the same moment, across all IPs. The allowlist does not bypass this ceiling. Open subscriptions/listen streams do not count against it. A request over the ceiling gets HTTP 503 with a JSON-RPC error (code -32006, the request id echoed, data.retryAfterSeconds of 1) and Retry-After: 1. Optional, defaults to 200.
• Concurrent body-read cap (fixed at 4, not a YAML key) — Applies only when PATHFINDER_MODERN_PROTOCOL is on. To tell a 2026-07-28 request from a 2025-era one, the server reads a POST /mcp body that is not JSON-parsed (a non-JSON or missing Content-Type), up to 4 MiB. One client IP may have at most 4 of these reads in progress at once, on either era. One more gets HTTP 429 with a JSON-RPC error (code -32005, a null id, data.retryAfterSeconds of 1) and Retry-After: 1, and its body is discarded unread. A request with Content-Type: application/json never counts. Entries in allowlist bypass this cap.
• modern_request_timeout_ms — Wall-clock deadline for one 2026-07-28 request, in milliseconds. When it passes, the request releases its modern_max_inflight slot, a warn line is logged, and a running bash command is aborted. The bash command stops at the start of its next statement (a sleep wakes at once), so the rest of a running && or || list can still run. Its result keeps the output so far, adds a deadline: line to stderr and exits 124. Open subscriptions/listen streams are not subject to the deadline. At most 2147483647; a larger value fails config load. Optional, defaults to 60000 (the MCP SDK's default client request timeout).
• PATHFINDER_MODERN_PROTOCOL (environment variable, not a YAML key) — Turns on 2026-07-28 stateless serving on /mcp. Set it to 1 or true (case-insensitive); any other value, or unset, leaves it off. It is off by default and is read at startup. The four modern_* keys apply only when it is on.
• trust_proxy — Whether to honor the X-Forwarded-For header for client-IP resolution (used by rate limiting, allowlist checks, tracing, and analytics). The value is passed to Express's trust proxy setting and takes one of three forms: true trusts every hop, so req.ip is the leftmost entry of X-Forwarded-For; a hop count (for example 1) trusts that many proxies nearest the server; a list of proxy IPs or CIDRs (for example ["10.0.0.0/8"]) trusts only hops inside those ranges. false and 0 ignore the header. An empty list is not accepted: it fails config load. Honoring the header is REQUIRED when the server runs behind a reverse proxy (Railway, Fly, Nginx, etc.) that sets X-Forwarded-For. When you know the ingress topology, prefer the hop count or the CIDR list to true: they stop a client-supplied entry from being taken as the client IP. Optional, defaults to false. ⚠️ Security: only use true when the proxy discards any client-supplied X-Forwarded-For AND sets its own trusted value. If the proxy passes through client-supplied X-Forwarded-For, attackers can send X-Forwarded-For: 160.79.106.35 to be seen as an allowlisted IP and bypass the session limiter and the modern per-IP rate limiter entirely. When false (default), X-Forwarded-For is ignored and the server uses the TCP socket's peer address.

sources

Define where your content lives. Each source becomes a virtual filesystem that agents can explore.

• unindexed_tolerance — After each reindex, an audit compares the files walked on disk against the files present in the index and reports a count_divergence finding (direction db_has_fewer) naming the files the index is missing. Some shortfall is normal — a file with no extractable prose is walked and then dropped — so the finding only fires above this share of the source's walked files (default 0.05, with a floor of 3 files so small sources stay quiet). Set it to 0 once a source is known to index everything it walks, and the audit will flag the very first regression. It does not affect what gets indexed, so changing it does not trigger a reindex.
• unclaimed_exempt_paths — The shortfall check above can only compare the index against the files the configured walk root reaches, so a walk root pointed at the wrong subtree leaves both halves blind in the same way and they agree perfectly. A separate audit therefore walks the whole repository and reports an unclaimed_content finding for any directory tree of at least 10 files sharing one extension that no source's path and file_patterns would ever reach — when that extension is one the repository already indexes through the tree's parent. Files a source's own exclude_patterns reject are treated as accounted for, since an exclusion is a decision on record. List here the repo-root-relative directories (prefixes or globs) an operator has reviewed and confirmed are correctly unclaimed — partials inlined at render time, example apps, maintenance scripts — and the audit stops reporting them. Entries are pooled across every source reading the same repository. It does not affect what gets indexed, so changing it does not trigger a reindex.
• type — Determines chunking strategy. markdown splits on headings and uses token-based chunks. code splits on function boundaries and uses line-based chunks. raw-text treats content as plain text with token-based chunks. html parses HTML structure and extracts text content with token-based chunks. document extracts text from PDF and DOCX files with token-based chunks.
• category — Optional tag applied to all chunks from this source. Sources with category: faq contribute to the /faq.txt endpoint and can be queried via knowledge tools.
• repo — If provided, Pathfinder clones and updates from this repo. Omit for local directories.
• version — Tags all indexed chunks with this version string. Used for version-filtered search queries.
• base_url — Base URL prepended to the derived slug when generating document links. Works together with url_derivation.
• url_derivation — Controls how file paths are transformed into URL slugs that get appended to base_url. Each step is applied in order:
  • strip_prefix — Remove this prefix from the file path before generating the URL. For example, "docs/" turns docs/guide/auth.mdx into guide/auth.mdx. It also accepts a list of candidate prefixes, of which the first one that matches the path is stripped — useful when one source spans sibling content trees that sit at different URL depths. Order matters: list a longer prefix before any shorter prefix that would also match it. For example ["content/docs/", "content/"] turns content/docs/guide.mdx into guide.mdx while turning content/reference/useAgent.mdx into reference/useAgent.mdx.
  • strip_suffix — Remove the file extension. For example, ".mdx" turns guide/auth.mdx into guide/auth.
  • strip_route_groups — When true, removes Next.js-style route group segments like (marketing) or (guides) from the path.
  • strip_index — When true, removes a trailing /index from the path (e.g. guide/auth/index becomes guide/auth).
• chunk — Use target_tokens/overlap_tokens for markdown and raw-text sources. Use target_lines/overlap_lines for code sources.

Note: The chunk config is available on all source types, but the effect varies. Markdown, raw-text, HTML, and document sources use token-based chunks (target_tokens/overlap_tokens). Code sources use line-based chunks (target_lines/overlap_lines). For Slack and Discord, the Q&A chunker produces one chunk per Q&A pair and chunk settings have no effect. For Notion, the markdown chunker respects target_tokens and overlap_tokens.

Source type: markdown

Splits content on Markdown headings and uses token-based chunks. Best for documentation written in .md or .mdx files. Configure with target_tokens and overlap_tokens.

Source type: code

Splits on function/class boundaries and uses line-based chunks. Best for source code. Configure with target_lines and overlap_lines.

Source type: raw-text

Treats content as unstructured plain text with token-based chunks. Use when content has no Markdown or code structure.

Source type: html

Parses HTML structure, extracts text content, and uses token-based chunks. Best for static sites, rendered documentation, or any HTML content.

Source type: slack

Indexes Slack threads as Q&A pairs. An LLM distills each qualifying thread into a question-answer pair with a confidence score. All pairs are stored; confidence filtering happens at query time.

• channels — Array of Slack channel IDs to monitor. The bot must be a member of each channel.
• confidence_threshold — Minimum confidence score (0-1) for Q&A pairs to appear in /faq.txt. Pairs below this threshold are still stored. The knowledge tool does not read this value; it uses its own min_confidence. Optional, defaults to 0.7.
• trigger_emoji — When a user reacts with this emoji on a thread, it triggers immediate reindexing of that thread. Useful for curating high-quality answers. Give the emoji name without colons. Optional, defaults to pathfinder.
• min_thread_replies — Threads with fewer replies are skipped during indexing. Optional, defaults to 2.
• distiller_model — The OpenAI model used to distill threads into Q&A pairs. Defaults to gpt-4o-mini.

Required environment variables: SLACK_BOT_TOKEN, SLACK_SIGNING_SECRET (for emoji-triggered webhook verification), and OPENAI_API_KEY (for thread distillation, whatever the embedding provider).

Source type: discord

Indexes Discord channels as Q&A pairs. Supports two channel types with different extraction strategies.

• guild_id — The Discord server ID.
• channels — Array of channel objects, each with an id and type.
• type: text — Text channels use LLM distillation (same as Slack). Threads are distilled into Q&A pairs with confidence scores.
• type: forum — Forum channels extract Q&A directly from the post title (question) and replies (answer). Confidence is always 1.0 since the structure is explicit.
• confidence_threshold — Minimum confidence for pairs listed in /faq.txt. Forum posts always pass (confidence 1.0). The knowledge tool does not read this value. Optional, defaults to 0.7.
• min_thread_replies — Threads and forum posts with fewer messages are skipped. Applies to both text-channel threads and forum posts. Optional, defaults to 2.
• distiller_model — OpenAI model for thread distillation on text channels. Optional, defaults to gpt-4o-mini.

Discord does not support emoji-triggered reindexing (requires Gateway WebSocket, which Pathfinder does not maintain).

Required environment variables: DISCORD_BOT_TOKEN, DISCORD_PUBLIC_KEY (for webhook Ed25519 verification), and OPENAI_API_KEY when any channel has type: text (for thread distillation, whatever the embedding provider). The bot needs the MESSAGE_CONTENT privileged intent and Read Message History + View Channels permissions.

Source type: notion

Index Notion pages and database entries as searchable markdown documents. Blocks are recursively converted to markdown, and database entry properties are serialized as YAML frontmatter.

• root_pages — Array of Notion page IDs. Each page and its children (up to max_depth) are indexed. Optional, defaults to [].
• databases — Array of Notion database IDs. All entries in each database are indexed with their properties. Optional, defaults to [].
• max_depth — Maximum depth for recursive child page discovery. Range 1-20, default 5.
• include_properties — When true, database entry properties (Status, Priority, Tags, etc.) are prepended as YAML frontmatter to the page content. Default true.

If both root_pages and databases are empty, all pages accessible to the integration token are indexed. Requires NOTION_TOKEN environment variable.

By default, Notion sources are indexed as documents and referenced by search tools. To also make them available via knowledge tools and the /faq.txt endpoint, add category: faq to the source config — useful for Notion databases structured as Q&A or FAQ collections.

Source type: document

Index PDF and DOCX files. Requires optional peer dependencies: npm install pdf-parse for PDF support and npm install mammoth for DOCX support. Uses token-based chunks like markdown sources.

• PDF support — Requires pdf-parse peer dependency. Text is extracted with pdf-parse. A PDF that yields almost no text (fewer than 50 characters, likely a scanned document without a text layer) logs a warning and is still indexed with whatever text it produced.
• DOCX support — Requires mammoth peer dependency. Document content is converted to plain text for chunking.
• max_file_size — Defaults to 10MB (10485760 bytes) for document sources. Large files are skipped with a warning.

Source type: atlas

Indexes the Atlas content that is already stored in Pathfinder's database (the atlas_seed_entries and atlas_cache_pages tables), not files. An atlas source takes no path or file_patterns. Its optional repositories list limits indexing to the listed repositories. Each entry has a repo_url and optional refs and subsystems lists. The optional seed_path and cache_namespace keys are strings.

url_derivation example

Given a file at docs/(guides)/getting-started/index.mdx with the following config:

The derivation steps produce:

tools

Tools are what agents actually call. Four types: search, bash, collect, and knowledge.

search

Semantic search over embedded content. Requires a database and embedding config.

Search Modes

The search_mode field controls how search queries are executed:

• vector (default) — cosine similarity on embeddings. Best for semantic/conceptual queries.
• keyword — PostgreSQL full-text search (tsvector/tsquery). Best for exact terms, error codes, and technical identifiers. Does not require an embedding call at query time.
• hybrid — runs both vector and keyword searches in parallel, then merges results using Reciprocal Rank Fusion (RRF, k=60). Best overall recall for mixed query types.

bash

Filesystem exploration with find, grep, cat, ls, head. Works with no database.

• grep_strategy — grep always runs as plain grep; this key does not change it. memory adds no semantic commands. vector and hybrid behave the same today: both add the qmd "<query>" semantic-search command and the related <path> command, and both require an embedding block.
• virtual_files — When true, adds the generated /INDEX.md and /SEARCH_TIPS.md files to the filesystem.
• max_file_size and cache — Accepted by the config schema but not used; they have no effect. To limit file size in the bash filesystem, set max_file_size on the source.
• workspace — When true, agents on 2025-era (session) connections can write files to /workspace/. Requires a persistent volume in production (see Deploy Guide). Apart from that, the bash filesystem is read-only for every client. Only /tmp is writable, and it is private: it lasts for the session on 2025-era connections and for one call on 2026-07-28 connections. Writes to /dev/null and /dev/zero also succeed; the data is discarded.
• session_state — When true, cd persists across tool calls within a session. This applies only to 2025-era (session) connections. On 2026-07-28 connections it has no effect: cd does not carry over to the next call.

collect

Data collection tools. Agents submit structured data based on a JSON schema you define. Submitted data is stored in the local PostgreSQL database in the collected_data table, with a tool_name column and a data JSONB column containing the fields from your schema. What you do with the collected data is up to you — query it directly, build dashboards, pipe it to analytics, or export it. Pathfinder stores it; you decide how to use it.

knowledge

FAQ and knowledge base tool. Exposes Q&A pairs from FAQ-category sources (Slack, Discord) to agents. Supports two modes: browse (no query, returns recent pairs) and search (semantic query over pairs).

• sources — Array of source names. These sources should have category: faq set.
• min_confidence — Minimum confidence for this tool's results. Q&A pairs below this score are filtered out. The source-level confidence_threshold does not apply here. An agent can pass its own min_confidence per call. Optional, defaults to 0.7.
• default_limit — Number of results returned when the agent doesn't specify a limit. Optional, defaults to 20.
• max_limit — Upper bound on results the agent can request. Optional, defaults to 100. default_limit must not exceed it.

When called without a query, the tool returns recent Q&A pairs (browse mode). When called with a query, it performs semantic search over the pairs.

embedding

Required when using search or knowledge tools, or a bash tool with grep_strategy: vector or hybrid. Configures how content is embedded for vector search.

Provider options

Three embedding providers are supported. There is no default: provider must always be set. OPENAI_API_KEY is required for the openai provider, and also for Slack sources and Discord text channels (thread distillation), whatever the provider.

OpenAI — uses the OpenAI embeddings API. Requires OPENAI_API_KEY.

Ollama — calls a local Ollama instance over HTTP. No API key needed. Requires Ollama running with the model pulled (ollama pull nomic-embed-text).

Local — runs @xenova/transformers in-process. Zero external dependencies, CPU-only. Install the optional peer dependency: npm install @xenova/transformers. Running in Docker? Use the prebuilt ghcr.io/copilotkit/pathfinder:latest-local image (also tagged :<version>-local), which ships this dependency preinstalled — no npm install needed. The default image fails loudly at startup if provider: local is set without the dependency present.

indexing

Optional. Controls automatic reindexing behavior. Every key has a default, so you can leave out the block. With the defaults, auto_reindex is false: there is no scheduled reindexing, only manual indexing.

webhook

Optional. Triggers targeted reindexing when you push to GitHub.

Point a GitHub webhook at https://your-server/webhooks/github with the push event. Set GITHUB_WEBHOOK_SECRET to match the webhook secret.

analytics

Optional. Enables query logging and the built-in analytics dashboard at /analytics.

• enabled — When true, Pathfinder logs queries and serves the analytics dashboard at /analytics. Default false.
• log_queries — Whether the query text is stored. Every tool call is logged either way, with latency, result count, client IP and User-Agent. When false, only the query text is replaced with <redacted>. Default true (when analytics is enabled).
• token — Bearer token for authenticating requests to /api/analytics/* endpoints. Prefer the ANALYTICS_TOKEN environment variable; set analytics.token only if you need the value inline (YAML is loaded as-is — no ${VAR} interpolation). When analytics is enabled and NODE_ENV is production, one of analytics.token or the ANALYTICS_TOKEN env var is required. In any other environment, a missing token is auto-generated at startup, and the log shows only a short fingerprint of it, not the token. Set ANALYTICS_TOKEN to get a token you can use.
• retention_days — How long to keep analytics data before automatic cleanup. Default 90 days.
• machine_relays — Machines that forward somebody else’s text into your tools verbatim (an issue-triage bot, a mirroring service). Their traffic is real, but it is not a user query, so matching rows are excluded from Top Queries, the empty-result panel, the unique-IP and session counts, and the scheduled reports. Each rule takes a name, an optional reason, and the predicates user_agent (exact, case-insensitive) and client_ip_cidr (IPv4 CIDR); a rule matches when every predicate it declares matches. Each rule must declare at least one of user_agent or client_ip_cidr; a rule with neither fails config load. Nothing is hidden: the dashboard lists every rule and what it removed, and ?request_source=relay on any analytics endpoint reads the excluded rows back. Prefer having the relay send X-Pathfinder-Source instead, with the value relay or one of its aliases outpost and github-triage (case-insensitive). Any other value that the server does not recognize, such as the relay's own name, is counted as real-user traffic. A 2026-07-28 client sends it on every request; a 2025-era client sends it on the request that opens the session (the /mcp initialize, or GET /sse). A self-declaring relay needs no rule here.

The analytics dashboard provides top queries, empty result tracking, and latency metrics. API endpoints: /api/analytics/summary, /api/analytics/queries, /api/analytics/empty-queries, /api/analytics/blocked-queries, /api/analytics/relay-exclusions, /api/analytics/tool-counts, /api/analytics/tool-breakdown, /api/analytics/auth-mode.

Authentication

Pathfinder runs an anonymous OAuth 2.1 flow for MCP clients automatically — no config needed beyond the MCP_JWT_SECRET environment variable documented in the Deployment Guide. No user accounts, no sign-up, no dashboard — clients that perform the handshake receive a token whose subject is always anonymous. This satisfies MCP clients that require OAuth (claude.ai, newer Claude Code builds) while keeping Pathfinder a pure knowledge server.

OAuth endpoints

All standard OAuth 2.1 + dynamic client registration endpoints are served automatically:

• /.well-known/oauth-protected-resource — resource metadata (RFC 9728).
• /.well-known/oauth-authorization-server — authorization server metadata (RFC 8414).
• /register — dynamic client registration (RFC 7591). Clients register themselves on first connect.
• /authorize — authorization endpoint. Renders a one-click approve page and redirects with a code.
• /token — token endpoint. Exchanges codes or refresh tokens for access tokens.
• /revoke — token revocation (RFC 7009).

Consent screen and abuse defense (v1.15+)

GET /authorize renders a server-rendered consent screen instead of auto-approving. The screen carries an HMAC-bound nonce; the form posts back to POST /authorize/consent, which re-verifies every bound parameter (client_id, redirect_uri, state, code_challenge, response_type, scope, resource) before issuing an auth code. The final redirect URL is built from the nonce-bound redirect_uri, never the form body — this is the phishing-resistance invariant.

The page is locked down with Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; form-action 'self'; frame-ancestors 'none'; base-uri 'none', X-Frame-Options: DENY, and Referrer-Policy: no-referrer. No JavaScript, no external resources.

POST /register applies a redirect_uri policy: https-only except for loopback (localhost, 127.0.0.0/8, [::1]); rejects 0.0.0.0/8, RFC 1918, link-local, ULA, IPv4-mapped private addresses, wildcards, userinfo, and fragments. Maximum 2048 chars per URI, 10 URIs per client.

Registered clients are capped at 10,000 total / 100 per-IP with lazy TTL eviction (30 days unused or 7 days never-used-after-registration). Overflow returns 429 (per-IP) or 503 (total) with a Retry-After header.

Bearer auth on /mcp and /sse

Both transport endpoints use opportunistic bearer authentication:

• Requests without an Authorization header pass through unchanged — useful for clients that don't speak OAuth.
• Requests with a valid bearer token attach the token claims to the session.
• Requests with an invalid or expired bearer are rejected with 401 Unauthorized and a WWW-Authenticate challenge pointing at the resource metadata endpoint.

Clients that see the 401 challenge automatically discover the OAuth server, register, and retry with a valid token. Rotating MCP_JWT_SECRET invalidates all issued tokens at once — clients re-authenticate transparently on the next request.

Example Configs

Bash-only (no database)

The simplest setup. No API keys, no database. Agents explore docs with shell commands.

Full stack with semantic search

Bash exploration plus RAG search. Requires Postgres and an OpenAI API key.

Multi-repo

Index docs from multiple repositories into a single Pathfinder instance.

Multi-version

Serve multiple versions of the same docs. Each source carries a version tag. A search tool reads one source, so each version gets its own search tool. The bash tool mounts both versions side by side.