data-formulator/DEVELOPMENT.md

Set up a local Data Formulator development environment

How to set up your local machine.

Prerequisites

• Python >= 3.11
• Node.js
• Yarn
• uv (recommended) or pip

Backend (Python)

Analyst Execution Defaults

AnalystExecutionConfig and ANALYST_EXECUTION_DEFAULTS in agent_config.py own analyst loop limits. Interactive chat and API callers share these defaults:

Setting	Default	Scope
max_actions	10	Action budget, accounting for completed steps on resume
max_tool_rounds_per_action	12	Model rounds before each committing action
empty_response_retries	2	Empty responses within each tool-loop invocation
empty_response_backoff_seconds	3	Linear retry delay: 3, then 6 seconds
stream_open_retries	2	Transient stream-opening errors, before consuming tokens
stream_open_backoff_seconds	1	Exponential retry delay: 1, then 2 seconds
outer_iteration_multiplier	3	Outer-loop safety bound relative to action budget
min_outer_iterations	12	Minimum outer-loop safety bound per invocation

The HTTP field max_iterations remains supported and overrides max_actions. Interactive chat omits this field and uses the shared default of 10 actions. Budgets must be positive integers. Retry counts and finite backoff durations may be zero, but not negative. Invalid request budgets fail before execution. No new upper cap, environment overrides, or user-facing controls are introduced.

Python callers and evaluations can pass execution_config=AnalystExecutionConfig(...) to AnalystAgent; an explicit legacy max_iterations argument takes precedence over its action budget. The configuration is immutable and resolved per instance. Effective settings are recorded in the reasoning log's session_start event. The legacy max_repair_attempts argument is accepted but ignored.

Empty-response retries still consume tool rounds; if no round remains, the agent reports the provider error without waiting. Stream-opening retries happen within one model round. Neither budget is a total token, cost, or wall-clock limit. Reasoning-effort settings remain independently configured in the same module.

Connector Timeouts

Connection validation and catalog discovery are separate requests. The connection form creates a definition with connect_params: {} before validating it, avoiding duplicate connection tests. Transport timeouts and retryable errors retain the connector and trigger a status check; confirmed authentication failures retain the existing cleanup behavior.

The catalog UI uses get-catalog-tree with background: true. Discovery runs on the catalog-refresh executor; subsequent requests use poll: true. Progress and completion state are stored in the user's catalog_discovery directory. A file lock prevents duplicate discovery across workers sharing that directory. Empty catalogs are cached too. Failed scans preserve the previous cache; retry: true starts another attempt. Legacy callers without background remain synchronous.

UI polling has a 10-second per-request limit, up to three retries with backoff for transient transport errors, and a five-minute waiting window. Ending that window does not cancel the server scan; retrying checks the existing job. Worker restarts interrupt jobs and require retry. This is not a distributed durable job queue: multiple instances need a shared data directory for status and locking.

Azure Blob SDK requests use a five-second connection timeout, ten-second read timeout, and two retries with backoff. These are per-request limits, not an overall deadline for identity acquisition, pagination, or PyArrow file reads.

Kusto ambient authentication keeps one access token in memory per credential instance, reusing it only for matching scopes and token options while more than five minutes of validity remain. Refresh is serialized across concurrent calls; changed scopes, tenant, claims, or CAE options require another acquisition. Unknown options bypass and clear the cache. This avoids repeated Azure CLI token lookups during connection checks and catalog requests, but does not eliminate initial authentication. Tokens are not shared across loader instances or written to disk. Delegated and service-principal authentication paths are unchanged.

Agent Query Workers

Agent streaming runs lazily start a dedicated subprocess for remote file queries and reuse it until that run ends. Each query creates a fresh loader and connection; workers are not shared across runs or users. Other connector methods retain direct execution. Calls without an agent cancellation context are also unchanged.

DF_QUERY_MAX_WORKERS bounds live query workers per backend server process (default 2). Additional runs wait for capacity, checking cancellation while queued. A worker keeps its slot between queries until the run finishes. With multiple server processes or replicas, multiply this limit by their count when budgeting memory; this is not an instance-wide or distributed quota. DuckDB's query memory limit is not a cap on the entire worker process.

DF_QUERY_MEMORY_LIMIT_MB sets the DuckDB memory limit of one file-scan query (Azure Blob, S3, local folders). By default each query gets half of physical memory divided by DF_QUERY_MAX_WORKERS, kept between 512 MB and 4 GB. Sorts, aggregates, and joins spill to a temp directory instead of failing, but the loaded result must fit. The agent is told the budget in the source's query capabilities, so it narrows columns, rows, or aggregates to stay within it.

DF_QUERY_QUEUE_TIMEOUT_SECONDS limits waiting for capacity (default 300). DF_QUERY_TIMEOUT_SECONDS limits worker startup and query response time after capacity is acquired (default 300). Cancellation, disconnect, or a query timeout terminates the worker; normal run completion also releases it. Ordinary query errors leave it reusable. A worker crash fails the current query without taking down the backend, and a subsequent query starts a replacement without replaying the failed query. Worker processes are daemons and are terminated during normal backend shutdown; runs are not durable across server restarts.

Sessions in Multiple Tabs

Each browser tab works on its own session, named in the URL (/app?session=<id>), so sessions can be compared side by side, reloaded, bookmarked, or opened from the Sessions list with Open in new tab (or Cmd/Ctrl/middle-click). The backend scopes every request by the signed-in or local identity plus the tab's X-Workspace-Id, so tabs share connectors, models, workflows, and schedules while agent runs, workflow runs, and scratch state stay per session.

On start, a tab opens the URL's session from the backend rather than trusting browser-restored state, which other tabs share. One tab edits a session at a time (src/app/sessionTabs.ts): opening a session claims it over a BroadcastChannel; a tab already editing it saves and becomes view-only with an Edit here action before the claimer loads the latest state. This coordinates tabs in one browser; concurrent edits to one session from different browsers or devices remain last-writer-wins until session saves are versioned.

Unified Workspace Loading

Agent propose_data_operation uses the manual-import row/byte thresholds for query-free source additions. Known large sources return result_references using the existing external-reference identity and metadata format, without fetching rows or creating Parquet. Small or unknown-size sources follow ordinary loading. Reference results are persisted with the operation and workflow checkpoint; the frontend upserts them into workspace state by source/table identity.

Supplying query, including {}, explicitly requests materialization. This intent is persisted in the plan hash, and concrete queries never fall back to virtual registration. Query execution also adds a virtual source reference when the source/table is absent from the current reference inventory and local-table provenance, regardless of source size. No separate preparation call is needed. The source retains its catalog name rather than the query-result label. If the query fails, registration and failure are returned separately; registration is not evidence of query success. Native queries retain source association without claiming verified row-level lineage. Native/aggregate limits and coverage rules remain unchanged. Agent observations distinguish virtual (compute_ready: false, no local path) from materialized (compute_ready: true, path and scope) outcomes. Workflow registration is input preparation, not a computed deliverable.

Structured Aggregate Loading

Loads with group_by or aggregates call the connector's query_data_as_arrow, and query_capabilities.aggregate_loading is supported only when a connector implements it. Kusto, S3, Azure Blob, and local folders support it, as do the SQL connectors PostgreSQL, MySQL, SQL Server, BigQuery, ClickHouse, and Athena. The SQL connectors reuse the probe compiler (probe_utils.query_via_native_sql) and run one generated SELECT on the source. It emits only quoted identifiers, escaped literals, and the fixed aggregate vocabulary; agent-written SQL is never executed. Each connector resolves the table as its ordinary load does (PostgreSQL database routing, the SQL Server dbo default, BigQuery whole-path quoting, ClickHouse's configured-database restriction). Unlike probes, durable loads fail on any filter or ordering they cannot compile rather than dropping it, so a result never widens past its requested scope. The 10,000-row aggregate limit and overflow rejection apply unchanged. MongoDB, Databricks, Cosmos DB, and Superset remain unsupported.

Native KQL Loading

Kusto advertises query_capabilities.native_query_languages: ["kql"]. propose_data_operation accepts query.native with language: "kql" and text, mutually exclusive with structured query fields except limit. Prefer bounded ordinary loads followed by local Python; native queries are for source-side reductions that cannot be expressed by structured loading or whose raw inputs cannot reasonably be loaded. Other connectors reject native queries.

The Kusto adapter uses the query endpoint, never command dispatch, and prepends an exact-table restrict access statement. Server request properties enforce read-only/hardline execution and disable callouts, external data/tables, remote entities, impersonation, and sandboxed execution. Agent text cannot contain commands, semicolons, comments, or request-setting statements. These conservative text restrictions are not the security boundary: Kusto permissions and request properties are. Use least-privilege, read-only connector credentials in deployment. Do not fall back to unrestricted execution if a server rejects these properties.

Requests have a 60-second server deadline, 16-MiB response cap, and at most 10,000 loaded rows. Partial failures fail the load. Without an explicit result limit, a 10,001-row sentinel rejects overflow rather than publishing partial data. Limits and sampling within native text still define partial coverage; native results are labeled query_defined, not complete-population aggregates. Small results do not bound scan cost. Cancellation is checked around the SDK call; an in-flight server query may continue until its deadline.

Preview, publication, and refresh use the same guarded adapter. Native text is persisted in import metadata; never put credentials or secrets in query text. Selected source identity is retained, but lineage is marked unverified and does not inherit a verified single-source shelf group. No native query is executed through a local Python fallback.

Starter Questions for External References

/api/agent/derive-starter-questions accepts input_tables and optional external_references. primary_table identifies either a loaded table name or an external reference ID. Focusing a reference uses the existing starter-question chips, even when no tables have been loaded. Cached questions are invalidated when the reference metadata or preview changes.

The starter agent uses cached schema, descriptions, row counts, inspection limits, query intent, and at most five bounded sample rows through the shared reference normalizer. It does not query connectors or materialize data during automatic suggestion generation. The prompt distinguishes preview evidence from verified source coverage and avoids assuming recent dates or complete category coverage. Selecting a question sends the reference and its focus to the normal analyst flow, where connector inspection and scoped queries can run as needed.

Option 1: With uv (recommended)

uv is faster and provides reproducible builds via lockfile.

uv sync                        # Creates .venv and installs all dependencies
uv run data_formulator         # Run app (opens browser automatically)
uv run data_formulator --dev   # Run backend only (for frontend development)

Which command to use:

• End users / testing the full app: uv run data_formulator - starts server and opens browser to http://localhost:5567
• Frontend development: uv run data_formulator --dev - starts backend server only, then run yarn start separately for the Vite dev server on http://localhost:5173

Option 2: With pip (fallback)

• Create a Virtual Environment

  python -m venv venv
  source venv/bin/activate  # Unix
  # or .\venv\Scripts\activate  # Windows
  

• Install Dependencies

  pip install -r requirements.txt
  

• Configure environment variables (optional)

  • copy .env.template to .env and fill in your values:
    • API keys: set {PROVIDER}_API_KEY=... (and/or {PROVIDER}_API_BASE=...) and {PROVIDER}_MODELS=... for each LLM provider you want to use; a provider with models and a key or base URL is enabled automatically. See the LiteLLM setup guide for provider-specific fields.
    • Server settings: DISABLE_DISPLAY_KEYS, SANDBOX, etc.
    • Azure Blob workspace (optional): see Azure Blob Storage Workspace below.
  • this lets Data Formulator automatically load API keys at startup so you don't need to enter them in the UI.

• Run the app

  # Unix
  ./local_server.sh
  
  # Windows
  .\local_server.bat
  
  # Or directly
  data_formulator         # Opens browser automatically
  data_formulator --dev   # Backend only (for frontend development)
  

Local Terminal Skill

The analyst and workflow agents can use the terminal skill as a data-acquisition route alongside connected sources. They first ground the question in relevant workspace inputs, then choose an available route for missing data. Local files, installed clients, public endpoints, and existing CLI logins can supply bounded datasets directly into scratch without setting up a connector. For example: "Analyze my Azure usage with my existing CLI account." The agent discovers the relevant source and scope, retrieves actual data, and registers one bounded reusable workspace input with create_data and acquisition metadata before continuing with Python analysis and chart/report tools. Parsing and normalization can happen in the registration call; its returned input ID/path needs no extra inventory call. These are agent-managed source tables, not user uploads or derived chart results. Source, scope, optional query/limitations, server-recorded acquisition time, and input file hashes persist with the table. Explicit refreshes use update_data with its current hash, new file inputs, and new acquisition metadata. Discovery alone is not acquisition, and acquisition alone does not complete an analysis request.

Connector forms remain appropriate when direct acquisition is unsuitable or the user wants reusable connected access. Running a CLI does not register a connector or publish a workspace table: acquired data stays in scratch and workspace tools perform registration. Discovery-only metadata, response fragments, and calculated intermediates stay in scratch; disposable caches use private runtime storage outside workspace exports. Relevant non-tabular inputs use create_file. Requests for uploads, authentication, or clarification should address concrete blockers, not replace an available authorized acquisition step.

Terminal access supports single-user local mode on macOS, Linux and Windows. The terminal is one of the agent's tools, alongside the user's MCP servers: all tools share one approval mode, and each has its own on/off switch. Both the analyst and workflow agents use the same server-owned policy:

• Off (off): no terminal tool, skill catalog entry, or application-supplied terminal guidance. Forged or stale execution requests are rejected. Sandboxed Python analysis remains available.
• Ask every time (ask, the default): terminal tools and operating guidance are preloaded from the first turn. Each exact invocation opens a Run once / Reject dialog showing arguments, working directory, purpose, and host-access warning. Approval covers the entire invocation, including a shell script, not each individual line.
• Auto when safe (auto): the same preloaded tools and guidance, with immediate sandboxed execution, including writes allowed by the filesystem policy. dangerouslyDisableSandbox always requires user approval and a reason.

Click Tools beside the chat input. The Tools dialog applies each change immediately: the shared mode (Off / Ask every time / Auto when safe), the Terminal switch (stored as overrides.terminal_enabled; off turns the terminal off without changing the shared mode), and, in the Terminal row's details, the sandbox write paths. This local-user setting is available without managed mode or Administration access. Its dedicated endpoint (/api/configurations/terminal) accepts only mode, enabled, sandbox and the configuration revision, requires a same-origin local request, and preserves all other settings.

Alternatively, launch with DF_TERMINAL_MODE=off|ask|auto, which sets the shared tools mode. The environment takes precedence and locks the selector; invalid environment values fail closed to Off. Persisted configuration uses overrides.terminal_mode. Deployment restrictions still apply. Chat text, workflow definitions, and tool arguments cannot set it.

Approvals are stored server-side for one invocation, bound to the identity, workspace, conversation, mode, and configuration revision. They expire after ten minutes or a backend restart. Configuration changes invalidate outstanding approvals, even if the mode is later restored. Execution rechecks current policy; a change while a command is running stops it at the next runner check. Resumes rebuild application capability guidance without erasing actual conversation history.

Commands normally run with OS-enforced filesystem write confinement: macOS uses /usr/bin/sandbox-exec; Linux requires Bubblewrap (bwrap) and enabled user namespaces. Writable areas include workspace scratch, private per-invocation runtime storage (mode 0700), and the filesystem policy below. Children inherit the restriction.

Without a sandbox (Linux without a working Bubblewrap, or Windows without WSL or without Bubblewrap inside WSL), the terminal still works, but every command runs unsandboxed with the user's normal access: proposals are marked dangerouslyDisableSandbox with sandboxUnavailable, so each one asks for approval even in Auto. The approval and Tools dialogs show a warning and an install hint, and the agent is told. Installing a sandbox takes effect after a backend restart.

On Windows, commands run in WSL (data_formulator/sandbox/wsl.py) when it has a working default distribution: wsl.exe -e bwrap … -- <argv>, as on Linux, with Windows paths bound as /mnt/<drive>/…, the DEFAULT_ALLOW_WRITE CLI-state paths also bound in the WSL home when present, and WSL interop (/run/WSL, WSL_INTEROP) hidden so a sandboxed command cannot start Windows programs. Command variables reach WSL through WSLENV (paths translated). Unsandboxed commands run as wsl.exe -e <argv>; without WSL they run natively and the agent is told to use PowerShell. Process start, output streaming, interrupt and tree kill go through data_formulator/analyst/process_io.py (process groups and a selector on POSIX; a reader thread, CTRL_BREAK and taskkill /T on Windows). The Windows paths are covered by simulated unit tests only. Commands receive DF_SCRATCH_DIR for outputs and DF_RUNTIME_DIR for disposable state. TMPDIR/TMP/TEMP, XDG_CACHE_HOME, UV_CACHE_DIR, PIP_CACHE_DIR, npm_config_cache, YARN_CACHE_FOLDER, MPLCONFIGDIR, HF_HOME, and NUMBA_CACHE_DIR point into runtime storage, removed on completion/cancellation and excluded from workspace exports. These caches are disposable, not shared across commands. The working directory does not grant write access. Missing or failing confinement never silently falls back to unrestricted execution.

The built-in persistent sandbox.filesystem.allowWrite policy is:

Paths	Purpose and scope
~/.azure	Existing Azure CLI state: command logs, MSAL token/HTTP caches, locks, atomic updates. Entire directory writable.
~/.config/gcloud	Existing gcloud state: token databases, SQLite journals, logs, configuration. Entire directory writable.
~/.aws/cli/cache, ~/.aws/sso/cache, ~/.aws/login/cache	AWS role, SSO, and login token caches. AWS credentials and config files remain read-only.
~/.kube/cache, ~/.kube/http-cache	Kubernetes discovery and HTTP caches. Kubeconfig remains read-only.

Existing Azure/gcloud config-directory environment overrides replace their default paths only when inside the home directory and passing the same path checks. Outside-home overrides need a user-configured grant or reviewed bypass. Missing AWS/Kubernetes cache children are created under existing state roots; absent Azure, gcloud, AWS, or Kubernetes installations are not initialized. Unsafe or unavailable paths are skipped, not broadened. No automatic grants cover SSH files, keychains, Git config, package installation environments, or the entire home.

Users can replace the built-in list in the installation configuration's overrides (an empty list disables all persistent grants):

{
  "sandbox": {
    "filesystem": {
      "allowWrite": ["~/.azure", "~/.aws/sso/cache", "/absolute/custom-cli-state"]
    }
  }
}

Configured entries are literal absolute paths or ~/ paths, at most 64; no globs. They must exist to be granted. Paths are re-resolved before every execution; symlink redirects, nonregular/hard-linked files, root/home-wide grants, and paths covering application configuration are rejected or skipped. Prefer directories for lock files and atomic replacement; Linux file bind mounts cannot be renamed. Directory grants allow modification/deletion of all contents, including credentials and executable configuration, not just harmless refreshes. Preexisting hard links inside a granted directory can alias other files; path grants are not a content-level security boundary. These are explicit compatibility tradeoffs. The local terminal access dialog shows resolved paths; settings changes invalidate pending commands through the configuration revision. Agents cannot submit path grants or change this policy through run_terminal.

For a necessary operation outside the policy, the agent submits the exact command with dangerouslyDisableSandbox: true and a nonempty sandboxDisablingReason. The dialog shows the reason, full host-write risk, and Run outside sandbox / Reject. Both Ask and Auto require this explicit approval. The stored command, flag, and reason are authoritative; client responses contain only request ID and decision. Approval is single-use and never applies to subsequent commands. An approved command bypasses filesystem confinement with normal host-user access, not root privileges; runtime storage, filtered environment, time limits, and local-policy checks remain. Changes made on the host remain afterward.

Retry flow: inspect the error and partial effects, redirect disposable state when possible, then submit a reasoned bypass request if required. The runtime never automatically retries outside the sandbox. Permission-looking stderr is a hint, not proof of sandbox denial or invalid credentials. Rejected requests must not be retried via another route. Legacy write_paths proposals are refused.

Each command has a fresh process, no interactive stdin, a 60-second timeout, and the last 32 KiB of combined output. The process group is terminated on timeout or when the execution generator closes; macOS process-group cleanup alone does not guarantee termination of deliberately detached descendants. Ordinary server API key environment variables are not inherited, but local files and cached CLI credentials remain accessible. Selected CLI profile/config-location variables and proxy/CA settings are inherited, but credential values such as API keys are not copied from the server environment. Login stores are never copied into scratch or runtime storage. Command arguments, bypass reasons, policy snapshots, and results appear in the conversation and are sent to the configured model, so do not print credentials or other sensitive data. Complete interactive authentication outside the agent. This is write confinement, not complete isolation: network access remains enabled, and remote mutations or effects delegated to external services are not prevented by the filesystem boundary. Auto approval does not remove these risks or authorize actions outside the user's requested task.

Scratch files are absent from the ordinary workspace listing. In Backend Log, the Scratch files tab lists visible scratch entries for the active workspace, with read-only previews and downloads. Hidden execution state remains internal.

Terminal access is rejected in hosted mode, when data connectors are disabled, or without a matching local Host and Origin. The Vite analyst proxy preserves Host for this check. Restart the backend after adding the skill; Vite reloads its proxy configuration automatically. Pending approvals are transient and cannot be restored after reloading the page; ask for a fresh proposal.

Focused checks (no browser automation required):

uv run pytest tests/backend/agents/test_terminal_skill.py tests/backend/agents/test_terminal_windows.py tests/backend/agents/test_background_tasks.py tests/backend/agents/test_analyst_skill_registry.py tests/backend/routes/test_analyst_data_operation_flow.py
npx vitest run tests/frontend/unit/views/TerminalApprovalDialog.test.tsx tests/frontend/unit/views/ToolsAccess.test.tsx

MCP Servers (Local Mode)

In single-user local mode the analyst agent can call tools from MCP servers the user adds, as tools next to the terminal: Tools beside the chat input lists them, and + Add MCP server takes one server's MCP configuration JSON (How to add a server in the form shows examples), the format VS Code (servers), Claude Desktop and Cursor (mcpServers) use; Codex TOML ([mcp_servers.*]) is also accepted. A remote server is a url ("type": "sse" for older servers) with optional headers; a local one is a command with args, env and cwd. MCP is off in hosted and managed deployments and when data connectors are disabled.

• Storage. Server definitions and the reviewed tool snapshots live in ~/.data_formulator/mcp.json (under DATA_FORMULATOR_HOME). API keys, header values, environment variables, OAuth tokens and client secrets go to the credential vault under mcp:<server-id>, never into the file.

• Sign-in. Public servers need nothing. A headers entry sends an API key or token. A server that answers 401 without either is retried with the MCP SDK's OAuth flow (PKCE, dynamic client registration), as other MCP clients do; when it offers no OAuth, the error suggests adding a header. For providers that require a pre-registered app, add "oauth": {"client_id": …, "client_secret": …, "scope": …}. The redirect URI is http://localhost:<port>/api/mcp/oauth/callback. Agent runs never start a sign-in; an expired sign-in asks the user to sign in again in settings.

• Remote URLs must be HTTPS except localhost/127.0.0.1. Local commands run as the user, outside the terminal sandbox, so add only trusted servers.

• Calls. Each enabled server becomes a gated skill (mcp-<id>) whose actions are mcp_<id>__<tool> with the tool's own input schema. They follow the shared tools mode: Off; Ask every time (default); Auto when safe (tools annotated readOnlyHint run without asking). Tools the user marked Always allow never ask. Approval pauses the run like a terminal command (Reject / Always allow this tool / Run once); the server-side request is single-use and bound to the conversation, and the stored arguments, not the client's, are what run.

• Tool changes. The first connection's tools count as reviewed. New or changed tools (name, description, schema or annotations) are held until the user allows them in settings; a call to a tool that changed since approval is refused.

• Results. Short text returns inline (marked untrusted). Large text, images, audio and binary content are saved under scratch/mcp/<server>/; the agent promotes anything worth keeping with its normal workspace actions.

• Agent-assisted setup. The configure skill's list_mcp_servers shows the agent each server's status, last error, missing key or sign-in, and tools (never secret values). propose_mcp_changes publishes an mcp setup form that opens in the Tools dialog (not the canvas; the thread entry reopens it): each proposed server appears as prefilled configuration JSON with empty "" slots for keys and environment values, which the user fills in before Add and connect. The dialog closes once the new server connects (it stays open while signing in or on errors); server/tool switches and the shared tools mode apply on Apply changes. The form never submits itself, proposals containing secret values are rejected, and approving changed tools or Always allow stay manual. Both tools exist only when MCP is available and are excluded from workflow runs.

• Toolkits (tool packs). + Add tool → Toolkits adds a ready-made server that ships with Data Formulator (py-src/data_formulator/tool_packs/) in one step; it runs with Data Formulator's own Python (no Node or uv download) and is then an ordinary MCP server row (pack records its origin). GET /api/mcp/packs lists packs; POST /api/mcp/packs/<id> adds one (once) with its optional settings.

  • Research (arXiv) (tool_packs/research): search_papers, recent_papers, paper_details (up to 20 ids), count_papers (per year/month, for trends) and get_full_text (arXiv's HTML as Markdown, else the PDF; saved to scratch). All are read-only. It uses only arXiv's own API and pages (no citation counts) and is polite by construction: one connection, requests at least 3 seconds apart, a descriptive User-Agent with the optional contact email, retries with backoff on 429/5xx (honouring Retry-After) and on the API's "results but no entries" glitch, and a disk cache under ~/.data_formulator/cache/research (searches a day, papers a week, versioned papers for good). Large JSON results save to scratch as .json so the agent can load them with create_data.

Routes live under /api/mcp (py-src/data_formulator/mcp_servers/routes.py); changes require a same-origin local request with X-DF-Configuration: 1. The Vite dev proxy preserves Host for /api/mcp, so the check and OAuth redirects use the app's own origin.

uv run pytest tests/backend/mcp

Azure CLI Deployment Discovery

In local mode, choose Add Model > Azure > Azure CLI, sign in, and select Browse deployments. The picker defaults to the CLI's current subscription and lists ready OpenAI model deployments grouped by resource. Selecting a deployment fills in its endpoint and deployment name; Test and save checks inference access using the existing Azure identity configuration.

Discovery uses read-only Azure CLI commands with explicit subscription arguments; it does not change the active CLI subscription, create deployments, or retrieve API keys. No additional app registration or Python package is required.

The initial picker supports public Azure OpenAI and Foundry (AIServices) resources in enabled subscriptions in the current CLI tenant. It does not list the undeployed Foundry catalog, other model formats, or sovereign-cloud endpoints. To change tenant, sign in with the intended tenant through Azure CLI and reopen the model dialog. Resource/deployment read permissions are separate from inference permissions. Partial discovery failures are shown per resource. Enter manually remains available for restricted discovery, unsupported endpoints, and custom configurations. Network restrictions still apply to inference.

OpenRouter Account Connection

In Select Model, choose Add Model > OpenRouter > Connect OpenRouter. Authorization uses OpenRouter's OAuth PKCE flow; no application client secret is needed. After authorization, choose a tool-capable model and use Test and save. Model tests and subsequent usage are billed to the user's OpenRouter account.

The returned API key stays in the backend's encrypted credential vault. Model configurations, including knowledge-distillation requests and workspace exports, carry only a per-user connection reference. One OpenRouter account connection can serve multiple models. Removing a model does not disconnect the account. Disconnect forgets the saved key locally; revoke it separately in OpenRouter's key settings when needed. Reconnect starts a new authorization flow rather than refreshing a subscription token.

Local loopback callback origins are supported in local mode, including Vite's dev port. Hosted deployments require HTTPS. When a reverse proxy changes the apparent origin, set MODEL_CONNECTION_ALLOWED_ORIGINS to the exact public frontend origin (comma-separated for multiple origins). The frontend origin must route /api/model-endpoints/connections/openrouter/callback to this backend. Authorization state expires after ten minutes and is bound to the initiating Data Formulator identity, so callbacks also work when opened in an external browser.

The credential vault must be available, including when user-created data connectors are disabled. Persist DATA_FORMULATOR_HOME and its vault key across restarts. If DF_ALLOWED_API_BASES is configured, include https://openrouter.ai/api/v1 to permit inference through this connection.

GitHub Copilot Account Connection (Experimental)

In Select Model, choose Add Model > Sign in > GitHub Copilot > Connect GitHub Copilot. Copy the displayed device code, open GitHub, and authorize the account. After authorization, select a compatible model and choose Test and save. Testing and subsequent agent requests consume the account's Copilot allowance; subscription limits, model access, and organization policies still apply. This is not GitHub Models or an API-key integration.

Device authorization uses the same default public OAuth client ID as LiteLLM's Copilot adapter. GITHUB_COPILOT_CLIENT_ID can override it, but an arbitrary OAuth app is not guaranteed Copilot entitlement. The adapter uses Copilot internal token exchange endpoints and client headers; this is not a claim of official GitHub support for third-party subscription clients. Review applicable GitHub terms and organization policies before enabling it in a deployment.

Data Formulator stores the GitHub OAuth token and expiring Copilot token in its identity-scoped encrypted vault. Device polling honors the provider interval and slow_down; cancellation and expiry prevent a late exchange from saving tokens. Copilot tokens are refreshed when resolving a connection for a new inference client or model refresh. The frontend and saved model configurations receive no tokens. Multiple saved models can share one connection; Edit > Disconnect forgets that connection locally without deleting the models or revoking GitHub authorization. Revocation is available separately in GitHub's application settings.

The installed LiteLLM github_copilot/ adapter ignores explicit API keys and reads a shared on-disk cache. Data Formulator therefore uses LiteLLM's OpenAI-compatible transport with explicit vault-resolved credentials and Copilot headers instead. It does not read or write LiteLLM's Copilot token files or launch terminal login. The picker includes enabled tool-calling chat models advertising /chat/completions or /responses. The backend caches each model's transport in the account connection when the catalog is refreshed; models advertising both keep Chat Completions. Responses-only models use the Responses transport. Models advertising only native protocols such as /v1/messages remain excluded.

ChatGPT Account Connection (Experimental)

In Select Model, choose Add Model > Sign in > ChatGPT > Sign in with ChatGPT. Enable device-code login in ChatGPT security settings, then enter the displayed code on OpenAI's authorization page. Select an account model and choose Test and save. Testing and agent requests use the account's subscription allowance; model availability, usage limits, and applicable OpenAI terms still apply. This is separate from OpenAI API-key access and does not provide API credits.

OAuth access and refresh tokens stay in Data Formulator's identity-scoped encrypted vault. The browser and saved models hold only a connection reference. Tokens are refreshed when resolving a new inference client or loading the model catalog. Edit > Disconnect deletes the local connection while retaining saved models; manage authorization separately in ChatGPT settings.

The pinned LiteLLM version's native ChatGPT adapter uses a shared token file by default. The small agents/chatgpt_transport.py compatibility override replaces its config factories with request-authenticated subclasses, without global tokens, environment changes, or token files. Native ChatGPT request transformation, Responses streaming, and response parsing remain in LiteLLM. Revalidate this override when upgrading LiteLLM. The integration uses ChatGPT's Codex backend and model catalog, which can change independently of the public OpenAI API; it is not a claim of official support for third-party subscription clients.

Model Client Transports

Agents use the same Client.get_completion and get_completion_with_tools methods for both transports. Client dispatches internally to Chat Completions or LiteLLM's Responses bridge, returning chat-style messages and streaming deltas. The provider model identity is unchanged; bridge-specific prefixes and parameter mapping are confined to the transport implementation.

Backend OpenAI and Azure configurations can select api_type: "responses" or api_type: "chat_completions"; omitting it preserves existing LiteLLM routing. This is a backend configuration option, not a new control in the model dialog. Copilot resolves the value from its server-side catalog, overriding caller input. Refresh the model list to pick up changed Copilot capabilities.

Explicit Responses requests use store: false and request encrypted reasoning items for replay in locally managed message history. Both streaming agent loops retain these opaque items without rendering them as user-visible text. The transport supports text and function tools, not provider-hosted tools or background Responses jobs. It does not retry failed generation on a different transport. Live provider behavior, billing, and immediate upstream cancellation still require integration testing; the network-free tests verify protocol conversion, tool turns, reasoning replay, streaming, usage, and failure handling.

The encrypted vault must be enabled and persisted as described above. No callback URL is needed for device authorization. Outbound access is required to github.com, api.github.com, and the Copilot API. Only these API bases are accepted: https://api.githubcopilot.com, https://api.individual.githubcopilot.com, https://api.business.githubcopilot.com, and https://api.enterprise.githubcopilot.com. Include the applicable bases in DF_ALLOWED_API_BASES when that allowlist is enabled. Custom GitHub Enterprise hosts are not supported by this initial implementation.

Frontend (TypeScript)

• Install NPM packages

  yarn
  

• Development mode

  First, start the backend server (in a separate terminal):

  uv run data_formulator --dev   # or ./local_server.sh
  

  Then, run the frontend in development mode with hot reloading:

  yarn start
  

  Open http://localhost:5173 to view it in the browser. The page will reload if you make edits. You will also see any lint errors in the console.

• Component galleries

  /dev shows UI components in their states; /dev/maps shows sample map artifacts (every layer type and control, with generated data) and lets you edit their JSON or open the Style panel. See design-docs/70-map-artifact.md.

Advanced: developing against a local flint-chart

DF depends on the published flint-chart package; by default (FLINT_CHART_LOCAL unset) the bare flint-chart import resolves to that npm package.

For co-development against a local Flint checkout with hot reload, point FLINT_CHART_LOCAL at its source when starting the dev server (or tests):

FLINT_CHART_LOCAL=../flint-chart/packages/flint-js/src yarn start
# tests:
FLINT_CHART_LOCAL=../flint-chart/packages/flint-js/src yarn test

Opt-in, advanced-dev only — leave it unset for normal work and CI, which use the installed package. The alias is wired in vite.config.ts and vitest.config.ts.

Build for Production

• Build the frontend and then the backend

  Compile the TypeScript files and bundle the project:

  yarn build
  

  This builds the app for production to the py-src/data_formulator/dist folder.

  Then, build python package:

  # With uv
  uv build
  
  # Or with pip
  pip install build
  python -m build
  

  This will create a python wheel in the dist/ folder. The name would be data_formulator-<version>-py3-none-any.whl

• Test the artifact

  You can then install the build result wheel (testing in a virtual environment is recommended):

  # replace <version> with the actual build version. 
  pip install dist/data_formulator-<version>-py3-none-any.whl 
  

  Once installed, you can run Data Formulator with:

  data_formulator
  

  or

  python -m data_formulator
  

  Open http://localhost:5567 to view it in the browser.

• Customize the home page (optional)

  A demo is a front page of your own at /demo/<id>, configured in a profile.yaml with no rebuild. See py-src/data_formulator/demos/README.md; the UIS demo at /demo/unesco is an example.

Desktop installer validation

The desktop builds GitHub Actions workflow builds unsigned test artifacts. Validate this path before integrating production signing. Windows installers must be built and exercised on Windows; a successful macOS build is not Windows installation evidence. Use a disposable Windows 11 x64 user account with an interactive desktop, PowerShell 7, Inno Setup 6, Node/Yarn, and uv:

yarn install --frozen-lockfile
yarn build
uv sync --extra desktop --frozen
uv run pytest tests/backend/test_desktop_packaging.py tests/backend/test_startup_spinner.py tests/backend/test_desktop_single_instance.py -q
uv run pyinstaller --noconfirm --clean packaging/data_formulator_desktop.spec
./packaging/windows/build-installer.ps1 -PayloadDir 'dist/Data Formulator' -OutputDir release -Unsigned

The wrapper emits a versioned *-Setup-unsigned.exe, SHA-256 sidecar, and .payload.json file manifest. Keep the manifest beside the installer when running the installed-app test (substitute the generated filename):

./packaging/windows/test-installer.ps1 -Installer 'release/Data-Formulator-0.8.0b1-Windows-x64-Setup-unsigned.exe'

The test refuses to replace an existing installed app. It checks payload hashes, native GUI/backend/sandbox startup, same-version reinstall, uninstall, and retention of isolated application data. Logs and installation timing are saved under build/installer-test; GitHub CI uploads them even when a step fails. Different-version upgrade and browser-download acceptance remain separate tests. Setup rejects destinations that would exceed the supported payload path length before writing application files; use /DIR="a shorter per-user path" if needed. The installed-app test covers this failure path as well as normal installation.

Setup installs per-user, preserves DATA_FORMULATOR_HOME/~/.data_formulator, and provisions Microsoft's WebView2 Runtime if absent (network access required in that case). Silent setup/uninstall supports /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /LOG="path" in the intended user context. Uninstall does not remove user data or the shared WebView2 Runtime.

Unsigned Windows installers are CI artifacts, not automatically published release assets. Production signing must cover the application, setup, and generated uninstaller before repeating validation on the actual browser download. Do not use manual unblocking or antivirus exclusions to declare a release usable.

For ADO task-based signing, the wrapper supports three explicit phases around an already-signed payload. These replace an inline -SignCommand; do not combine them with -Unsigned:

./packaging/windows/build-installer.ps1 -PayloadDir 'dist/Data Formulator' -OutputDir candidate -SigningPhase PrepareUninstaller -SignedUninstallerDir build/signed-uninstaller
# ESRP signs the single generated EXE in build/signed-uninstaller.
./packaging/windows/build-installer.ps1 -PayloadDir 'dist/Data Formulator' -OutputDir candidate -SigningPhase AssembleInstaller -SignedUninstallerDir build/signed-uninstaller
# ESRP signs the generated candidate/*-Setup.exe.
./packaging/windows/build-installer.ps1 -PayloadDir 'dist/Data Formulator' -OutputDir candidate -SigningPhase VerifyInstaller

Use an empty per-candidate cache and identical compiler/version/icon settings for preparation and assembly. The preparation phase recognizes only Inno's documented request to externally sign the generated uninstaller; other compilation failures are fatal. Assembly verifies the cached uninstaller but does not emit release checksums. Final verification requires valid payload signatures and timestamped Microsoft signatures on the launcher/setup before emitting checksum and manifest sidecars. Run test-installer.ps1 -RequireSignatures on the resulting installer before any promotion; this also verifies the installed uninstaller.

On a service-session ADO agent, test-installer.ps1 -ValidationMode Headless can exercise installation, signatures, payload integrity, sandbox/CLR, reinstall, and uninstall without an interactive desktop. This is candidate-only validation: installation.json records guiVerified: false, and the GUI report explicitly records that it was skipped. Full validation remains the default. Publish headless results only as distinctly labeled candidate artifacts; require full interactive and browser-download acceptance before release promotion.

Docker

Docker is the easiest way to run Data Formulator without installing Python or Node.js locally.

Quick start

1. Copy the environment template and add your API keys:

   cp .env.template .env
   # Edit .env and set your OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.
   

2. Build and start the container:

   docker compose up --build
   

3. Open http://localhost:5567 in your browser.

To stop the container: docker compose down

Workspace data (uploaded files, sessions) is persisted in a Docker volume (data_formulator_home) so it survives container restarts.

Compose publishes the port on 127.0.0.1 only and sets DF_SINGLE_USER=1, so you get single-user local mode (no login, all local features). To share the container with other machines, remove both: other clients then get separate anonymous workspaces instead of acting as you.

Build the image manually

docker build -t data-formulator .
docker run --rm -p 127.0.0.1:5567:5567 -e DF_SINGLE_USER=1 --env-file .env data-formulator

Without DF_SINGLE_USER=1 the container (which listens on 0.0.0.0) runs in anonymous multi-user mode. Set it only when the published port is reachable from your machine alone.

Docker sandbox (SANDBOX=docker) is not supported inside a container

The Docker sandbox backend works by calling docker run -v <host_path>:... to bind-mount temporary workspace directories into child containers. When Data Formulator itself runs in a Docker container those paths refer to the container filesystem, not the host, so Docker daemon cannot mount them and the feature does not work.

Use SANDBOX=docker only when running Data Formulator directly on the host (e.g. with uv run data_formulator --sandbox docker or python -m data_formulator --sandbox docker). When using the Docker image, keep the default SANDBOX=local.

Sandbox

AI-generated Python code runs inside a sandbox to isolate it from the main server process. Two backends are available:

Backend	Flag	How it works	Overhead
local (default)	--sandbox local	Persistent warm subprocess with pre-imported pandas/numpy/duckdb. Audit hooks block file writes and dangerous operations (subprocess, shutil, etc.).	~1 ms
docker	--sandbox docker	Each execution runs in a disposable docker run --rm container. Workspace is mounted read-only; output is returned via a bind-mounted parquet file. Memory/CPU/PID limits enforced.	~700 ms

# Use the default local sandbox
python -m data_formulator

# Use Docker sandbox (requires Docker daemon)
python -m data_formulator --sandbox docker

The Docker sandbox image is built from py-src/data_formulator/sandbox/Dockerfile.sandbox:

docker build -t data-formulator-sandbox -f py-src/data_formulator/sandbox/Dockerfile.sandbox .

Source: py-src/data_formulator/sandbox/

Azure Blob Storage Workspace

By default, workspace data (uploaded files, parquet tables, metadata) is stored on the local filesystem under ~/.data_formulator/workspaces/. For cloud deployments you can switch to Azure Blob Storage so all workspace data lives in a blob container instead.

Quick start (local dev with connection string)

1. Install extra dependencies:

   pip install azure-storage-blob
   # or with uv:
   uv pip install azure-storage-blob
   

2. Create a storage account & container (one-time setup):

   az storage account create -n <account> -g <resource-group> -l eastus --sku Standard_LRS
   az storage container create -n data-formulator --account-name <account>
   

3. Get the connection string:

   az storage account show-connection-string -n <account> -g <resource-group> -o tsv
   

4. Add to .env:

   WORKSPACE_BACKEND=azure_blob
   AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...
   # AZURE_BLOB_CONTAINER=data-formulator   # default, change if needed
   

5. Run normally:

   uv run data_formulator --dev
   

   Or pass as CLI flags:

   data_formulator --workspace-backend azure_blob \
     --azure-blob-connection-string "DefaultEndpointsProtocol=https;AccountName=..."
   

Production setup with Entra ID (no secrets)

In production (Azure App Service, AKS, etc.) you can authenticate the app to blob storage via Managed Identity instead of a connection string. This eliminates secrets entirely.

1. Install extra dependencies:

   pip install azure-storage-blob azure-identity
   

2. Assign a role to the app's Managed Identity:

   # Get the App Service's principal ID
   PRINCIPAL_ID=$(az webapp identity show -n <app-name> -g <rg> --query principalId -o tsv)
   
   # Grant it "Storage Blob Data Contributor" on the storage account
   az role assignment create \
     --assignee "$PRINCIPAL_ID" \
     --role "Storage Blob Data Contributor" \
     --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>"
   

3. Set environment variables (no secrets needed):

   WORKSPACE_BACKEND=azure_blob
   AZURE_BLOB_ACCOUNT_URL=https://<account>.blob.core.windows.net
   # AZURE_BLOB_CONTAINER=data-formulator
   

   The app uses DefaultAzureCredential, which automatically picks up the Managed Identity.

4. For local development with the same Entra ID path, log in with the Azure CLI:

   az login
   # Grant your user the same "Storage Blob Data Contributor" role
   az role assignment create \
     --assignee "<your-email@example.com>" \
     --role "Storage Blob Data Contributor" \
     --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>"
   

   Then set:

   WORKSPACE_BACKEND=azure_blob
   AZURE_BLOB_ACCOUNT_URL=https://<account>.blob.core.windows.net
   

   DefaultAzureCredential will use your az login session.

Authentication methods summary

Method	Env var	When to use
Connection string	AZURE_BLOB_CONNECTION_STRING	Local dev, quick tests
Entra ID (Managed Identity)	AZURE_BLOB_ACCOUNT_URL	Azure App Service, AKS — no secrets
Entra ID (az login)	AZURE_BLOB_ACCOUNT_URL	Local dev without secrets
Entra ID (service principal)	AZURE_BLOB_ACCOUNT_URL + AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_CLIENT_SECRET	CI/CD pipelines

If both AZURE_BLOB_CONNECTION_STRING and AZURE_BLOB_ACCOUNT_URL are set, the connection string takes precedence.

Blob layout

All workspace data is stored under <datalake_root>/<sanitized_identity_id>/ inside the container:

data-formulator/                          ← container
  workspaces/                             ← datalake_root (default)
    browser_550e8400.../                  ← anonymous user workspace
      workspace.yaml
      sales_data.parquet
    user_alice_example_com/               ← authenticated user workspace
      workspace.yaml
      quarterly_report.parquet

CLI flags reference

Flag	Env var	Default	Description
--workspace-backend	WORKSPACE_BACKEND	local	local, azure_blob, or ephemeral
--azure-blob-connection-string	AZURE_BLOB_CONNECTION_STRING	—	Shared-key connection string
--azure-blob-account-url	AZURE_BLOB_ACCOUNT_URL	—	Account URL for Entra ID auth
--azure-blob-container	AZURE_BLOB_CONTAINER	data-formulator	Blob container name

Deployment Profiles

Data Formulator runs in default mode unless administrator-managed operation is enabled. The deployment profiles below describe authentication and storage choices, not separate product editions.

Managed Mode

Start with data_formulator --managed or set DF_MANAGED=true to enable administrator-managed resources and policies. Managed mode is off by default. It does not change authentication, workspace storage, or the execution sandbox. It can be used locally for setup as well as on a hosted installation.

Authorized administrators see Admin alongside App (or in the compact navigation menu); other users see no page links. The page remains at /configurations for existing links. Ordinary users do not see it, and the backend denies configuration access unless both managed mode and administrator authorization are present. Personal Settings remains separate from installation administration.

In Administration > Appearance, set an optional App name (up to 80 characters) and Tagline (up to 300 characters), then Save changes. A custom name replaces Data Formulator in the landing heading, navigation, and browser title. Long headings use smaller text and wrap. Clearing a field restores its default; the default tagline follows the selected UI language. These plain-text values are saved in installation configuration, not environment variables, and are visible to all users. Other open clients pick up changes on reload.

In single-user localhost identity mode, the local owner is the administrator. For hosted installations, configure an authentication provider and list verified identities in DF_ADMIN_IDENTITIES, for example user:<verified-subject> (comma separated). Anonymous browser identities cannot administer the installation. An anonymous demo can be provisioned locally before hosting, or use authenticated administrators alongside anonymous visitors. Remote shared-connection saves also require CREDENTIAL_VAULT_KEY.

For Azure App Service, configure administrators by full sign-in address at deployment:

DF_MANAGED=true
AUTH_PROVIDER=azure_easyauth
ALLOW_ANONYMOUS=false
DF_ADMIN_EMAILS=alice@example.com,bob@example.com

At runtime, the backend compares the trusted X-MS-CLIENT-PRINCIPAL-NAME provided by EasyAuth against this comma-separated allowlist, ignoring case and surrounding whitespace. Use the actual sign-in address, which can differ from a mailbox alias or a guest user's home address. Missing addresses, short aliases, display names, and wildcard patterns do not grant access. No directory lookup, Graph permissions, or synchronization with Azure owners/roles is involved. Email authorization currently supports Azure EasyAuth only; other providers keep using DF_ADMIN_IDENTITIES. Workspace and credential identity remain based on the verified object ID, not email. Admin access follows the address if reassigned, so maintain the list when users leave or change addresses.

Alternatively, DF_ADMIN_IDENTITIES=user:<principal-id> matches the trusted X-MS-CLIENT-PRINCIPAL-ID (the user's object ID for Entra). If both lists are configured, matching either grants access; remove old ID entries when switching to email-only administration. Azure subscription/resource ownership does not automatically grant application administrator access. Without an allowlisted authenticated identity or sign-in address, no hosted user is an application administrator. Enable App Service Authentication and prevent direct access that bypasses its trusted-header boundary. Do not expose a deployment in single-user localhost identity mode through an unauthenticated proxy.

DISABLE_DATA_CONNECTORS=true / --disable-data-connectors force shared-only connector access. DISABLE_CUSTOM_MODELS=true / --disable-custom-models force shared-only model access. These deployment settings override saved configuration, including previously saved false values. Administration disables the policy controls, and the configuration API rejects attempts to set the corresponding restriction to false, including JSON edits. Administrators can still manage shared resources; changing a deployment lock requires changing the deployment environment or startup flags and restarting the server.

Disabling an individual shared model or connector also blocks subsequent API and agent lookups by ID, including cached connector loaders. Administration retains access to inspect, test, and re-enable disabled resources. Already-running calls are not cancelled by a configuration change.

Managed-mode startup checks warn about missing administrator access, unsupported email authentication, and detectable hosted use of local-owner identity. They do not replace correct proxy/authentication configuration. Successful configuration saves log the verified actor ID, revision, and changed top-level sections without configuration values or credentials; retain these logs under your audit policy.

A fresh managed installation defaults to administrator-provided models and connections. These are editable defaults: Administration can permit user-created resources or restrict model endpoints. Explicit deployment restrictions remain locked. Existing saved configurations keep their policies when managed mode is enabled or disabled; turning it off hides administration, not policy enforcement. Legacy configurations trigger a startup notice explaining how to enable access.

--disable-database / DISABLE_DATABASE=true is deprecated. It still selects managed mode plus its legacy demo restrictions and ephemeral workspace behavior, and turns the credential vault off unless CREDENTIAL_VAULT is set (shared connections that store secrets need CREDENTIAL_VAULT=local). For new deployments, use --managed with explicit authentication, storage, and policy settings.

Profile 1: Single-User Local (default)

A personal instance running on localhost. No login required, full feature access.

# Everything uses defaults — just run it:
data_formulator

# Or equivalently:
data_formulator \
  --workspace-backend local \
  --sandbox local

Setting	Value	Why
AUTH_PROVIDER	(unset)	Single user, no login needed
WORKSPACE_BACKEND	local	Persist workspaces to ~/.data_formulator/
DISABLE_DATA_CONNECTORS	false	Full access to MySQL, PostgreSQL, etc.
DISABLE_CUSTOM_MODELS	false	User can add any LLM endpoint
DISABLE_DISPLAY_KEYS	false	User can see/manage their own API keys
Credential vault	auto-enabled	Remembers DB credentials across restarts
Identity	local:<os_username>	Fixed, OS-derived — survives localStorage clear

Security notes: In single-user localhost mode, the server ignores the X-Identity-Id header entirely and uses a fixed identity derived from the OS username (e.g., local:alice). This means vault credentials and workspaces are tied to your OS account, not a random browser UUID — clearing localStorage won't orphan your data.

Single-user mode applies only when the server listens on a loopback address (--host 127.0.0.1, the CLI default). A server on --host 0.0.0.0, or under a WSGI server such as gunicorn without HOST=127.0.0.1, uses anonymous multi-user mode, because anyone who can reach it would otherwise act as you. DF_SINGLE_USER=1 keeps single-user mode where access is restricted another way; docker-compose.yml sets it and publishes the port on 127.0.0.1 only. Data-loader plugins (*_data_loader.py in ~/.data_formulator/plugins) load only in single-user mode, unless DF_ALLOW_PLUGINS=1.

Profile 2: Multi-User Anonymous (demo / public hosting)

A shared server (e.g., for demos, workshops, public access). No login, short-lived server-side workspaces, and no sensitive features.

data_formulator \
  --managed \
  --workspace-backend ephemeral \
  --disable-data-connectors \
  --disable-custom-models \
  --disable-display-keys

Legacy shortcut: --disable-database (or DISABLE_DATABASE=true) retains this preset but is deprecated.

Or via environment variables:

DF_MANAGED=true
WORKSPACE_BACKEND=ephemeral
# Ephemeral retention only; local mode is durable and ignores these settings.
EPHEMERAL_WORKSPACE_TTL_HOURS=24
EPHEMERAL_WORKSPACE_MAX_BYTES=10737418240
EPHEMERAL_WORKSPACE_CLEANUP_INTERVAL_SECONDS=1800
DISABLE_DATA_CONNECTORS=true
DISABLE_CUSTOM_MODELS=true
DISABLE_DISPLAY_KEYS=true
# Pre-configure the LLM models users can access:
OPENAI_API_KEY=sk-...
OPENAI_MODELS=gpt-4.1

Each model connection can optionally specify Small Model on the same endpoint, using the same credentials. An empty Small Model uses Model instead. Test and save succeeds only when both distinct model names pass; identical names are tested once. For environment-managed connections, set {PROVIDER}_SMALL_MODEL, for example OPENAI_SMALL_MODEL=gpt-4.1-mini. It applies to every Model configured for that provider. Backend callers explicitly opt in with get_client(config, use_small_model=True). Workspace naming, starter questions, Smart Sort, and code explanation use Small Model (or Model when Small Model is not configured). Other agent tasks continue using Model, including the analyst and workflow agents, semantic annotation, chart restyling, and knowledge distillation.

Setting	Value	Why
AUTH_PROVIDER	(unset)	Anonymous access for demos
WORKSPACE_BACKEND	ephemeral	Temporary server-local workspaces with TTL/LRU cleanup
DISABLE_DATA_CONNECTORS	true	Critical — allows only administrator-configured sources; blocks personal connectors
DISABLE_CUSTOM_MODELS	true	Prevents users from adding arbitrary LLM endpoints (SSRF risk)
DISABLE_DISPLAY_KEYS	true	Hides server-configured API keys from UI
Credential vault	Available	Protects administrator-configured connection credentials
Identity	anonymous (browser:<uuid>)	Isolates temporary workspaces by browser identity

Retention notes: Ephemeral workspaces may disappear after inactivity or when the configured byte cap is reached. The browser keeps only a row-free recovery snapshot for read-only viewing. Use WORKSPACE_BACKEND=local for durable workspaces; ephemeral TTL/LRU cleanup never scans or deletes local-mode workspaces.

Security notes: Disable user-created connectors and custom models for anonymous deployments. Administrator-configured sources remain available, so publish only sources whose data may be shared with every app user. Browser identities are client-provided and are suitable for isolating disposable demo workspaces, not for protecting durable credentials or sensitive server-side state.

Profile 3: Multi-User Authenticated (team)

A shared server with SSO login. Full features, proper identity isolation.

data_formulator \
  --managed \
  --workspace-backend azure_blob \
  --disable-display-keys

DF_MANAGED=true
DF_ADMIN_IDENTITIES=user:<verified-admin-subject>
AUTH_PROVIDER=oidc
OIDC_ISSUER_URL=https://your-idp.example.com/realms/main
OIDC_CLIENT_ID=data-formulator
ALLOW_ANONYMOUS=false
WORKSPACE_BACKEND=azure_blob
AZURE_BLOB_ACCOUNT_URL=https://<account>.blob.core.windows.net
DISABLE_DISPLAY_KEYS=true
DISABLE_CUSTOM_MODELS=true
FLASK_SECRET_KEY=<generate-with-secrets-token-hex-32>

Setting	Value	Why
AUTH_PROVIDER	oidc / github / azure_easyauth	Verified identity from SSO
ALLOW_ANONYMOUS	false	Login required — no anonymous fallback
WORKSPACE_BACKEND	azure_blob or local	Persistent per-user workspaces
DISABLE_DATA_CONNECTORS	false	Not deployment-locked; Administration controls whether user-created connections are permitted
DISABLE_CUSTOM_MODELS	true	Users only use server-configured models
DISABLE_DISPLAY_KEYS	true	Hide server keys; users add their own
FLASK_SECRET_KEY	set explicitly	Required for stable sessions across server restarts
Credential vault	auto-enabled	DB credentials scoped to verified user:<id>
Identity	user:<sub> from auth provider	Server-verified, cannot be spoofed

Security notes: With an auth provider, get_identity_id() returns user:<verified_id> from the IdP token — the X-Identity-Id header is ignored entirely. Workspaces, vault credentials, and DB connections are all scoped to the verified identity. Set ALLOW_ANONYMOUS=false to prevent unauthenticated access.

Profile Comparison

Feature	Profile 1 (Local)	Profile 2 (Demo)	Profile 3 (Team)
Login required	No	No	Yes
Data connectors (DB)	Yes	Administrator-configured only	Administrator policy
Custom LLM endpoints	Yes	No	Operator choice
Credential vault	Yes	Yes (configured sources/models)	Yes
Workspace persistence	Local disk	Ephemeral server storage	Cloud / disk
Identity	local:<os_user> (fixed)	browser:<uuid> (client)	user:<sub> (SSO)

CLI Flags Reference (complete)

Flag	Env var	Default	Description
--managed	DF_MANAGED	false	Enable managed resources and administrator-only Administration page; independent of auth, storage, and sandbox
—	DF_ADMIN_IDENTITIES	(unset)	Comma-separated verified user:<subject> identities allowed to administer a managed installation
—	DF_ADMIN_EMAILS	(unset)	Comma-separated full EasyAuth sign-in addresses allowed to administer a managed installation; case-insensitive exact matches
--workspace-backend	WORKSPACE_BACKEND	local	local, azure_blob, or ephemeral
--sandbox	SANDBOX	local	Code execution backend: local or docker
--disable-database	DISABLE_DATABASE	false	Deprecated demo preset: managed mode + ephemeral + configured connectors only + no custom models + hide keys + credential vault off (unless CREDENTIAL_VAULT is set)
--disable-display-keys	DISABLE_DISPLAY_KEYS	false	Hide API keys in frontend UI
--disable-data-connectors	DISABLE_DATA_CONNECTORS	false	Allow configured sources only; block personal connector creation and use
--disable-custom-models	DISABLE_CUSTOM_MODELS	false	Prevent users from adding custom LLM endpoints
--max-display-rows	MAX_DISPLAY_ROWS	10000	Max rows sent to frontend
--data-dir	DATA_FORMULATOR_HOME	~/.data_formulator	Data directory
--host	HOST	127.0.0.1	Network interface to bind. Only a loopback address selects single-user local mode; under gunicorn, set HOST=127.0.0.1 for that
	DF_SINGLE_USER	unset	1 keeps single-user local mode on a non-loopback address. Only for servers no one else can reach (e.g. a container port published on 127.0.0.1)
	DF_ALLOW_PLUGINS	unset	1 loads data-loader plugins outside single-user local mode. The plugin directory must be trusted
-p, --port	—	5567	Port number
--dev	DEV_MODE	false	Development mode (no auto-open browser)
—	AUTH_PROVIDER	(unset)	oidc, github, azure_easyauth, or unset for anonymous
—	ALLOW_ANONYMOUS	true	Allow unauthenticated access when auth provider is set
—	DF_ALLOWED_API_BASES	(unset, all allowed)	Comma-separated URL globs for LLM endpoint allowlist
—	FLASK_SECRET_KEY	auto-generated	Session signing key (set explicitly for production)
--azure-blob-connection-string	AZURE_BLOB_CONNECTION_STRING	—	Azure Blob shared-key connection string
--azure-blob-account-url	AZURE_BLOB_ACCOUNT_URL	—	Azure Blob account URL for Entra ID auth
--azure-blob-container	AZURE_BLOB_CONTAINER	data-formulator	Azure Blob container name

Configured-Only Models

In Administration > Models, enable Disable user-created models and save. This persists disable_user_models and restricts users to administrator- configured models, including blocking use of previously saved personal models and creation of personal account connections. Administrator model setup remains available. DISABLE_CUSTOM_MODELS=true (also included in DISABLE_DATABASE) enforces this restriction and locks the checkbox. Turning off the saved setting restores personal models unless a deployment flag still restricts them. The endpoint URL allowlist is independent and continues to apply when personal models are permitted.

Configured-Only Data Sources

In Administration > Data Sources, enable Disable user-created connections and save. This persists disable_user_connectors in the installation configuration. Users can still connect to and browse administrator sources from the configuration page, connectors.yaml, or DF_SOURCES__*; personal connections (including previously saved ones) cannot be created or used. Disabling the setting restores access to personal definitions without deleting them.

DISABLE_DATA_CONNECTORS=true / --disable-data-connectors enforces the same policy and locks the checkbox. DISABLE_DATABASE includes this restriction as part of its existing deployment preset. These flags no longer disable configured sources or the credential vault. Administrator connection testing and saving remain available.

In restricted mode, configure complete connection parameters on the server; user-supplied parameters cannot replace the configured host, URL, path, or credentials. Discovery requires a configured connector ID. Agent connection-creation tools and local terminal access are also disabled. This is a connector policy, not a network sandbox: uploads, other application capabilities, and database permissions need their own controls. Configure read-only database credentials where appropriate. Configured sources are shared with app users, so do not publish data that those users must not access.

Shared Connection Settings

Application Configuration stores model and connector settings inline under overrides.connections.models and overrides.connections.connectors. Model entries contain the provider, model name, endpoint URL, and authentication mode; connector entries contain the loader type, display name, and non-sensitive parameters. These settings are not stored in separate workflow-style files.

Each entry has an internal credential_ref linking it to encrypted credentials. The referenced permanent vault record contains secrets, not the full connection definition. API keys, passwords, and loader-declared sensitive parameters are never written to configuration JSON. Connection tests use temporary encrypted staging records; saving promotes their credentials and writes the readable settings. API changes to connection settings require a fresh connection test. Environment-provided connections remain managed by the deployment.

Legacy bare vault references still load. The configuration view expands them into readable settings; the next save persists the inline form and migrates credentials without changing them. The installation's credential vault and encryption key are still needed when moving or restoring the configuration.

Shared Workflow Files

Custom workflows saved through Application Configuration are stored as separate YAML files in workflows/ next to configuration.json in the installation data directory. Configuration holds references, for example:

{
  "workflows": {
    "server/team-review.yaml": {
      "file": "workflows/team-review.yaml",
      "enabled": true
    }
  }
}

This is the workflows section inside overrides. Administrators can place YAML files in that directory and reference them directly. Only simple .yaml filenames are accepted; absolute paths, traversal, and symlinks are rejected. Bundled defaults retain their demo/<filename>.yaml IDs and use "file": "builtin:<filename>.yaml" when saved without content changes. Editing a built-in creates a separate custom file without modifying the bundled original.

The editor continues to load and edit YAML. Saving edited content creates a new uniquely named file and updates the reference, preserving the previous file if the configuration save fails. Removing a reference or resetting an override does not delete files. Old unreferenced versions can be removed manually. Existing inline content remains readable and migrates to file references on the next configuration save. Personal workspace workflow storage is unchanged.

Conversational Workflow Authoring

In local or managed mode, ask the main chat to create a workflow from the current analysis. Managed deployments (including the legacy DISABLE_DATABASE=true preset) support workflow authoring, personal workflow libraries, and execution for the current application identity; application administrator access is not required. Libraries remain user-scoped and run checkpoints remain workspace-scoped. Existing model, connector, and execution-sandbox policies still apply. Terminal commands remain restricted to single-user local mode; managed mode does not enable host-shell access or provide additional sandbox isolation. The Define a workflow shortcut in the Workflows panel submits a guidance prompt to that same chat without changing its conversation focus. There is no separate authoring dialog. The analyst uses the current conversation and data context, asks clarification questions when needed, and calls propose_workflow to publish a validated definition in the chat rather than a Markdown file. The proposal action does not save files or execute the workflow.

propose_workflow accepts a structured definition object and a short summary. The canonical JSON Schema lives in workflows/instances.py; the skill registry embeds it in the tool schema, and the proposal handler validates the object before serializing YAML for display and storage. YAML imports use the same contract, and adapt_plan reuses its step schema. Unknown definition, parameter, step, and checker fields are rejected; source mappings remain open descriptive guidance. New proposals require step descriptions, while older saved steps without descriptions remain valid. Structural validation is supplemented by parameter-value, unique-ID, and transition checks. Date interpretation and analytical correctness still require task-specific verification; schema validity alone does not establish either.

Save to workspace writes a .workflow.yaml file visible under Workspace workflows. Existing files require their current content hash to be overwritten. Run opens the usual setup form and can execute the reviewed definition without saving it first. These actions are independent. Proposals are persisted with ordinary chat turns, and their complete YAML is included in focused-thread context for follow-up revisions. They are not entries in the shared workflow library.

New chat-authored definitions require version: 1, name, overview, deliverables, and concrete ordered steps. Each step identifies its operation, inputs, expected results, and relevant verification conditions. Optional prompt provides cross-step constraints and adaptation rules; it does not replace the procedure. source and parameters support fixed inputs, parameterized inputs, and mixtures. propose_workflow returns a repair request for step-free definitions. Older saved definitions without steps remain runnable through an initial planning phase; newly authored definitions seed the run with their concrete steps.

Each run stores an independent definition snapshot, mutable plan.steps, and execution state (progress, checks, evidence, outputs, and history). Adapting a run's plan never rewrites its definition or the saved YAML. Existing checkpoints are migrated when resumed. Workflow authoring belongs to the main analyst's configure skill; the execution agent cannot call propose_workflow.

Configuring Data Formulator in Chat

The analyst's gated configure skill (analyst/skills/configure/) sets up the application for the user: connections, workflows, workflow schedules, sessions, and (local app only) MCP servers. Read-only tools inspect the current setup (list_connectors, describe_connector, read_connector_form, list_workflows, list_schedules, list_sessions, list_mcp_servers). Each committing action publishes one prefilled setup form artifact: propose_connection / update_connector_form (connector form), propose_schedule (schedule form), and propose_session_changes (a session panel whose cards each offer rename, open in a new tab, and delete; the agent only lists sessions and suggests names, and deletion is always the user's confirmed action). propose_workflow publishes the existing workflow proposal artifact.

All forms share one interact event shape (form.kind, title, response, auto_submit, and a body keyed by the kind) and are submitted by the frontend through the same APIs as the corresponding dialogs, so credentials, permissions, and session state follow the manual path. Setup tools act for the identity the app resolves for the request (get_identity_id), which must own the active workspace. Review is the default. With user_review_needed: false, a schedule or session form submits itself once when the live agent stream creates it, only if it is complete and not elevated; schedules that auto-approve or publish always wait for review. The request is never persisted, so reloaded or imported sessions cannot replay it. Connections always wait for the user's Connect, because connecting opens a server-side network connection to agent-supplied hosts. Workflow runs keep propose_connection but cannot author workflows or manage schedules and sessions. See design-docs/56-configure-skill.md.

Workflow Setup

Workflows can declare optional top-level parameters. Clicking Run opens a setup form before creating a session or calling the agent. Every workflow also accepts optional additional instructions, including workflows without parameters.

parameters:
  - name: symbol
    label: Stock symbol
    type: text
    default: MSFT
    required: true
  - name: period
    label: Review period
    type: select
    options: [Latest month, Latest year]
    default: Latest month
    allow_custom: true
    description: Relative to the latest available data.

Supported types are text (the default), number, boolean, and select. Names must be unique identifiers; labels are required. description, default, and required are optional. Select fields require unique string options; allow_custom: true permits a typed alternative. An unchecked boolean is a valid false value, including for required fields. There are at most 20 parameters, 50 options per select, 4,000 characters per text value, and 8,000 characters of additional instructions. Avoid requesting passwords or other secrets in setup.

New-run requests accept setup: {parameters: {...}, instructions: "..."}. The server validates values against the current workflow, resolves missing defaults, and saves the confirmed setup separately from the workflow snapshot. The agent receives it as user guidance, with precedence over workflow defaults, not as code substitution or additional authorization. Workflow instructions should explain how each parameter affects the task and label fallback values as defaults. Source constraints, data verification, and tool approvals still apply.

Setup is immutable on resume; later changes use normal workflow steering. Saved run state includes the initial setup. No setup agent call is made: a future assisted setup step can supply the same validated payload without changing the execution contract.

Security Considerations for Production Deployment

⚠️ IMPORTANT SECURITY WARNING FOR PRODUCTION DEPLOYMENT

Identity System

Data Formulator uses a namespaced identity system with three tiers:

• Local mode (127.0.0.1, no auth provider): Identity is local:<os_username>, determined by the server. The X-Identity-Id header is ignored. Vault and workspaces are tied to the OS user.
• Anonymous mode (multi-user, no auth provider): Identity is browser:<uuid> where the UUID is generated in the browser's localStorage. The server trusts the client-provided X-Identity-Id header, but always forces the browser: prefix.
• Authenticated mode (auth provider configured): Identity is user:<verified_id> from the auth provider. The X-Identity-Id header is ignored entirely.

Key security principle: An attacker sending X-Identity-Id: user:alice@... gets browser:alice@... — completely separate from the real user:alice@... that only authenticated Alice can access.

Anonymous spoofing risk: In anonymous mode, if an attacker knows another user's browser UUID, they can impersonate them via the X-Identity-Id header. This is why Profile 2 disables data connectors (no DB credentials to steal) and Profile 3 requires authentication (header is ignored).

Data Storage

Backend	Flag	Storage	Persistence
local (default)	--workspace-backend local	~/.data_formulator/users/<identity>/workspaces/	Server filesystem
azure_blob	--workspace-backend azure_blob	Azure Blob container	Cloud
ephemeral	--workspace-backend ephemeral	Browser IndexedDB (frontend) + temp dirs (backend)	Browser session only

Recommended Security Measures

1. Multi-user anonymous (demos): Use Profile 2 — --workspace-backend ephemeral --disable-data-connectors --disable-custom-models --disable-display-keys (or --disable-database as shortcut)
2. Multi-user authenticated: Use Profile 3 — set AUTH_PROVIDER, ALLOW_ANONYMOUS=false, and DISABLE_CUSTOM_MODELS=true
3. HTTPS: Use a reverse proxy (nginx, Azure App Gateway) with TLS termination
4. FLASK_SECRET_KEY: Set explicitly for production (auto-generated key changes on restart)

Server Migration Checklist

When migrating Data Formulator to a new server (or rebuilding a Docker container), the following secrets and data files must be carried over. Losing any of them will break the corresponding functionality.

Item	Location	What breaks if lost
FLASK_SECRET_KEY	.env (env var)	All user sessions invalidated (SSO login, plugin tokens). Agent code signatures fail verification — saved charts cannot refresh until re-executed.
.vault_key	DATA_FORMULATOR_HOME/.vault_key	credentials.db becomes undecryptable — all stored database passwords / service credentials are lost.
credentials.db	DATA_FORMULATOR_HOME/credentials.db	Same as above — the encrypted credential store itself.
DF_CODE_SIGNING_SECRET	.env (env var, optional)	If set, overrides Flask-derived signing key. Must match the old value or all code signatures break.
.code-signing-key	DATA_FORMULATOR_HOME/.code-signing-key	Code-signing key used when neither DF_CODE_SIGNING_SECRET nor FLASK_SECRET_KEY is set. Saved charts cannot refresh until re-executed.
CREDENTIAL_VAULT_KEY	.env (env var, optional)	If set, overrides .vault_key file. Must match or vault data is unreadable.
CREDENTIAL_VAULT=off	env var (optional)	Disables the vault: no .vault_key or credentials.db is created, credentials last only for the session, and shared connections or account model sign-ins that need stored secrets are unavailable. The default under DISABLE_DATABASE; set CREDENTIAL_VAULT=local to keep the vault there.
DF_SANDBOX_BWRAP	env var (optional)	Path to a bubblewrap binary for the Python sandbox's OS isolation on Linux, when it is neither on PATH nor bundled at py-src/data_formulator/sandbox/bin/bwrap (a git-ignored location that deployment packages fill). The startup log reports the isolation in use: Python sandbox isolation: ....
DF_REQUIRE_OS_SANDBOX=1	env var (optional)	Refuse to run generated code when OS isolation is unavailable, instead of falling back to audit hooks only. Recommended for hosted deployments once the startup log confirms isolation.
DF_SANDBOX_IDLE_SECONDS	env var (optional)	How long an idle Python sandbox worker (a separate ~200 MB interpreter) is kept for reuse. Default 600 in local mode, 120 otherwise; multi-user servers also keep at most two used workers idle per server process.
users/ & workspaces/	DATA_FORMULATOR_HOME/	User workspace data (parquet files, session metadata).
configuration.json	DATA_FORMULATOR_HOME/configuration.json	Installation policies, enabled resources, defaults, and shared-connection vault references.
workflows/	DATA_FORMULATOR_HOME/workflows/	Administrator-published workflow YAML referenced by installation configuration.

Minimum migration steps:

Stop all application workers before taking the backup, and restore before any worker starts. Keep installation configuration, workflow files, credential vault, and encryption keys from the same backup. Blob workspace storage does not back up this installation state. If --data-dir is set, use that directory for installation configuration and workflows. Preserve deployment environment settings too, including admin allowlists and immutable policy flags; securely export these through your deployment platform if they are not stored in .env.

# On the OLD server — back up secrets + data
cp .env                             /backup/.env
cp $DATA_FORMULATOR_HOME/.vault_key /backup/.vault_key
cp $DATA_FORMULATOR_HOME/credentials.db /backup/credentials.db
if [[ -f "$DATA_FORMULATOR_HOME/configuration.json" ]]; then
  cp "$DATA_FORMULATOR_HOME/configuration.json" /backup/configuration.json
fi
if [[ -d "$DATA_FORMULATOR_HOME/workflows" ]]; then
  cp -R "$DATA_FORMULATOR_HOME/workflows" /backup/workflows
fi
# Copy workspace data if using local backend
cp -r $DATA_FORMULATOR_HOME/users   /backup/users
cp -r $DATA_FORMULATOR_HOME/workspaces /backup/workspaces

# On the NEW server — restore before first start
cp /backup/.env .env
cp /backup/.vault_key $DATA_FORMULATOR_HOME/.vault_key
cp /backup/credentials.db $DATA_FORMULATOR_HOME/credentials.db
if [[ -f /backup/configuration.json ]]; then
  cp /backup/configuration.json "$DATA_FORMULATOR_HOME/configuration.json"
fi
if [[ -d /backup/workflows ]]; then
  cp -R /backup/workflows "$DATA_FORMULATOR_HOME/workflows"
fi
cp -r /backup/users $DATA_FORMULATOR_HOME/users
cp -r /backup/workspaces $DATA_FORMULATOR_HOME/workspaces

Tip: If you forgot to back up FLASK_SECRET_KEY and it was auto-generated, there is no way to recover it. Users will need to log in again, and any chart with a cached code signature will need to be re-executed by the Agent.

中文版: 详细的迁移操作指南见 docs/docs-cn/7-server-migration-guide.md。

Authentication Architecture

Data Formulator supports a hybrid identity system with anonymous and authenticated modes. See Deployment Profiles above for which mode to use in each scenario.

Identity Flow

┌─────────────────────────────────────────────────────────────────────┐
│                         Frontend Request                             │
├─────────────────────────────────────────────────────────────────────┤
│  Headers:                                                            │
│    X-Identity-Id: "local:alice" / "browser:550e8400-..." / ...      │
│    Authorization: Bearer <jwt>  (if auth provider configured)        │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Backend Identity Resolution                       │
│                       (auth.py: get_identity_id)                    │
├─────────────────────────────────────────────────────────────────────┤
│  Priority 1: Auth provider (OIDC/GitHub/EasyAuth) → "user:<id>"     │
│  Priority 2: Localhost mode (127.0.0.1)           → "local:<user>"  │
│              (ignores X-Identity-Id header)                          │
│  Priority 3: X-Identity-Id header                 → "browser:<id>"  │
│              (client-provided namespace prefix is IGNORED)           │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      Storage Isolation                               │
├─────────────────────────────────────────────────────────────────────┤
│  "user:alice@example.com"  → alice's workspace (ONLY via auth)       │
│  "local:alice"             → localhost user's workspace (fixed)      │
│  "browser:550e8400-..."    → anonymous user's workspace              │
└─────────────────────────────────────────────────────────────────────┘

Auth Provider Setup

See the AUTH_PROVIDER section in .env.template for configuration details.

Provider	AUTH_PROVIDER	Setup
OIDC / OAuth2	oidc	Set OIDC_ISSUER_URL + OIDC_CLIENT_ID
GitHub	github	Set GITHUB_CLIENT_ID + GITHUB_CLIENT_SECRET
Azure EasyAuth	azure_easyauth	Enable in Azure App Service (no extra env vars)
Anonymous only	(unset)	Default — no login, browser:<uuid> identity

Usage

See the Usage section on the README.md page.