nprasad2077/data-formulator
1
0 Fork 0
mirror of https://github.com/microsoft/data-formulator.git synced 2026-11-04 12:01:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
data-formulator/DEVELOPMENT.md
T
Chenglong Wang 229ad57c68 agent unification
2026-08-11 00:54:55 -07:00

27 KiB
Raw Permalink Blame History

Set up a local Data Formulator development environment

How to set up your local machine.

Prerequisites

  • Python >= 3.11
  • Node.js
  • Yarn
  • uv (recommended) or pip

Backend (Python)

Option 1: With uv (recommended)

uv is faster and provides reproducible builds via lockfile.

uv sync                        # Creates .venv and installs all dependencies
uv run data_formulator         # Run app (opens browser automatically)
uv run data_formulator --dev   # Run backend only (for frontend development)

Which command to use:

  • End users / testing the full app: uv run data_formulator - starts server and opens browser to http://localhost:5567
  • Frontend development: uv run data_formulator --dev - starts backend server only, then run yarn start separately for the Vite dev server on http://localhost:5173

Option 2: With pip (fallback)

  • Create a Virtual Environment

    python -m venv venv
source venv/bin/activate  # Unix
# or .\venv\Scripts\activate  # Windows

  • Install Dependencies

    pip install -r requirements.txt

  • Configure environment variables (optional)

    • copy .env.template to .env and fill in your values:
      • API keys: set {PROVIDER}_ENABLED=true, {PROVIDER}_API_KEY=..., and {PROVIDER}_MODELS=... for each LLM provider you want to use. See the LiteLLM setup guide for provider-specific fields.
      • Server settings: DISABLE_DISPLAY_KEYS, SANDBOX, etc.
      • Azure Blob workspace (optional): see Azure Blob Storage Workspace below.
    • this lets Data Formulator automatically load API keys at startup so you don't need to enter them in the UI.

  • Run the app

    # Unix
./local_server.sh

# Windows
.\local_server.bat

# Or directly
data_formulator         # Opens browser automatically
data_formulator --dev   # Backend only (for frontend development)

Frontend (TypeScript)

  • Install NPM packages

    yarn

  • Development mode

    First, start the backend server (in a separate terminal):

    uv run data_formulator --dev   # or ./local_server.sh

    Then, run the frontend in development mode with hot reloading:

    yarn start

    Open http://localhost:5173 to view it in the browser. The page will reload if you make edits. You will also see any lint errors in the console.

Advanced: developing against a local flint-chart

DF depends on the published flint-chart package; by default (FLINT_CHART_LOCAL unset) the bare flint-chart import resolves to that npm package.

For co-development against a local Flint checkout with hot reload, point FLINT_CHART_LOCAL at its source when starting the dev server (or tests):

FLINT_CHART_LOCAL=../flint-chart/packages/flint-js/src yarn start
# tests:
FLINT_CHART_LOCAL=../flint-chart/packages/flint-js/src yarn test

Opt-in, advanced-dev only — leave it unset for normal work and CI, which use the installed package. The alias is wired in vite.config.ts and vitest.config.ts.

Build for Production

  • Build the frontend and then the backend

    Compile the TypeScript files and bundle the project:

    yarn build

    This builds the app for production to the py-src/data_formulator/dist folder.

    Then, build python package:

    # With uv
uv build

# Or with pip
pip install build
python -m build

    This will create a python wheel in the dist/ folder. The name would be data_formulator-<version>-py3-none-any.whl

  • Test the artifact

    You can then install the build result wheel (testing in a virtual environment is recommended):

    # replace <version> with the actual build version. 
pip install dist/data_formulator-<version>-py3-none-any.whl

    Once installed, you can run Data Formulator with:

    data_formulator

    or

    python -m data_formulator

    Open http://localhost:5567 to view it in the browser.

Docker

Docker is the easiest way to run Data Formulator without installing Python or Node.js locally.

Quick start

  1. Copy the environment template and add your API keys:

    cp .env.template .env
# Edit .env and set your OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.

  2. Build and start the container:

    docker compose up --build

  3. Open http://localhost:5567 in your browser.

To stop the container: docker compose down

Workspace data (uploaded files, sessions) is persisted in a Docker volume (data_formulator_home) so it survives container restarts.

Build the image manually

docker build -t data-formulator .
docker run --rm -p 5567:5567 --env-file .env data-formulator

Docker sandbox (SANDBOX=docker) is not supported inside a container

The Docker sandbox backend works by calling docker run -v <host_path>:... to bind-mount temporary workspace directories into child containers. When Data Formulator itself runs in a Docker container those paths refer to the container filesystem, not the host, so Docker daemon cannot mount them and the feature does not work.

Use SANDBOX=docker only when running Data Formulator directly on the host (e.g. with uv run data_formulator --sandbox docker or python -m data_formulator --sandbox docker). When using the Docker image, keep the default SANDBOX=local.

Sandbox

AI-generated Python code runs inside a sandbox to isolate it from the main server process. Two backends are available:

Backend Flag How it works Overhead
local (default) --sandbox local Persistent warm subprocess with pre-imported pandas/numpy/duckdb. Audit hooks block file writes and dangerous operations (subprocess, shutil, etc.). ~1 ms
docker --sandbox docker Each execution runs in a disposable docker run --rm container. Workspace is mounted read-only; output is returned via a bind-mounted parquet file. Memory/CPU/PID limits enforced. ~700 ms 
# Use the default local sandbox
python -m data_formulator

# Use Docker sandbox (requires Docker daemon)
python -m data_formulator --sandbox docker

The Docker sandbox image is built from py-src/data_formulator/sandbox/Dockerfile.sandbox:

docker build -t data-formulator-sandbox -f py-src/data_formulator/sandbox/Dockerfile.sandbox .

Source: py-src/data_formulator/sandbox/

Azure Blob Storage Workspace

By default, workspace data (uploaded files, parquet tables, metadata) is stored on the local filesystem under ~/.data_formulator/workspaces/. For cloud deployments you can switch to Azure Blob Storage so all workspace data lives in a blob container instead.

Quick start (local dev with connection string)

  1. Install extra dependencies:

    pip install azure-storage-blob
# or with uv:
uv pip install azure-storage-blob

  2. Create a storage account & container (one-time setup):

    az storage account create -n <account> -g <resource-group> -l eastus --sku Standard_LRS
az storage container create -n data-formulator --account-name <account>

  3. Get the connection string:

    az storage account show-connection-string -n <account> -g <resource-group> -o tsv

  4. Add to .env:

    WORKSPACE_BACKEND=azure_blob
AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...
# AZURE_BLOB_CONTAINER=data-formulator   # default, change if needed

  5. Run normally:

    uv run data_formulator --dev

    Or pass as CLI flags:

    data_formulator --workspace-backend azure_blob \
  --azure-blob-connection-string "DefaultEndpointsProtocol=https;AccountName=..."

Production setup with Entra ID (no secrets)

In production (Azure App Service, AKS, etc.) you can authenticate the app to blob storage via Managed Identity instead of a connection string. This eliminates secrets entirely.

  1. Install extra dependencies:

    pip install azure-storage-blob azure-identity

  2. Assign a role to the app's Managed Identity:

    # Get the App Service's principal ID
PRINCIPAL_ID=$(az webapp identity show -n <app-name> -g <rg> --query principalId -o tsv)

# Grant it "Storage Blob Data Contributor" on the storage account
az role assignment create \
  --assignee "$PRINCIPAL_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>"

  3. Set environment variables (no secrets needed):

    WORKSPACE_BACKEND=azure_blob
AZURE_BLOB_ACCOUNT_URL=https://<account>.blob.core.windows.net
# AZURE_BLOB_CONTAINER=data-formulator

    The app uses DefaultAzureCredential, which automatically picks up the Managed Identity.

  4. For local development with the same Entra ID path, log in with the Azure CLI:

    az login
# Grant your user the same "Storage Blob Data Contributor" role
az role assignment create \
  --assignee "<your-email@example.com>" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>"

    Then set:

    WORKSPACE_BACKEND=azure_blob
AZURE_BLOB_ACCOUNT_URL=https://<account>.blob.core.windows.net

    DefaultAzureCredential will use your az login session.

Authentication methods summary

Method Env var When to use
Connection string AZURE_BLOB_CONNECTION_STRING Local dev, quick tests
Entra ID (Managed Identity) AZURE_BLOB_ACCOUNT_URL Azure App Service, AKS — no secrets
Entra ID (az login) AZURE_BLOB_ACCOUNT_URL Local dev without secrets
Entra ID (service principal) AZURE_BLOB_ACCOUNT_URL + AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_CLIENT_SECRET CI/CD pipelines

If both AZURE_BLOB_CONNECTION_STRING and AZURE_BLOB_ACCOUNT_URL are set, the connection string takes precedence.

Blob layout

All workspace data is stored under <datalake_root>/<sanitized_identity_id>/ inside the container:

data-formulator/                          ← container
  workspaces/                             ← datalake_root (default)
    browser_550e8400.../                  ← anonymous user workspace
      workspace.yaml
      sales_data.parquet
    user_alice_example_com/               ← authenticated user workspace
      workspace.yaml
      quarterly_report.parquet

CLI flags reference

Flag Env var Default Description
--workspace-backend WORKSPACE_BACKEND local local, azure_blob, or ephemeral
--azure-blob-connection-string AZURE_BLOB_CONNECTION_STRING — Shared-key connection string
--azure-blob-account-url AZURE_BLOB_ACCOUNT_URL — Account URL for Entra ID auth
--azure-blob-container AZURE_BLOB_CONTAINER data-formulator Blob container name

Deployment Profiles

Data Formulator supports three deployment configurations. All defaults are optimized for Profile 1 (single-user local) — you only need to set flags when deploying as multi-user.

Profile 1: Single-User Local (default)

A personal instance running on localhost. No login required, full feature access.

# Everything uses defaults — just run it:
data_formulator

# Or equivalently:
data_formulator \
  --workspace-backend local \
  --sandbox local
Setting Value Why
AUTH_PROVIDER (unset) Single user, no login needed
WORKSPACE_BACKEND local Persist workspaces to ~/.data_formulator/
DISABLE_DATA_CONNECTORS false Full access to MySQL, PostgreSQL, etc.
DISABLE_CUSTOM_MODELS false User can add any LLM endpoint
DISABLE_DISPLAY_KEYS false User can see/manage their own API keys
Credential vault auto-enabled Remembers DB credentials across restarts
Identity local:<os_username> Fixed, OS-derived — survives localStorage clear

Security notes: In single-user localhost mode, the server ignores the X-Identity-Id header entirely and uses a fixed identity derived from the OS username (e.g., local:alice). This means vault credentials and workspaces are tied to your OS account, not a random browser UUID — clearing localStorage won't orphan your data.

Profile 2: Multi-User Anonymous (demo / public hosting)

A shared server (e.g., for demos, workshops, public access). No login, short-lived server-side workspaces, and no sensitive features.

data_formulator \
  --workspace-backend ephemeral \
  --disable-data-connectors \
  --disable-custom-models \
  --disable-display-keys

Shortcut: --disable-database (or DISABLE_DATABASE=true) bundles all of the above into a single flag.

Or via environment variables:

WORKSPACE_BACKEND=ephemeral
# Ephemeral retention only; local mode is durable and ignores these settings.
EPHEMERAL_WORKSPACE_TTL_HOURS=24
EPHEMERAL_WORKSPACE_MAX_BYTES=10737418240
EPHEMERAL_WORKSPACE_CLEANUP_INTERVAL_SECONDS=1800
DISABLE_DATA_CONNECTORS=true
DISABLE_CUSTOM_MODELS=true
DISABLE_DISPLAY_KEYS=true
# Pre-configure the LLM models users can access:
OPENAI_ENABLED=true
OPENAI_API_KEY=sk-...
OPENAI_MODELS=gpt-4.1
Setting Value Why
AUTH_PROVIDER (unset) Anonymous access for demos
WORKSPACE_BACKEND ephemeral Temporary server-local workspaces with TTL/LRU cleanup
DISABLE_DATA_CONNECTORS true Critical — prevents DB credential exposure via identity spoofing
DISABLE_CUSTOM_MODELS true Prevents users from adding arbitrary LLM endpoints (SSRF risk)
DISABLE_DISPLAY_KEYS true Hides server-configured API keys from UI
Credential vault N/A No connectors → no credentials to store
Identity anonymous (browser:<uuid>) Isolates temporary workspaces by browser identity

Retention notes: Ephemeral workspaces may disappear after inactivity or when the configured byte cap is reached. The browser keeps only a row-free recovery snapshot for read-only viewing. Use WORKSPACE_BACKEND=local for durable workspaces; ephemeral TTL/LRU cleanup never scans or deletes local-mode workspaces.

Security notes: Keep data connectors and custom models disabled for anonymous deployments. Browser identities are client-provided and are suitable for isolating disposable demo workspaces, not for protecting durable credentials or sensitive server-side state.

Profile 3: Multi-User Authenticated (enterprise / team)

A shared server with SSO login. Full features, proper identity isolation.

data_formulator \
  --workspace-backend azure_blob \
  --disable-display-keys

AUTH_PROVIDER=oidc
OIDC_ISSUER_URL=https://your-idp.example.com/realms/main
OIDC_CLIENT_ID=data-formulator
ALLOW_ANONYMOUS=false
WORKSPACE_BACKEND=azure_blob
AZURE_BLOB_ACCOUNT_URL=https://<account>.blob.core.windows.net
DISABLE_DISPLAY_KEYS=true
DISABLE_CUSTOM_MODELS=true
FLASK_SECRET_KEY=<generate-with-secrets-token-hex-32>
Setting Value Why
AUTH_PROVIDER oidc / github / azure_easyauth Verified identity from SSO
ALLOW_ANONYMOUS false Login required — no anonymous fallback
WORKSPACE_BACKEND azure_blob or local Persistent per-user workspaces
DISABLE_DATA_CONNECTORS false Safe — identity comes from auth provider, not spoofable
DISABLE_CUSTOM_MODELS true Users only use server-configured models
DISABLE_DISPLAY_KEYS true Hide server keys; users add their own
FLASK_SECRET_KEY set explicitly Required for stable sessions across server restarts
Credential vault auto-enabled DB credentials scoped to verified user:<id>
Identity user:<sub> from auth provider Server-verified, cannot be spoofed

Security notes: With an auth provider, get_identity_id() returns user:<verified_id> from the IdP token — the X-Identity-Id header is ignored entirely. Workspaces, vault credentials, and DB connections are all scoped to the verified identity. Set ALLOW_ANONYMOUS=false to prevent unauthenticated access.

Profile Comparison

Feature Profile 1 (Local) Profile 2 (Demo) Profile 3 (Enterprise)
Login required No No Yes
Data connectors (DB) Yes No Yes
Custom LLM endpoints Yes No Operator choice
Credential vault Yes N/A Yes
Workspace persistence Local disk Browser only Cloud / disk
Identity local:<os_user> (fixed) browser:<uuid> (client) user:<sub> (SSO)

CLI Flags Reference (complete)

Flag Env var Default Description
--workspace-backend WORKSPACE_BACKEND local local, azure_blob, or ephemeral
--sandbox SANDBOX local Code execution backend: local or docker
--disable-database DISABLE_DATABASE false Multi-user anonymous preset: bundles ephemeral + no connectors + no custom models + hide keys
--disable-display-keys DISABLE_DISPLAY_KEYS false Hide API keys in frontend UI
--disable-data-connectors DISABLE_DATA_CONNECTORS false Disable external DB connectors
--disable-custom-models DISABLE_CUSTOM_MODELS false Prevent users from adding custom LLM endpoints
--max-display-rows MAX_DISPLAY_ROWS 10000 Max rows sent to frontend
--data-dir DATA_FORMULATOR_HOME ~/.data_formulator Data directory
--host HOST 127.0.0.1 Network interface to bind
-p, --port — 5567 Port number
--dev DEV_MODE false Development mode (no auto-open browser)
— AUTH_PROVIDER (unset) oidc, github, azure_easyauth, or unset for anonymous
— ALLOW_ANONYMOUS true Allow unauthenticated access when auth provider is set
— DF_ALLOWED_API_BASES (unset, all allowed) Comma-separated URL globs for LLM endpoint allowlist
— FLASK_SECRET_KEY auto-generated Session signing key (set explicitly for production)
--azure-blob-connection-string AZURE_BLOB_CONNECTION_STRING — Azure Blob shared-key connection string
--azure-blob-account-url AZURE_BLOB_ACCOUNT_URL — Azure Blob account URL for Entra ID auth
--azure-blob-container AZURE_BLOB_CONTAINER data-formulator Azure Blob container name

Security Considerations for Production Deployment

⚠️ IMPORTANT SECURITY WARNING FOR PRODUCTION DEPLOYMENT

Identity System

Data Formulator uses a namespaced identity system with three tiers:

  • Local mode (127.0.0.1, no auth provider): Identity is local:<os_username>, determined by the server. The X-Identity-Id header is ignored. Vault and workspaces are tied to the OS user.
  • Anonymous mode (multi-user, no auth provider): Identity is browser:<uuid> where the UUID is generated in the browser's localStorage. The server trusts the client-provided X-Identity-Id header, but always forces the browser: prefix.
  • Authenticated mode (auth provider configured): Identity is user:<verified_id> from the auth provider. The X-Identity-Id header is ignored entirely.

Key security principle: An attacker sending X-Identity-Id: user:alice@... gets browser:alice@... — completely separate from the real user:alice@... that only authenticated Alice can access.

Anonymous spoofing risk: In anonymous mode, if an attacker knows another user's browser UUID, they can impersonate them via the X-Identity-Id header. This is why Profile 2 disables data connectors (no DB credentials to steal) and Profile 3 requires authentication (header is ignored).

Data Storage

Backend Flag Storage Persistence
local (default) --workspace-backend local ~/.data_formulator/users/<identity>/workspaces/ Server filesystem
azure_blob --workspace-backend azure_blob Azure Blob container Cloud
ephemeral --workspace-backend ephemeral Browser IndexedDB (frontend) + temp dirs (backend) Browser session only

Recommended Security Measures

  1. Multi-user anonymous (demos): Use Profile 2 — --workspace-backend ephemeral --disable-data-connectors --disable-custom-models --disable-display-keys (or --disable-database as shortcut)
  2. Multi-user authenticated: Use Profile 3 — set AUTH_PROVIDER, ALLOW_ANONYMOUS=false, and DISABLE_CUSTOM_MODELS=true
  3. HTTPS: Use a reverse proxy (nginx, Azure App Gateway) with TLS termination
  4. FLASK_SECRET_KEY: Set explicitly for production (auto-generated key changes on restart)

Server Migration Checklist

When migrating Data Formulator to a new server (or rebuilding a Docker container), the following secrets and data files must be carried over. Losing any of them will break the corresponding functionality.

Item Location What breaks if lost
FLASK_SECRET_KEY .env (env var) All user sessions invalidated (SSO login, plugin tokens). Agent code signatures fail verification — saved charts cannot refresh until re-executed.
.vault_key DATA_FORMULATOR_HOME/.vault_key credentials.db becomes undecryptable — all stored database passwords / service credentials are lost.
credentials.db DATA_FORMULATOR_HOME/credentials.db Same as above — the encrypted credential store itself.
DF_CODE_SIGNING_SECRET .env (env var, optional) If set, overrides Flask-derived signing key. Must match the old value or all code signatures break.
CREDENTIAL_VAULT_KEY .env (env var, optional) If set, overrides .vault_key file. Must match or vault data is unreadable.
users/ & workspaces/ DATA_FORMULATOR_HOME/ User workspace data (parquet files, session metadata).

Minimum migration steps:

# On the OLD server — back up secrets + data
cp .env                             /backup/.env
cp $DATA_FORMULATOR_HOME/.vault_key /backup/.vault_key
cp $DATA_FORMULATOR_HOME/credentials.db /backup/credentials.db
# Copy workspace data if using local backend
cp -r $DATA_FORMULATOR_HOME/users   /backup/users
cp -r $DATA_FORMULATOR_HOME/workspaces /backup/workspaces

# On the NEW server — restore before first start
cp /backup/.env .env
cp /backup/.vault_key $DATA_FORMULATOR_HOME/.vault_key
cp /backup/credentials.db $DATA_FORMULATOR_HOME/credentials.db
cp -r /backup/users $DATA_FORMULATOR_HOME/users
cp -r /backup/workspaces $DATA_FORMULATOR_HOME/workspaces

Tip: If you forgot to back up FLASK_SECRET_KEY and it was auto-generated, there is no way to recover it. Users will need to log in again, and any chart with a cached code signature will need to be re-executed by the Agent.

中文版: 详细的迁移操作指南见 docs/docs-cn/7-server-migration-guide.md。

Authentication Architecture

Data Formulator supports a hybrid identity system with anonymous and authenticated modes. See Deployment Profiles above for which mode to use in each scenario.

Identity Flow

┌─────────────────────────────────────────────────────────────────────┐
│                         Frontend Request                             │
├─────────────────────────────────────────────────────────────────────┤
│  Headers:                                                            │
│    X-Identity-Id: "local:alice" / "browser:550e8400-..." / ...      │
│    Authorization: Bearer <jwt>  (if auth provider configured)        │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Backend Identity Resolution                       │
│                       (auth.py: get_identity_id)                    │
├─────────────────────────────────────────────────────────────────────┤
│  Priority 1: Auth provider (OIDC/GitHub/EasyAuth) → "user:<id>"     │
│  Priority 2: Localhost mode (127.0.0.1)           → "local:<user>"  │
│              (ignores X-Identity-Id header)                          │
│  Priority 3: X-Identity-Id header                 → "browser:<id>"  │
│              (client-provided namespace prefix is IGNORED)           │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      Storage Isolation                               │
├─────────────────────────────────────────────────────────────────────┤
│  "user:alice@example.com"  → alice's workspace (ONLY via auth)       │
│  "local:alice"             → localhost user's workspace (fixed)      │
│  "browser:550e8400-..."    → anonymous user's workspace              │
└─────────────────────────────────────────────────────────────────────┘

Auth Provider Setup

See the AUTH_PROVIDER section in .env.template for configuration details.

Provider AUTH_PROVIDER Setup
OIDC / OAuth2 oidc Set OIDC_ISSUER_URL + OIDC_CLIENT_ID
GitHub github Set GITHUB_CLIENT_ID + GITHUB_CLIENT_SECRET
Azure EasyAuth azure_easyauth Enable in Azure App Service (no extra env vars)
Anonymous only (unset) Default — no login, browser:<uuid> identity

Usage

See the Usage section on the README.md page.

Reference in new issue View Git Blame Copy Permalink