From c13bfa00825291e4b946f1e78aa399be1db118e9 Mon Sep 17 00:00:00 2001 From: Ravi Prasad Date: Wed, 28 Oct 2026 01:38:26 -0500 Subject: [PATCH] pipeline --- .gitea/workflows/pipeline.yaml | 2 +- .gitignore | 4 +++- .opencode/context.md | 39 ++++++++++++++++++++++------------ 3 files changed, 30 insertions(+), 15 deletions(-) diff --git a/.gitea/workflows/pipeline.yaml b/.gitea/workflows/pipeline.yaml index 623dea9..7fab2f0 100644 --- a/.gitea/workflows/pipeline.yaml +++ b/.gitea/workflows/pipeline.yaml @@ -2,7 +2,7 @@ name: NBA Data Pipeline on: schedule: - - cron: '0 6 * * *' + - cron: '0 6 40 * *' workflow_dispatch: jobs: diff --git a/.gitignore b/.gitignore index 8eff1c1..1a4b564 100644 --- a/.gitignore +++ b/.gitignore @@ -196,4 +196,6 @@ AGENTS.md .agents .claude -.env.old \ No newline at end of file +.env.old + +.opencode \ No newline at end of file diff --git a/.opencode/context.md b/.opencode/context.md index 442fd02..ec5663d 100644 --- a/.opencode/context.md +++ b/.opencode/context.md @@ -1,10 +1,13 @@ # Codebase Context — Streamlit_NBA -Updated: 2026-10-27 + +Updated: 2026-10-28 ## Overview + Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visualization platform. It extracts raw game, team, and player box score and shot telemetry from PostgreSQL into DuckDB, transforms and models the data through a 3-layer medallion star schema (staging views → intermediate tables → dimension/incremental fact marts) using dbt-duckdb, and serves analytical insights via a 7-page Streamlit application and an Evidence BI report suite. Production DuckDB database artifacts are synced to and from Cloudflare R2 / S3 object storage with automated CI/CD and daily scheduled ETL builds orchestrated through Gitea Actions. ## Stack + - **Languages**: Python 3.12 (pinned in `pyproject.toml:4`, `Dockerfile:1`), SQL / Jinja (dbt SQL dialect for DuckDB), JavaScript / Markdown (Evidence BI) - **Data Warehouse & Transformation**: dbt (`dbt-duckdb`, dbt Core 1.12.5, dbt adapter 1.11.0 in `pyproject.toml:6`), DuckDB (`duckdb>=1.3.0` in `pyproject.toml:8`), `dbt-labs/dbt_utils` 1.3.1 (`dbt_nba/packages.yml:2-3`) - **Dashboards & Visualization**: Streamlit (`streamlit>=1.45.1` in `pyproject.toml:7`), Plotly (`plotly>=6.1.2` in `pyproject.toml:9`), `streamlit-echarts>=0.4.0` (`pyproject.toml:10`), Evidence BI (`@evidence-dev/evidence: ^40.1.8` in `dbt_nba/reports/package.json:20`) @@ -13,6 +16,7 @@ Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visu - **CI/CD**: Gitea Actions (`.gitea/workflows/pipeline.yaml`, `.gitea/workflows/test.yml`) ## Layout + - `streamlit_app/`: Frontend application code (`streamlit_app/app.py:1-1464` containing all 7 analytical views and navigation). - `dbt_nba/`: Core dbt transformation project: - `models/staging/`: 10 views cleaning raw extracts and deriving dynamic metric thresholds. @@ -24,6 +28,7 @@ Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visu - `scripts/`: Operational automation: - `pipeline.sh`: Full extraction from Postgres, dbt rebuild, and R2 artifact upload. - `extract.sql`: DuckDB script attaching to Postgres and pulling source tables into `main.*`. + - `extract_data.py`: Python script orchestrating SQL extraction from Postgres into DuckDB. - `db_storage.py`: CLI manager for R2 database upload, download, and hash/status checking. - `run_app.sh`: Streamlit launcher with auto-download fallback from R2. - `data/`: Local storage directory for DuckDB database files (`data/DB/dbt_nba.duckdb`) and court assets (`shot_chart.png`, `shot_chart.svg`). @@ -31,23 +36,25 @@ Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visu - `docs/`: Historical documentation, prompts, and tournament records (`docs/init_prompt.md`, `docs/all_nba_cup_games_history.md`). ## Architecture -- **Raw Ingestion Layer**: DuckDB PostgreSQL extension (`scripts/extract.sql:5-26`) attaches to the live PostgreSQL source (`POSTGRES_URL`) and copies 7 tables (`games`, `line_scores`, `player_game_basic_stats`, `player_game_adv_stats`, `player_shot_charts`, `team_game_basic_stats`, `team_game_adv_stats`) into `main` schema. + +- **Raw Ingestion Layer**: DuckDB PostgreSQL extension (`scripts/extract.sql:5-26`, invoked via `scripts/extract_data.py`) attaches to the live PostgreSQL source (`POSTGRES_URL`) and copies 7 tables (`games`, `line_scores`, `player_game_basic_stats`, `player_game_adv_stats`, `player_shot_charts`, `team_game_basic_stats`, `team_game_adv_stats`) into `main` schema. - **Staging Layer (Views)**: `dbt_nba/dbt_project.yml:22-25` materializes models into `main_staging`. Includes percentile breakpoints (`dbt_nba/models/staging/stg_season_thresholds.sql`) and dynamic tiering (`dbt_nba/models/staging/stg_player_game_adv_stats_extended.sql:1-40`). - **Intermediate Layer (Tables)**: `dbt_nba/dbt_project.yml:27-30` materializes into `main_intermediate`. Handles temporal team mapping (`dbt_nba/models/intermediate/int_games_enriched.sql:15-35`) and unification of field goal shots with box-score free throws (`dbt_nba/models/intermediate/int_player_shots_enriched.sql:1-50`). - **Marts Layer (Star Schema)**: `dbt_nba/dbt_project.yml:32-44` materializes into `main_marts`: - 7 Dimensions: `dim_teams`, `dim_players`, `dim_seasons`, `dim_dates`, `dim_arenas`, `dim_shot_zones`, `dim_player_game_archetypes`. - 6 Facts (Incremental): Surrogate keys generated via `dbt_utils.generate_surrogate_key` (`dbt_nba/models/marts/facts/fct_player_game_stats.sql:10-25`), using 30-day lookback windows for idempotency. - **Application & Presentation**: - - `streamlit_app/app.py:70-83`: `get_connection()` establishes persistent read-only connection to `dbt_nba.duckdb`; queries cached via `@st.cache_data(ttl=3600)`. - - Seven dashboard modules: Overview (`app.py:151`), Teams (`app.py:297`), Player Lab (`app.py:493`), Shot Charts (`app.py:769`), Head to Head (`app.py:1052`), Game Trends (`app.py:1231`), Quarter Analysis (`app.py:1364`). + - `streamlit_app/app.py:70-83`: `get_connection()` establishes persistent read-only DuckDB connection using DuckDB's Python API, falling back between local dev path (`dbt_nba/reports/sources/nba/dbt_nba.duckdb`) and data storage target (`data/DB/dbt_nba.duckdb`). + - Evidence BI queries `sources/nba/dbt_nba.duckdb` directly through SQL source models. ## Entrypoints & flows -- **Daily Automated ETL**: - 1. Triggered via Gitea Actions cron `0 6 * * *` (`.gitea/workflows/pipeline.yaml:4-6`). - 2. Runs `scripts/pipeline.sh` (`scripts/pipeline.sh:22-45`). - 3. Pulls Postgres source into `data/DB/dbt_nba.duckdb` via DuckDB CLI (`scripts/extract.sql:18-24`). - 4. Executes `dbt deps` and `dbt build --full-refresh` (`scripts/pipeline.sh:33-34`). - 5. Copies DB to `dbt_nba/reports/sources/nba/dbt_nba.duckdb` for Evidence BI (`scripts/pipeline.sh:37`). + +- **Daily Automated ETL Pipeline** (`.gitea/workflows/pipeline.yaml:26-38`, `scripts/pipeline.sh:1-44`): + 1. Triggered on schedule (`cron: '0 6 * * *'`) or manual dispatch. + 2. Runs `scripts/extract_data.py` executing `scripts/extract.sql` to pull raw Postgres tables into local `data/DB/dbt_nba.duckdb`. + 3. Executes `uv run dbt deps` and `uv run dbt build --full-refresh` (`scripts/pipeline.sh:34`). + 4. Runs 38 data and referential integrity tests in dbt. + 5. Copies refreshed DuckDB file to Evidence BI path (`scripts/pipeline.sh:37`). 6. Uploads the finalized DuckDB database file to Cloudflare R2 bucket via `scripts/db_storage.py upload` (`scripts/pipeline.sh:41`). - **Local Application Startup**: 1. User/container launches `./scripts/run_app.sh` or Docker container entrypoint (`Dockerfile:36`). @@ -55,14 +62,16 @@ Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visu 3. Launches Streamlit application (`streamlit run streamlit_app/app.py`). ## Data & integrations + - **Source Database**: PostgreSQL database (`POSTGRES_URL` in `.env`), read-only extraction. - **Analytical Storage**: DuckDB (`dbt_nba.duckdb`) containing 6 schemas: `main` (raw tables), `main_staging` (views), `main_intermediate` (tables), `main_marts` (star schema), `main_seeds` (reference CSVs), `main_test_results` (audit tables). - **Object Storage**: Cloudflare R2 (or any S3-compatible store: SeaweedFS, MinIO, Hetzner S3) configured via `S3_ENDPOINT_URL`, `S3_BUCKET_NAME`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_REGION`, `S3_DB_KEY` in `.env`. - **Authentication**: Zero authentication on public Streamlit app; API/DB credentials stored in `.env` and Gitea Action secrets. ## Commands + - **Install dependencies**: `uv sync` (Python) / `npm --prefix dbt_nba/reports install` (Evidence BI) -- **Compile Python scripts**: `uv run python -m py_compile streamlit_app/app.py scripts/db_storage.py` +- **Compile Python scripts**: `uv run python -m py_compile streamlit_app/app.py scripts/db_storage.py scripts/extract_data.py` - **Validate dbt models**: `uv run dbt parse --project-dir dbt_nba --profiles-dir dbt_nba` - **Install dbt dependencies**: `uv run dbt deps --project-dir dbt_nba --profiles-dir dbt_nba` - **Run dbt tests**: `uv run dbt test --project-dir dbt_nba --profiles-dir dbt_nba` @@ -75,12 +84,14 @@ Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visu - **Build Docker image**: `docker build -t streamlit-nba .` ## Conventions + - **Team Abbreviation Conforming**: Historical franchise relocations and name changes handled via `dbt_nba/seeds/team_maps.csv` using `start_year`/`end_year` filtering in intermediate SQL models. - **dbt Testing & Severity**: Schema tests configured with `severity: warn` and `store_failures: true` storing failures into `main_test_results` (`dbt_nba/dbt_project.yml:45-50`). -- **Branch & Commit Strategy**: Git branches `main` (production deployment) and `dev` (active development); conventional commit prefixes (`feat:`, `fix:`, `ci:`). Current working branch: `update`. +- **Branch & Commit Strategy**: Git branches `main` (production deployment), `dev` (active development), and working branch `update`; conventional commit prefixes (`feat:`, `fix:`, `ci:`). - **Streamlit Caching**: Data queries decorated with `@st.cache_data(ttl=3600)`, database connection with `@st.cache_resource` (`streamlit_app/app.py:70-83`). ## Testing + - **dbt Schema & Data Tests**: 38 data tests defined across models in `dbt_nba/models/schema.yml` and `_int_models.yml` (testing unique constraints, non-null values, referential integrity). Executed via `uv run dbt test --project-dir dbt_nba --profiles-dir dbt_nba`. - **CI Quality Gate**: `.gitea/workflows/test.yml:11-75` runs on every push and pull request to `main` and `dev`: 1. Syntax & bytecode verification (`python -m py_compile`). @@ -89,12 +100,14 @@ Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visu 4. Live production app endpoint healthcheck (`https://streamlit.turbo-data.com/_stcore/health`). ## Open questions + - **Postgres Source Availability for Local Dev**: Local execution of `scripts/pipeline.sh` requires live `POSTGRES_URL`; developers without Postgres access rely on pre-built DuckDB downloads from Cloudflare R2 via `db_storage.py download`. - **Evidence BI Production Deployment**: Evidence project in `dbt_nba/reports` has build scripts (`npm run build`) and queries local DuckDB, but whether it is independently hosted alongside Streamlit or used strictly for static export is not defined in Gitea workflows. ## Starting points + - **Adding or modifying a Streamlit chart/page**: Start in `streamlit_app/app.py` around lines 150-1464; check available columns in `main_marts` or `main_intermediate`. - **Adding or updating a dbt metric / table**: Start in `dbt_nba/models/`; update corresponding `.yml` test definitions in `dbt_nba/models/schema.yml` or `_int_models.yml`. - **Fixing team or franchise mappings**: Edit `dbt_nba/seeds/team_maps.csv` or `arena_maps.csv`, then run `dbt seed`. -- **Adjusting ETL pipeline or object storage**: Inspect `scripts/pipeline.sh` and `scripts/db_storage.py`. +- **Adjusting ETL pipeline or object storage**: Inspect `scripts/pipeline.sh`, `scripts/extract_data.py`, and `scripts/db_storage.py`. - **Modifying CI/CD workflows**: Inspect `.gitea/workflows/pipeline.yaml` (daily pipeline) and `test.yml` (health check).