The registry's release-tag filter never ran in CI: breeze ran
extract_metadata.py inside a container with no .git, so git found no tags
and the script fell back to every version in provider.yaml. A live build
on 2026-09-29 published common-dataquality 0.1.0, which was never released.
extract_metadata.py now runs on the host, where the tags are visible, and
skips providers with state: not-ready. A live registry build fails if the
checkout has no providers-* tags; local runs without tags still warn and
keep every version.
The Stats page's Quick Links card listed only Browse All Providers and
Explore by Category. It now also links /durable/, which was reachable
only from the bottom of Explore and from provider pages rebuilt since it
shipped.
The Integrations stat counted unique connection types, which undercounts
the services providers integrate with because many integrations share
one connection type (the Amazon provider declares 53 integrations and 6
connection types). Count the integrations each provider.yaml declares
instead.
The provider pages show Durable and Deferrable badges and filters, but
finding every such operator meant opening each provider in turn. The new
/durable/ page lists all of them in one place, grouped by provider, with
filters for capability, provider, module type and a name search. It reads
the supports_durable_execution and supports_deferrable flags already in
modules.json, so it needs no extractor or schema change.
* Update docs for the Python 3.11 baseline
Contributing docs, Breeze docs, Docker stack docs and the user-facing
installation pages used Python 3.10 as the example and default
version. Examples and defaults now use 3.11; historical changelog
entries and ADRs keep their original wording.
* Re-pad the image naming table after dropping Python 3.10
* Fix stale Python 3.10 examples left in the Python 3.11 docs update
Review found commands that still pin a 3.10.x interpreter, which can no
longer install Breeze or Airflow, shell prompts and pytest transcripts
that still show 3.10, and a note that tied the supported Python range to
Airflow 3.2.0 even though the 3.2 to 3.4 release lines still accept 3.10.
Two Breeze examples used options that do not exist. The python-client
release notes presented `breeze --python 3.11` as equivalent to
`UV_PYTHON=3.11`, but the reproducibility guard checks the interpreter
Breeze itself runs on, so only the `UV_PYTHON` form works. The Docker
stack build examples now refer to the default Python version instead of
a hard-coded one, so they stop drifting with every bump.
* Fix remaining Python 3.10 references in docs and image examples
Review found more shell prompts that still show a 3.10 Breeze image.
The air-gapped image example pinned Airflow 2.5.3, which publishes no
3.11 constraints, so its download step fetched a 404 page and failed;
it also used extras that Airflow 3 no longer defines. The contributor
quick start called 3.10 end-of-life before its October 2026 end-of-life
date.
* Update Python version in database setup guide
Updated Python version in setup instructions from 3.8.10 to 3.11.16.
* Fix stale Python 3.10 constraints examples in INSTALL
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The registry ignored the filesystems, asset-uris and remote-logging
sections of provider.yaml, so a provider page never said which URI
schemes the provider handles (for example, that microsoft-azure serves
sharepoint:// and onedrive:// paths through ObjectStoragePath).
Collect one entry per scheme from those three sections, expose it as
uri_schemes in providers.json and per-version metadata, and render a
URI Schemes table on the provider page.
A scheme-by-feature matrix showed a dash wherever a provider registers
no handler, which reads as missing support even when the feature does
not apply (redshift:// is never an object store). List the schemes
under each feature instead, so only what is registered is shown.
* Find providers in the registry by the external services they reach
The names people type into registry search are often the service they want to
talk to, not the Airflow distribution that talks to it -- "openai" or "ollama"
rather than "common-ai". Those names were declared in provider.yaml and rendered
in a table on the provider page, but neither search path read them, so the
queries most likely to lead someone to Common AI returned nothing.
Both search paths now read the same deduped list off the same field, so a query
that works in the global search box also works in the /providers/ filter box.
* Badge the external service that actually matched the search query
The badge is meant to name the service that surfaced a provider, but it took
the first entry the query appeared anywhere inside, so "ai" credited OpenAI
and "llm" credited vLLM. It also compared raw text while the two other search
paths collapse runs of "-_\s" first, so a hyphenated query like "mistral-ai"
matched no service at all and got no badge.
Matching a word start after the same collapse gives all three paths the same
notion of a match. That normalize() now has a third hand-written copy, so the
sync note in each of the three files names the other two.
* Let the ai-ml explore category match a provider's external services
The filter box on /providers/ already matched the services a provider's
connections reach, but the category dropdown beside it did not, so a provider
that gets to OpenAI only through external-services could be found by typing
and not by filtering.
Feeding those names to every category would misfile providers instead. A bare
service name like "Google" collides with the cloud category, whose keywords
are provider distribution ids -- an identity claim, not something a provider
reaches. Only ai-ml keys on integration names, the same axis external services
describe, so it is the one category that opts in and the default stays off.
* Stop badging unrelated services on search hits that matched by name
The service badge next to a search result exists to show which external
service a provider was found through, when that is not obvious from the
provider's own name. It was derived purely from the query string, with no
check for whether the query already matched the provider's own identity:
searching "ai" surfaced Common AI on its own name and then badged it
"Mistral AI", and searching "a" badged it "Anthropic" — services the query
had nothing to do with.
Skipping the service lookup whenever the provider's own name or id already
explains the hit removes those misleading badges without losing any of the
genuine ones.
* Replace the ai-ml opt-in comment's false rule with a measured fact
The comment explaining why only ai-ml opts into external-service matching
is the only written rationale for that flag anywhere, and it claimed every
other category's keywords are this-distribution-only identity claims. That
is false in both directions: dingtalk, dataflow, couchbase and twilio are
not directory ids either, while 7 of ai-ml's own 13 keywords are.
What replaces it is falsifiable: turning the flag on for every category,
ai-ml included, changes exactly one other — cloud would gain Common AI via
`google` matching "Google" and "Google Vertex AI". Common AI is not a cloud
platform, so that is the false positive the opt-in stays scoped to avoid.
* Document the external-service opt-in in the category matcher's README
The registry README is the only place the explore-category matcher is
explained, and it still described keywords as matching a provider's id and
declared integration names only. Since a category can now opt in to also
matching the provider's declared external services, anyone reading the
README would have had no way to know the option exists, what it defaults
to, or where it is declared.
* Keep a provider out of the search text for services another provider owns
Indexing every external service name a provider reaches gives short records
an advantage that BM25's length normalisation turns into a ranking win over
the provider that actually implements the service: Common AI's record reaches
for "AWS Bedrock" and "Google Vertex AI" in a fraction of the words Amazon
and Google spend describing their own.
Compound service names whose words all belong to another provider's own
integrations are now left out of the indexed text, while the badge and the
/providers/ filter still see the full list. Bare vendor names stay indexed:
dropping them on an exact-name match would have cost the discovery this PR
adds, since a query like "ollama" or "groq" has no other provider to find.
The source tarball is what a consumer of an Apache Airflow release reads for
licensing, and its root LICENSE was bare Apache-2.0 with no subcomponents
section. Since the provider registry landed it has also shipped two web fonts
that are not ours and not Apache-2.0: JetBrains Mono and Plus Jakarta Sans,
both under the SIL Open Font License 1.1, vendored as woff2 and loaded through
@font-face. The OFL requires the copyright notice and licence to travel with
the font, so redistributing them with no notice anywhere in the tree left the
tarball own licensing incomplete.
The licence texts are the ones published by each upstream project, at the
versions recorded in the woff2 name tables. They sit next to the component
they cover, matching how airflow-core and the fab and google providers keep
their third-party texts, and the root LICENSE gains the subcomponents section
that airflow-core/LICENSE already carries so the pointer is where a reader
looks first.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_012zrnHJHPchB83FtwrYRf5q
* Match providers to Explore categories by integration name, not just id
The Explore page and each provider's category chips only matched a
provider to a category by substring-matching the category's keywords
against the provider's id. Providers whose id doesn't happen to
contain a relevant keyword never showed up, even when their
provider.yaml declares integrations that obviously belong there. For
example, common-ai (wraps pydantic-ai) ships hooks named "Pydantic AI",
"MCP Server", "LangChain", and "LlamaIndex", each tagged ai, but none of
those words are substrings of the id "common-ai", so it never appeared
under AI & Machine Learning or in a search for "LangChain"/"MCP".
Match against each provider's declared integration names (already
extracted as provider.categories[].name) in addition to its id, and
add the ai-ml category keywords needed for common-ai's integrations.
Also scan connection_types[].external_integrations when present so a
future field populating that data benefits automatically.
* Add more provider explore-category keywords found via integration-name audit
Auditing all provider.yaml integration names/tags against each Explore
category's keyword list turned up further clear misses of the same
id-substring bug fixed for common-ai: MSSQL, OpenSearch, IBM Cloudant,
and ArangoDB never showed up under Databases; Exasol and ClickHouse
never showed up under Data Warehouses; DingTalk and Apprise never
showed up under Messaging & Notifications (same tags: [service] class
as the Slack/Discord/Telegram/PagerDuty/Opsgenie entries already
there); and gRPC never showed up under Workflow & Orchestration despite
sharing the same tags: [protocol] as ssh/http/ftp already listed.
Deliberately left out: Apache Kafka and IBM MQ, which also lack any
matching keyword. Both are enterprise message-broker/queueing
middleware (tags: [apache]/[software]) rather than the
notification/alerting channels the Messaging & Notifications category
is themed around and its description names (Slack, email, SMS,
alerting services), so adding them is a judgment call left for a
maintainer rather than an obvious fix.
* Registry: Tighten Explore category keyword matching
The keyword matcher compared in both directions, which was safe while the
left-hand side was always a provider slug. Now that it also runs against
free-text integration names, a long keyword could swallow short ones the
tree already has ("Tpt", "Ttu", "YDB", "Bteq"). Matching forward only
removes that hazard without changing any provider's category membership.
Keywords and integration names also spell separators differently, which
forced duplicate entries such as 'pydantic ai' alongside 'pydantic-ai'.
Collapsing separator runs to a single space lets one keyword cover both
spellings. Collapsing rather than stripping matters: stripping would turn
"Microsoft Power BI" into a string containing "ftp".
The connection-type branch read a field that cannot exist, since the
contract that validates providers.json forbids unknown keys and does not
declare it.
* Registry: Let the providers search box find integration names
Matching providers to Explore categories by integration name fixed the
category dropdown but not the search box beside it, which still compared
the provider's display name alone. Typing the name of an integration a
provider declares — "langchain", "opensearch" — found nothing, which is
the case people actually reach for the search box to solve.
* Registry: Include integration names in the search index
The site has two independent searches: the filter box on /providers/, and
the global modal reached from the header and the home page hero. The modal
is backed by Pagefind, whose provider records are built from custom
content rather than page HTML, so making integration names available as a
data attribute did nothing for it.
That left the more prominent of the two searches unable to find a provider
by an integration it declares — "langchain" returned nothing even though
Common AI lists LangChain.
* Registry: Match the providers search box on separator variants
The build-time category matcher collapses runs of "-_ " to a single space
before comparing, but the client-side filter box compared raw strings, so
the two halves of the same feature disagreed: "pydantic-ai", "pydantic_ai"
and "mcp-server" found nothing, and "speech to text" missed Google because
the integration is named "Google Cloud Speech-to-Text".
* Registry: Build the provider card's integration list inline
The loop that accumulated integration names relied on a push side effect
inside a comma expression, which reads as if it were working around
Nunjucks scoping when a single inline expression renders the same output.
* Registry: Rank Explore category badges by monthly downloads
Each Explore card shows six provider badges taken with `| slice(0, 6)` off
an unsorted list, so the row was whatever providers.json happened to list
first. Widening a category's membership then pushed recognisable providers
out of the row: Messaging & Notifications lost Slack and SMTP while the
card's own copy still read "Slack, email, SMS, and alerting services",
Databases lost MySQL and PostgreSQL, Workflow lost HTTP and SSH.
Membership is unchanged; only which six are shown. The Top Providers and
Incubating Providers rows on the same page already rank before slicing.
* Registry: Correct the docs for what category keywords match
The keyword matcher now compares against a provider's declared integration
names as well as its id, and normalises separators before comparing, but the
prose a contributor reads before adding a keyword still described the
id-only substring rule.
The comment on the shared matcher also credited the per-provider page's
category chips; providerCategoryMap's only consumer is the category
dropdown on /providers/.
* Registry: Let provider search find providers by id
The build-time matcher searches a provider's id alongside its integration
names, but neither search entry point did. On /providers/ the filter box saw
only the display name and the integration names, so `clickhousedb`,
`cncf-kubernetes`, `microsoft-mssql`, `edge3`, `common-dataquality`,
`microsoft-psrp` and `microsoft-winrm` were unreachable from their own id,
and the distribution name the card prints under the title matched nothing at
all. The Pagefind index had the same gap.
The id is indexed on its own rather than as part of the distribution name,
and the shared `apache-airflow-providers-` prefix is dropped off the query
instead: indexing the full name would make `apache`, `airflow` and
`providers` match every provider.
Two bugs on the /registry/explore/ page:
1. Category cards all showed "6+ providers" because exploreCategoryProviders.js
capped matching at 6 providers per category. Removed the cap so the real count
is displayed (the template already slices to 6 badges for display).
2. Top Providers and Incubating Providers sections were empty/showing 0 modules
because explore.njk read module_counts from providers.json (always zeros).
Switched to moduleCountsByProvider which computes counts from modules.json.
The version-less provider URL (e.g. /providers/amazon/) redirects to
the latest version using a meta refresh, which drops the #fragment.
This breaks deep links like /providers/amazon/#amazon-s3-S3Hook.
Use a JS redirect that appends window.location.hash, with a noscript
meta refresh fallback for JS-disabled browsers.
The Nunjucks loop that filters providers with first_released dates
emits blank lines for each iteration (~99 providers x 3 lines).
Use whitespace-trimming delimiters ({%- -%}) on the data-processing
loop to eliminate blank lines from the rendered HTML.
The stats page "Top 10 by Module Count" table showed 0 for all
per-type columns (Operators, Hooks, Sensors, etc.) while the Total
column was correct. This was caused by incremental builds: the
`...p` spread copied stale `module_counts` zeros from providers.json,
but `totalModules` was correctly computed from modules.json.
Override `module_counts` with the actual per-type counts from
modules.json (the source of truth), matching how `totalModules`
is already computed.
The `--provider` flag was only passed to `extract_metadata.py` but not
to `extract_parameters.py` or `extract_connections.py`. This caused
incremental builds to scan all 99 providers and 1625 modules instead
of just the requested one.
The registry workflow was building the CI image from scratch every run
(~24 min) because it lacked the BuildKit mount cache that
ci-image-build.yml provides. Inline `breeze ci-image build` with
registry cache doesn't help because Docker layer cache invalidates
on every commit when the build context changes.
Split into two jobs following the established pattern used by
ci-amd-arm.yml and update-constraints-on-push.yml:
- `build-ci-image`: calls ci-image-build.yml which handles mount cache
restore, ghcr.io login, registry cache, and image stashing
- `build-and-publish-registry`: restores the stashed image via
prepare_breeze_and_image action, then runs the rest unchanged
* Fix merge crash when incremental extract skips modules.json
extract_parameters.py with --provider intentionally skips writing
modules.json (only the targeted provider's parameters are extracted).
The merge script assumed modules.json always exists, causing a
FileNotFoundError during incremental builds.
Handle missing new_modules_path the same way missing
existing_modules_path is already handled: treat it as an empty list.
* Fix /mnt not writable when loading stashed CI image
The prepare_breeze_and_image action loads the CI image from /mnt, which
requires make_mnt_writeable.sh to run first. Each job gets a fresh
runner, so the writeable /mnt from the build job doesn't carry over.
* Regenerate pnpm lockfile for workspace mode
Adding `packages: ['.']` to pnpm-workspace.yaml changed how pnpm
processes overrides, causing ERR_PNPM_LOCKFILE_CONFIG_MISMATCH with
--frozen-lockfile. Regenerate the lockfile with pnpm 9 to match.
* Scope prebuild uv resolution to dev/registry project
The prebuild script ran `uv run` without --project, causing uv to
resolve the full workspace including samba → krb5 which needs
libkrb5-dev (not installed on the CI runner).
Eleventy pagination templates emit empty fallback JSON for every provider,
even when only one provider's data was extracted. A plain `aws s3 sync`
uploads those stubs and overwrites real connection/parameter data.
Changes:
- Exclude per-provider connections.json and parameters.json from the main
S3 sync during incremental builds, then selectively upload only the
target provider's API files
- Filter connections early in extract_connections.py (before the loop)
and support space-separated multi-provider IDs
- Suppress SCARF_ANALYTICS and DO_NOT_TRACK telemetry in CI
- Document the Eleventy pagination limitation in README and AGENTS.md
* Exclude all per-provider API files during incremental S3 sync
The previous exclude only covered connections.json and parameters.json,
but modules.json and versions.json for non-target providers also contain
incomplete data (no version info extracted) and would overwrite correct
data on S3. Simplify to exclude the entire api/providers/* subtree and
selectively upload only the target provider's directory.
* Also exclude provider HTML pages during incremental S3 sync
Non-target provider pages are rebuilt without connection/parameter data
(the version-specific extraction files don't exist locally). Without
this exclude, the incremental build overwrites complete HTML pages on
S3 with versions missing the connection builder section.
The providers listing page uses merged data (all providers) and must
be updated during incremental builds — especially for new providers.
AWS CLI --include after --exclude re-includes the specific file.
The sort dropdown defaulted to "Most Downloads" but the sort logic only
ran on the `change` event, so providers stayed in alphabetical order
(the template render order) until the user manually toggled the dropdown.
Pydantic contract models validate every JSON payload the registry
produces (providers catalog, modules, parameters, connections,
version metadata, provider versions). Validation runs at generation
time — no separate jsonschema layer needed.
An OpenAPI 3.1 spec is generated from these contracts at build time
and powers an interactive API Explorer at /api-explorer/, using
swagger-ui vendored from node_modules (Apache Infra blocks external
CDN via CSP). The explorer shares the site layout (header, nav,
footer) and uses swagger-ui's native dark-mode class for theme sync.
Also adds a /api/providers/{id}/versions.json endpoint used by CI
to publish new provider versions without a full site rebuild.
Wei Lee
Kaxil Naik
dependabot[bot]
Shahar EpsteinandJarek Potiuk
Amogh Desai
Wei Lee
Jens Scheffler