Match providers to Explore categories by integration name, not just id (#70498)

* Match providers to Explore categories by integration name, not just id

The Explore page and each provider's category chips only matched a
provider to a category by substring-matching the category's keywords
against the provider's id. Providers whose id doesn't happen to
contain a relevant keyword never showed up, even when their
provider.yaml declares integrations that obviously belong there. For
example, common-ai (wraps pydantic-ai) ships hooks named "Pydantic AI",
"MCP Server", "LangChain", and "LlamaIndex", each tagged ai, but none of
those words are substrings of the id "common-ai", so it never appeared
under AI & Machine Learning or in a search for "LangChain"/"MCP".

Match against each provider's declared integration names (already
extracted as provider.categories[].name) in addition to its id, and
add the ai-ml category keywords needed for common-ai's integrations.
Also scan connection_types[].external_integrations when present so a
future field populating that data benefits automatically.

* Add more provider explore-category keywords found via integration-name audit

Auditing all provider.yaml integration names/tags against each Explore
category's keyword list turned up further clear misses of the same
id-substring bug fixed for common-ai: MSSQL, OpenSearch, IBM Cloudant,
and ArangoDB never showed up under Databases; Exasol and ClickHouse
never showed up under Data Warehouses; DingTalk and Apprise never
showed up under Messaging & Notifications (same tags: [service] class
as the Slack/Discord/Telegram/PagerDuty/Opsgenie entries already
there); and gRPC never showed up under Workflow & Orchestration despite
sharing the same tags: [protocol] as ssh/http/ftp already listed.

Deliberately left out: Apache Kafka and IBM MQ, which also lack any
matching keyword. Both are enterprise message-broker/queueing
middleware (tags: [apache]/[software]) rather than the
notification/alerting channels the Messaging & Notifications category
is themed around and its description names (Slack, email, SMS,
alerting services), so adding them is a judgment call left for a
maintainer rather than an obvious fix.

* Registry: Tighten Explore category keyword matching

The keyword matcher compared in both directions, which was safe while the
left-hand side was always a provider slug. Now that it also runs against
free-text integration names, a long keyword could swallow short ones the
tree already has ("Tpt", "Ttu", "YDB", "Bteq"). Matching forward only
removes that hazard without changing any provider's category membership.

Keywords and integration names also spell separators differently, which
forced duplicate entries such as 'pydantic ai' alongside 'pydantic-ai'.
Collapsing separator runs to a single space lets one keyword cover both
spellings. Collapsing rather than stripping matters: stripping would turn
"Microsoft Power BI" into a string containing "ftp".

The connection-type branch read a field that cannot exist, since the
contract that validates providers.json forbids unknown keys and does not
declare it.

* Registry: Let the providers search box find integration names

Matching providers to Explore categories by integration name fixed the
category dropdown but not the search box beside it, which still compared
the provider's display name alone. Typing the name of an integration a
provider declares — "langchain", "opensearch" — found nothing, which is
the case people actually reach for the search box to solve.

* Registry: Include integration names in the search index

The site has two independent searches: the filter box on /providers/, and
the global modal reached from the header and the home page hero. The modal
is backed by Pagefind, whose provider records are built from custom
content rather than page HTML, so making integration names available as a
data attribute did nothing for it.

That left the more prominent of the two searches unable to find a provider
by an integration it declares — "langchain" returned nothing even though
Common AI lists LangChain.

* Registry: Match the providers search box on separator variants

The build-time category matcher collapses runs of "-_ " to a single space
before comparing, but the client-side filter box compared raw strings, so
the two halves of the same feature disagreed: "pydantic-ai", "pydantic_ai"
and "mcp-server" found nothing, and "speech to text" missed Google because
the integration is named "Google Cloud Speech-to-Text".

* Registry: Build the provider card's integration list inline

The loop that accumulated integration names relied on a push side effect
inside a comma expression, which reads as if it were working around
Nunjucks scoping when a single inline expression renders the same output.

* Registry: Rank Explore category badges by monthly downloads

Each Explore card shows six provider badges taken with `| slice(0, 6)` off
an unsorted list, so the row was whatever providers.json happened to list
first. Widening a category's membership then pushed recognisable providers
out of the row: Messaging & Notifications lost Slack and SMTP while the
card's own copy still read "Slack, email, SMS, and alerting services",
Databases lost MySQL and PostgreSQL, Workflow lost HTTP and SSH.

Membership is unchanged; only which six are shown. The Top Providers and
Incubating Providers rows on the same page already rank before slicing.

* Registry: Correct the docs for what category keywords match

The keyword matcher now compares against a provider's declared integration
names as well as its id, and normalises separators before comparing, but the
prose a contributor reads before adding a keyword still described the
id-only substring rule.

The comment on the shared matcher also credited the per-provider page's
category chips; providerCategoryMap's only consumer is the category
dropdown on /providers/.

* Registry: Let provider search find providers by id

The build-time matcher searches a provider's id alongside its integration
names, but neither search entry point did. On /providers/ the filter box saw
only the display name and the integration names, so `clickhousedb`,
`cncf-kubernetes`, `microsoft-mssql`, `edge3`, `common-dataquality`,
`microsoft-psrp` and `microsoft-winrm` were unreachable from their own id,
and the distribution name the card prints under the title matched nothing at
all. The Pagefind index had the same gap.

The id is indexed on its own rather than as part of the distribution name,
and the shared `apache-airflow-providers-` prefix is dropped off the query
instead: indexing the full name would make `apache`, `airflow` and
`providers` match every provider.

registry/README.md

@@ -256,11 +256,15 @@ The Explore page and provider filtering use categories defined in
 
 - `id` — URL-safe identifier (e.g., `cloud`, `databases`, `ai-ml`)
 - `name` — Display name
-- `keywords` — List of substrings matched against `provider.id`
+- `keywords` — List of substrings matched against `provider.id` and the provider's
+  declared integration names (`provider.categories[].name`)
 - `icon`, `color`, `description` — Visual properties
 
 Providers are assigned to categories at build time by checking if any keyword in a
-category matches (substring) the provider's ID. A provider can belong to multiple
+category is a substring of the provider's ID or of one of its declared integration
+names, compared case-insensitively after collapsing runs of `-`, `_` and whitespace to
+a single space (so `pydantic-ai` matches the integration named "Pydantic AI"). The
+matcher lives in `src/_data/providerKeywordMatch.js`. A provider can belong to multiple
 categories.
 
 ### Homepage "New Providers" Section

registry/scripts/build-pagefind-index.mjs

@@ -40,9 +40,17 @@ async function buildPagefindIndex() {
   let modulesAdded = 0;
 
   for (const provider of providers.providers) {
+    const integrations = (provider.categories || [])
+      .map((category) => category.name)
+      .filter(Boolean)
+      .join(' ');
+
     await index.addCustomRecord({
       url: `/providers/${provider.id}/${provider.version}/`,
-      content: `${provider.name} ${provider.description}`,
+      // The id is indexed on its own, not as part of the distribution name:
+      // the `apache-airflow-providers-` prefix is shared by every provider and
+      // would make 'apache' or 'airflow' match all of them.
+      content: `${provider.name} ${provider.id} ${provider.description} ${integrations}`,
       language: 'en',
       meta: {
         type: 'provider',

registry/src/_data/exploreCategories.js

@@ -31,7 +31,7 @@ module.exports = [
     name: 'Databases',
     icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 7v10c0 2.21 3.582 4 8 4s8-1.79 8-4V7M4 7c0 2.21 3.582 4 8 4s8-1.79 8-4M4 7c0-2.21 3.582-4 8-4s8 1.79 8 4" /></svg>',
     color: 'blue',
-    keywords: ['postgres', 'mysql', 'mongo', 'redis', 'neo4j', 'elasticsearch', 'cassandra', 'couchbase', 'influxdb', 'sqlite', 'odbc', 'jdbc', 'common-sql'],
+    keywords: ['postgres', 'mysql', 'mongo', 'redis', 'neo4j', 'elasticsearch', 'cassandra', 'couchbase', 'influxdb', 'sqlite', 'odbc', 'jdbc', 'common-sql', 'mssql', 'opensearch', 'cloudant', 'arangodb'],
     description: 'SQL, NoSQL, and time-series databases',
   },
   {
@@ -39,7 +39,7 @@ module.exports = [
     name: 'Data Warehouses',
     icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 11H5m14 0a2 2 0 012 2v6a2 2 0 01-2 2H5a2 2 0 01-2-2v-6a2 2 0 012-2m14 0V9a2 2 0 00-2-2M5 11V9a2 2 0 012-2m0 0V5a2 2 0 012-2h6a2 2 0 012 2v2M7 7h10" /></svg>',
     color: 'purple',
-    keywords: ['snowflake', 'databricks', 'teradata', 'vertica', 'trino', 'presto', 'dbt'],
+    keywords: ['snowflake', 'databricks', 'teradata', 'vertica', 'trino', 'presto', 'dbt', 'exasol', 'clickhouse'],
     description: 'Snowflake, Databricks, and analytics platforms',
   },
   {
@@ -47,7 +47,7 @@ module.exports = [
     name: 'Messaging & Notifications',
     icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 12h.01M12 12h.01M16 12h.01M21 12c0 4.418-4.03 8-9 8a9.863 9.863 0 01-4.255-.949L3 20l1.395-3.72C3.512 15.042 3 13.574 3 12c0-4.418 4.03-8 9-8s9 3.582 9 8z" /></svg>',
     color: 'green',
-    keywords: ['slack', 'sendgrid', 'smtp', 'discord', 'telegram', 'pagerduty', 'opsgenie', 'twilio'],
+    keywords: ['slack', 'sendgrid', 'smtp', 'discord', 'telegram', 'pagerduty', 'opsgenie', 'twilio', 'dingtalk', 'apprise'],
     description: 'Slack, email, SMS, and alerting services',
   },
   {
@@ -55,7 +55,7 @@ module.exports = [
     name: 'AI & Machine Learning',
     icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9.663 17h4.673M12 3v1m6.364 1.636l-.707.707M21 12h-1M4 12H3m3.343-5.657l-.707-.707m2.828 9.9a5 5 0 117.072 0l-.548.547A3.374 3.374 0 0014 18.469V19a2 2 0 11-4 0v-.531c0-.895-.356-1.754-.988-2.386l-.548-.547z" /></svg>',
     color: 'amber',
-    keywords: ['openai', 'cohere', 'anthropic', 'huggingface', 'mlflow', 'pinecone', 'qdrant', 'weaviate', 'pgvector'],
+    keywords: ['openai', 'cohere', 'anthropic', 'huggingface', 'mlflow', 'pinecone', 'qdrant', 'weaviate', 'pgvector', 'langchain', 'llamaindex', 'mcp', 'pydantic ai'],
     description: 'OpenAI, vector DBs, and ML platforms',
   },
   {
@@ -71,7 +71,7 @@ module.exports = [
     name: 'Workflow & Orchestration',
     icon: '<svg fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15" /></svg>',
     color: 'rose',
-    keywords: ['standard', 'celery', 'docker', 'kubernetes', 'ssh', 'http', 'ftp', 'sftp'],
+    keywords: ['standard', 'celery', 'docker', 'kubernetes', 'ssh', 'http', 'ftp', 'sftp', 'grpc'],
     description: 'Core operators, executors, and connectivity',
   },
 ];

registry/src/_data/exploreCategoryProviders.js

@@ -19,6 +19,7 @@
 
 const providersData = require("./providers.json");
 const exploreCategories = require("./exploreCategories");
+const { providerMatchesKeyword } = require("./providerKeywordMatch");
 
 module.exports = function () {
   const map = {};
@@ -26,15 +27,19 @@ module.exports = function () {
     const matched = [];
     for (const provider of providersData.providers) {
       for (const keyword of category.keywords) {
-        if (
-          provider.id.includes(keyword) ||
-          keyword.includes(provider.id)
-        ) {
+        if (providerMatchesKeyword(provider, keyword)) {
           matched.push(provider);
           break;
         }
       }
     }
+    // explore.njk shows only the first six as badges, so rank before slicing
+    // (as its Top/Incubating rows already do) — otherwise the row is whatever
+    // providers.json happened to list first, and widening a category's
+    // membership silently pushes the well-known names off it.
+    matched.sort(
+      (a, b) => (b.pypi_downloads?.monthly || 0) - (a.pypi_downloads?.monthly || 0),
+    );
     map[category.id] = matched;
   }
   return map;

registry/src/_data/providerCategoryMap.js

@@ -19,6 +19,7 @@
 
 const providersData = require("./providers.json");
 const exploreCategories = require("./exploreCategories");
+const { providerMatchesKeyword } = require("./providerKeywordMatch");
 
 module.exports = function () {
   const map = {};
@@ -26,7 +27,7 @@ module.exports = function () {
     const cats = [];
     for (const category of exploreCategories) {
       for (const keyword of category.keywords) {
-        if (provider.id.includes(keyword) || keyword.includes(provider.id)) {
+        if (providerMatchesKeyword(provider, keyword)) {
           cats.push(category.id);
           break;
         }

registry/src/_data/providerKeywordMatch.js

@@ -0,0 +1,58 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// Shared by providerCategoryMap.js and exploreCategoryProviders.js so the
+// two directions of the same provider <-> explore-category matching (used
+// respectively by the category dropdown on /providers/ and the Explore landing
+// page's per-category provider listing) can't drift apart.
+
+// A value "matches" a keyword if the value contains the keyword,
+// case-insensitively, after collapsing runs of "-_\s" to a single space
+// (so 'pydantic-ai' matches "Pydantic AI"). Collapse, don't strip:
+// stripping would turn "Microsoft Power BI" into "microsoftpowerbi",
+// which contains "ftp" and would falsely match the orchestration category.
+function normalize(text) {
+  return text.toLowerCase().replace(/[-_\s]+/g, ' ');
+}
+
+function fuzzyIncludes(value, keyword) {
+  if (!value) {
+    return false;
+  }
+  return normalize(value).includes(normalize(keyword));
+}
+
+// Every string a provider is searchable by: its id/slug and its declared
+// integration names (provider.categories[].name — e.g. "LangChain",
+// "Pydantic AI").
+function collectSearchableValues(provider) {
+  const values = [provider.id];
+  for (const category of provider.categories || []) {
+    if (category.name) {
+      values.push(category.name);
+    }
+  }
+  return values;
+}
+
+function providerMatchesKeyword(provider, keyword) {
+  return collectSearchableValues(provider).some((value) => fuzzyIncludes(value, keyword));
+}
+
+module.exports = { providerMatchesKeyword };

registry/src/js/provider-filters.js

@@ -33,16 +33,40 @@
   let currentSearch = '';
   let debounceTimer;
 
+  // Kept in sync by hand with normalize() in src/_data/providerKeywordMatch.js:
+  // that module runs at build time under CommonJS and this file is a browser
+  // IIFE, so the two can't share it. If the two drift, a query like
+  // 'pydantic-ai' stops finding the integration named "Pydantic AI".
+  function normalize(text) {
+    return text.toLowerCase().replace(/[-_\s]+/g, ' ');
+  }
+
+  // The card prints the full distribution name under the title, so that string
+  // has to be a working query. Matching against it directly would make
+  // 'apache', 'airflow' and 'providers' match every card, so drop the shared
+  // prefix off the query instead and match the id alone.
+  const PACKAGE_PREFIX = normalize('apache-airflow-providers-');
+
+  function normalizeSearch(text) {
+    const search = normalize(text);
+    return search.startsWith(PACKAGE_PREFIX) ? search.slice(PACKAGE_PREFIX.length) : search;
+  }
+
   function filterProviders() {
     let visibleCount = 0;
+    const search = normalizeSearch(currentSearch);
 
     providerItems.forEach(item => {
       const lifecycle = item.dataset.lifecycle;
       const name = item.dataset.name || '';
+      const id = item.dataset.id || '';
       const categories = item.dataset.categories || '';
+      const integrations = item.dataset.integrations || '';
 
       const matchesLifecycle = currentLifecycle === 'all' || lifecycle === currentLifecycle;
-      const matchesSearch = name.includes(currentSearch.toLowerCase());
+      const matchesSearch = normalize(name).includes(search) ||
+        normalize(id).includes(search) ||
+        normalize(integrations).includes(search);
       const matchesCategory = currentCategory === 'all' ||
         categories.split(',').includes(currentCategory);
 

registry/src/providers.njk

@@ -66,9 +66,11 @@ mainClass: providers-page
       <li class="provider-item"
           data-lifecycle="{{ lcDisplay }}"
           data-name="{{ provider.name | lower }}"
+          data-id="{{ provider.id | lower }}"
           data-downloads="{{ provider.pypi_downloads.monthly }}"
           data-updated="{{ provider.last_updated or '' }}"
-          data-categories="{{ cats | join(',') }}">
+          data-categories="{{ cats | join(',') }}"
+          data-integrations="{{ (provider.categories or []) | selectattr('name') | join(',', 'name') | lower }}">
         {% include "provider-card.njk" %}
       </li>
       {% endfor %}