diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index a1acdc4cd..8c9a8beb0 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -17,7 +17,7 @@ - [ ] **One tool per PR**: This PR adds or modifies only one MCP tool for faster review cycles - [ ] Updated `servers/Azure.Mcp.Server/README.md` and/or `servers/Fabric.Mcp.Server/README.md` documentation - [ ] Validate `README.md` changes running the script `./eng/scripts/Process-PackageReadMe.ps1`. See [Package README](https://github.com/microsoft/mcp/blob/main/CONTRIBUTING.md#package-readme) - - [ ] For new or modified tool descriptions, ran [`ToolDescriptionEvaluator`](https://github.com/microsoft/mcp/blob/main/eng/tools/ToolDescriptionEvaluator/Quickstart.md) and obtained a score of `0.4` or more and a top 3 ranking for all related test prompts + - [ ] For new or modified tool descriptions (excluding Fabric MCP tools), ran [`ToolDescriptionEvaluator`](https://github.com/microsoft/mcp/blob/main/eng/tools/ToolDescriptionEvaluator/Quickstart.md) and obtained a score of `0.4` or more and a top 3 ranking for all related test prompts - [ ] For tools with new names, including new tools or renamed tools, update [`consolidated-tools.json`](https://github.com/microsoft/mcp/blob/main/servers/Azure.Mcp.Server/src/Resources/consolidated-tools.json) - [ ] For **renamed** tools, follow the [Tool Rename Checklist](https://github.com/microsoft/mcp/blob/main/docs/tool-rename-checklist.md) and tag the PR with the `breaking-change` label - [ ] For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description diff --git a/.github/agents/onboarding.agent.md b/.github/agents/onboarding.agent.md index 2b4e03453..86429c44a 100644 --- a/.github/agents/onboarding.agent.md +++ b/.github/agents/onboarding.agent.md @@ -284,7 +284,7 @@ You can duplicate an existing configuration to debug your own new command. - [ ] `.\eng\common\spelling\Invoke-Cspell.ps1` — no spelling errors - [ ] `./eng/scripts/Test-Code.ps1` — unit tests pass - [ ] `.\eng\scripts\Update-AzCommandsMetadata.ps1` — metadata up-to-date -- [ ] Tool descriptions validated with `ToolDescriptionEvaluator` (score ≥ 0.4) +- [ ] Tool descriptions validated with `ToolDescriptionEvaluator` (score ≥ 0.4) (not required for Fabric MCP tools) - [ ] Live tests recorded and passing in playback (Azure commands) - [ ] AOT check for new toolsets: `./eng/scripts/Build-Local.ps1 -BuildNative` - [ ] Changelog entry created if applicable diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 403e09ee3..d1b2d1fb0 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -56,6 +56,6 @@ When reviewing PRs that add or modify MCP tools, cross-reference the [pre-merge - **`consolidated-tools.json`**: If a tool is renamed or removed, `servers/Azure.Mcp.Server/src/Resources/consolidated-tools.json` must be updated. Flag a tool rename when this file is not in the diff. - **`e2eTestPrompts.md`**: If a tool is added or modified, `servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` must include prompts for it. Flag a diff that removes all prompts for a tool that remains registered, or that adds a tool without corresponding prompts. - **`azmcp-commands.md`**: If a tool is added, modified, or removed, `servers/Azure.Mcp.Server/docs/azmcp-commands.md` must be updated. Flag command registration changes when this file is not in the diff. -- **ToolDescriptionEvaluator**: If tool descriptions are added or changed, the PR description must confirm a ToolDescriptionEvaluator score of at least `0.4`. Flag new tool descriptions when the PR body does not mention the score. +- **ToolDescriptionEvaluator** (excluding Fabric MCP tools): If tool descriptions are added or changed, the PR description must confirm a ToolDescriptionEvaluator score of at least `0.4`. Flag new tool descriptions when the PR body does not mention the score. - **TreatWarningsAsErrors**: This repository sets `TreatWarningsAsErrors=true`. An `async` method with no `await` produces CS1998 and fails the build. Report these findings as build breaks, not style suggestions. - **Changelog entries**: Entries under `servers/Azure.Mcp.Server/changelog-entries/` must conform to the YAML schema in `docs/changelog-entries.md`. Require a non-empty `changes` array whose entries contain a valid `section` and a `description`; `pr` and `subsection` are optional. Check schema conformance, and do not apply generic prose style suggestions to changelog entry files. diff --git a/.github/skills/mcp-code-reviewer/reviewers.md b/.github/skills/mcp-code-reviewer/reviewers.md index 07eb0f9ed..e0f68131a 100644 --- a/.github/skills/mcp-code-reviewer/reviewers.md +++ b/.github/skills/mcp-code-reviewer/reviewers.md @@ -97,7 +97,7 @@ Cross-reference the repository instructions and pull request checklist. Focus on - Command reference and README updates - End-to-end prompt coverage - Consolidated tool mappings for new, renamed, or removed Azure MCP tools; do not apply this check to Fabric MCP changes -- Tool description evaluation evidence when descriptions change +- Tool description evaluation evidence when descriptions change (not required for Fabric MCP tools) - Changelog entry schema and required entries - Rename, compatibility, and breaking-change requirements diff --git a/AGENTS.md b/AGENTS.md index b0f5416dd..81b37150b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -188,7 +188,7 @@ dotnet build - Live tests: Include live tests for all commands that interact with Azure resources - Recorded tests: All live tests **must** be recorded for playback (see `/docs/recorded-tests.md`) - Documentation: Update `/servers/Azure.Mcp.Server/docs/azmcp-commands.md` and add test prompts to `/servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` -- Tool validation: Run `ToolDescriptionEvaluator` for command descriptions (target: top 3 ranking, ≥0.4 confidence) +- Tool validation (optional for Fabric MCP tools): Run `ToolDescriptionEvaluator` for command descriptions (target: top 3 ranking, ≥0.4 confidence) - Spelling check: `.\eng\common\spelling\Invoke-Cspell.ps1` - Changelog: Create changelog entry YAML file if the change is a new feature, bug fix, or breaking change. See `/docs/changelog-entries.md` for instructions. Always use the `-ChangelogPath` parameter (e.g., `/servers/Azure.Mcp.Server/CHANGELOG.md` or `/servers/Fabric.Mcp.Server/CHANGELOG.md`). - One tool per PR: Submit single toolsets for faster review cycles @@ -298,7 +298,7 @@ dotnet format --include="tools/Azure.Mcp.Tools.Storage/**/*.cs" # AOT compatibility analysis ./eng/scripts/Analyze-AOT-Compact.ps1 -# Tool description quality validation +# Tool description quality validation (optional for Fabric MCP tools) pushd 'eng/tools/ToolDescriptionEvaluator/src' dotnet run -- --validate --tool-description "Your command description" --prompt "user query" --test-single-tool 'your-tool-name' popd @@ -576,7 +576,7 @@ dotnet run -- --validate \ dotnet run -- --tools-file my-tools.json --prompts-file my-prompts.md popd -# Target: Top 3 ranking and confidence score ≥ 0.4 +# Target: Top 3 ranking and confidence score ≥ 0.4 (optional for Fabric MCP tools) ``` ## Local Development and Testing @@ -767,7 +767,7 @@ When working in a linked git worktree (multiple worktrees sharing one repository - **Run all tests**: `./eng/scripts/Test-Code.ps1` - **Format code**: `dotnet format` - **Check spelling**: `.\eng\common\spelling\Invoke-Cspell.ps1` -- **Validate tool descriptions**: Use ToolDescriptionEvaluator +- **Validate tool descriptions**: Use ToolDescriptionEvaluator (optional for Fabric MCP tools) - **Follow contribution guidelines**: See `CONTRIBUTING.md` - **One tool per PR**: Submit single toolsets for faster review diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6bab78e5c..472cb6623 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -216,7 +216,7 @@ Do not assume the Pull Request pipeline will always ingest a missing package aut - Include tests in the `/tests` folder - Ensure all tests pass - Follow code style requirements - - Run [`ToolDescriptionEvaluator`](https://github.com/microsoft/mcp/blob/main/eng/tools/ToolDescriptionEvaluator/Quickstart.md) for the new tool description and obtain a score of `0.4` or more and a top 3 ranking for all related test prompts + - Run [`ToolDescriptionEvaluator`](https://github.com/microsoft/mcp/blob/main/eng/tools/ToolDescriptionEvaluator/Quickstart.md) for the new tool description (excluding Fabric MCP tools) and obtain a score of `0.4` or more and a top 3 ranking for all related test prompts ## Testing diff --git a/eng/tools/ToolDescriptionEvaluator/Quickstart.md b/eng/tools/ToolDescriptionEvaluator/Quickstart.md index b48ff432f..4d49470bf 100644 --- a/eng/tools/ToolDescriptionEvaluator/Quickstart.md +++ b/eng/tools/ToolDescriptionEvaluator/Quickstart.md @@ -1,6 +1,6 @@ # Tool Description Evaluator – Quickstart Guide -This tool helps you test and validate the descriptions of new Azure MCP Server tools. It checks how well your tool descriptions match real user prompts, ensuring users get the right tool when they ask for something. +This tool helps you test and validate the descriptions of new Azure MCP Server tools. It checks how well your tool descriptions match real user prompts, ensuring users get the right tool when they ask for something. Evaluation is optional for Fabric MCP tools. The tool returns a confidence score between `0.00` and `1.00` for each tool-prompt combination. The higher the score, the better chance the tool will be selected given a specific prompt. Scores below `0.40` generally indicate low tool selection chances and tool descriptions should be improved.