diff --git a/AGENTS.md b/AGENTS.md index afd314a4f..396f4b583 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,8 +14,10 @@ - Use extension methods `.AsRequired()` and `.AsOptional()` for option handling - Use name-based binding with `parseResult.GetValueOrDefault()` - Always call `HandleException(context, ex)` in catch blocks +- Include live tests for all commands that interact with Azure resources - Create Bicep templates for Azure service commands (`test-resources.bicep`) - Include post-deployment scripts (`test-resources-post.ps1`) +- Record all live tests according to the guidelines in `/docs/recorded-tests.md` - Submit one tool per pull request - Use `BaseAzureResourceService` for Resource Graph queries when possible - Register all response models in JSON serialization context for AOT safety @@ -35,7 +37,7 @@ - Use `subscriptionId` parameter name - Add unnecessary "-name" suffixes (use `--account` vs `--account-name`) - Use readonly option fields in commands -- Skip live test infrastructure for Azure service commands +- Skip live tests, live test infrastructure, or test recordings for Azure service commands - Use `parseResult.GetValue()` without generic type parameter - Redefine base class properties in Options classes - Skip `base.RegisterOptions()` or `base.Dispose()` calls @@ -164,9 +166,10 @@ dotnet build ## API Docs and References - API documentation: `/servers/Azure.Mcp.Server/docs/azmcp-commands.md` - Complete command reference - Implementation guide: `/servers/Azure.Mcp.Server/docs/new-command.md` - Step-by-step command creation -- Test prompts: `servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` - Example prompts for testing -- Contributing guide: `CONTRIBUTING.md` - Development workflow and standards -- Code guidelines: `.github/copilot-instructions.md` - Specific coding standards +- Test prompts: `/servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` - Example prompts for testing +- Recorded tests: `/docs/recorded-tests.md` - Guide for converting live tests to recorded (playback) tests +- Contributing guide: `/CONTRIBUTING.md` - Development workflow and standards +- Code guidelines: `/.github/copilot-instructions.md` - Specific coding standards ## When Stuck - Ask clarifying questions about Azure service requirements or command patterns @@ -179,10 +182,12 @@ dotnet build - Format and type check: `dotnet format && dotnet build` - all green - Unit tests: Add comprehensive tests following existing patterns - Live test infrastructure: Include Bicep template and post-deployment script for Azure services +- Live tests: Include live tests for all commands that interact with Azure resources +- Recorded tests: All live tests **must** be recorded for playback (see `/docs/recorded-tests.md`) - Documentation: Update `/servers/Azure.Mcp.Server/docs/azmcp-commands.md` and add test prompts to `/servers/Azure.Mcp.Server/docs/e2eTestPrompts.md` - Tool validation: Run `ToolDescriptionEvaluator` for command descriptions (target: top 3 ranking, ≥0.4 confidence) - Spelling check: `.\eng\common\spelling\Invoke-Cspell.ps1` -- Changelog: Create changelog entry YAML file if the change is a new feature, bug fix, or breaking change. See `docs/changelog-entries.md` for instructions. Always use the `-ChangelogPath` parameter (e.g., `servers/Azure.Mcp.Server/CHANGELOG.md` or `servers/Fabric.Mcp.Server/CHANGELOG.md`). +- Changelog: Create changelog entry YAML file if the change is a new feature, bug fix, or breaking change. See `/docs/changelog-entries.md` for instructions. Always use the `-ChangelogPath` parameter (e.g., `/servers/Azure.Mcp.Server/CHANGELOG.md` or `/servers/Fabric.Mcp.Server/CHANGELOG.md`). - One tool per PR: Submit single toolsets for faster review cycles ## Architecture and Project Structure @@ -314,6 +319,9 @@ All commands must include comprehensive unit tests: Command unit tests should extend `CommandUnitTestsBase` where `TCommand` is the command class being tested and `TService` is the service that the command uses. +### Live Testing Requirements +Azure service commands require live tests to validate functionality against actual Azure resources. Live tests must be recorded for playback using `RecordedCommandTestsBase`. See `/docs/recorded-tests.md` for the full recording workflow, sanitizer configuration, and migration guide. + ### Live Test Infrastructure Azure service commands require Bicep templates for test resources: ```powershell diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9765400a2..182511d8e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,6 +29,7 @@ If you are contributing significant changes, or if the issue is already assigned - [Testing Local Build with Docker](#testing-local-build-with-docker) - [Live Tests](#live-tests) - [NPX Live Tests](#npx-live-tests) + - [Recording Live Tests](#recording-live-tests) - [Debugging Live Tests](#debugging-live-tests) - [Quality and Standards](#quality-and-standards) - [Code Style](#code-style) @@ -183,7 +184,7 @@ If you are contributing significant changes, or if the issue is already assigned ## Testing -Command authors must provide both unit tests and end-to-end test prompts. +Command authors must provide unit tests and end-to-end test prompts. Commands that interact with Azure resources **must** also include live tests with recorded playback coverage (see [Recording Live Tests](#recording-live-tests)). ### Unit Tests @@ -498,6 +499,8 @@ To build a local image for testing purposes: ### Live Tests > [!IMPORTANT] +> Live tests are **required** for all commands that interact with Azure resources. +> > If you are a **Microsoft employee** with Azure source permissions then please review our [Azure Internal Onboarding Documentation](https://aka.ms/azmcp/intake). As part of reviewing community contributions, Azure team members can run live tests by adding this comment to the PR `/azp run mcp - pullrequest - live`. Before running live tests: @@ -575,6 +578,10 @@ This will produce .tgz files in the `.dist` directory and set the `TestPackage` "TestPackage": "file://D:\\repos\\azure-mcp\\.dist\\wrapper\\azure-mcp-0.0.12-alpha.1746488279.tgz" ``` +### Recording Live Tests + +All new live tests **must** be recorded for playback. Live tests use the Azure SDK Test Proxy to capture and replay HTTP traffic, ensuring CI can validate tests without live Azure resources. For the full migration guide, recording workflow, sanitizer/matcher configuration, and troubleshooting tips, see [docs/recorded-tests.md](https://github.com/microsoft/mcp/blob/main/docs/recorded-tests.md). + ### Debugging Live Tests This section assumes that the necessary Azure resources for live tests are already deployed and that the `.testsettings.json` file with deployment information is located in the area's `/tests/` directory. diff --git a/servers/Azure.Mcp.Server/docs/new-command.md b/servers/Azure.Mcp.Server/docs/new-command.md index bbd33b0d3..a4dba02ab 100644 --- a/servers/Azure.Mcp.Server/docs/new-command.md +++ b/servers/Azure.Mcp.Server/docs/new-command.md @@ -20,13 +20,15 @@ This keeps all code, options, models, JSON serialization contexts, and tests for **CRITICAL DECISION POINT**: Does your command interact with Azure resources? -### **Azure Service Commands (REQUIRES Test Infrastructure)** +### **Azure Service Commands (REQUIRE Test Infrastructure and Live Tests)** If your command interacts with Azure resources (storage accounts, databases, VMs, etc.): - ✅ **MUST create** `tools/Azure.Mcp.Tools.{Toolset}/tests/test-resources.bicep` - ✅ **MUST create** `tools/Azure.Mcp.Tools.{Toolset}/tests/test-resources-post.ps1` (required even if basic template) - ✅ **MUST include** RBAC role assignments for test application - ✅ **MUST validate** with `az bicep build --file tools/Azure.Mcp.Tools.{Toolset}/tests/test-resources.bicep` - ✅ **MUST test deployment** with `./eng/scripts/Deploy-TestResources.ps1 -Tool 'Azure.Mcp.Tools.{Toolset}'` +- ✅ **MUST include** live tests in `Azure.Mcp.Tools.{Toolset}.LiveTests` +- ✅ **MUST record** live tests for playback using `RecordedCommandTestsBase` (see [`/docs/recorded-tests.md`](https://github.com/microsoft/mcp/blob/main/docs/recorded-tests.md)) ### **Non-Azure Commands (No Test Infrastructure Needed)** If your command is a wrapper/utility (CLI tools, best practices, documentation): @@ -115,7 +117,7 @@ Every new command (whether purely computational or Azure-resource backed) requir 5. Service implementation: `tools/Azure.Mcp.Tools.{Toolset}/src/Services/{ServiceName}Service.cs` - Most toolsets have one primary service; some may have multiple where domain boundaries justify separation 6. Unit test: `tools/Azure.Mcp.Tools.{Toolset}/tests/Azure.Mcp.Tools.{Toolset}.UnitTests/{Resource}/{Resource}{Operation}CommandTests.cs` -7. Integration test: `tools/Azure.Mcp.Tools.{Toolset}/tests/Azure.Mcp.Tools.{Toolset}.LiveTests/{Toolset}CommandTests.cs` +7. Live test: `tools/Azure.Mcp.Tools.{Toolset}/tests/Azure.Mcp.Tools.{Toolset}.LiveTests/{Toolset}CommandTests.cs` 8. Command registration in RegisterCommands(): `tools/Azure.Mcp.Tools.{Toolset}/src/{Toolset}Setup.cs` 9. Toolset registration in RegisterAreas(): `servers/Azure.Mcp.Server/src/Program.cs` 10. **Live test infrastructure** (for Azure service commands): @@ -1318,13 +1320,13 @@ Guidelines: [assembly: Xunit.CollectionBehavior(Xunit.CollectionBehavior.CollectionPerAssembly)] ``` -### 8. Integration Tests +### 8. Live Tests -Integration tests inherit from `CommandTestsBase` and use test fixtures: +Live tests **must** inherit from `RecordedCommandTestsBase` and use test fixtures. All live tests are required to be recorded for playback. See [`/docs/recorded-tests.md`](https://github.com/microsoft/mcp/blob/main/docs/recorded-tests.md) for the full recording workflow. ```csharp -public class {Toolset}CommandTests(ITestOutputHelper output) - : CommandTestsBase( output) +public class {Toolset}CommandTests(ITestOutputHelper output, TestProxyFixture fixture, LiveServerFixture liveServerFixture) + : RecordedCommandTestsBase(output, fixture, liveServerFixture) { [Theory] [InlineData(AuthMethod.Credential)] @@ -1615,9 +1617,12 @@ dotnet test --filter "FullyQualifiedName~EntraAdminListCommandTests" --verbosity dotnet test --verbosity normal ``` -### Integration Tests +### Live Tests + Azure service commands requiring test resource deployment must add a bicep template, `tests/test-resources.bicep`, to their toolset directory. Additionally, all Azure service commands must include a `test-resources-post.ps1` file in the same directory, even if it contains only the basic template without custom logic. See `/tools/Azure.Mcp.Tools.Storage/tests/test-resources.bicep` and `/tools/Azure.Mcp.Tools.Storage/tests/test-resources-post.ps1` for canonical examples. +All live tests **must** be recorded for playback using `RecordedCommandTestsBase`. See [`/docs/recorded-tests.md`](https://github.com/microsoft/mcp/blob/main/docs/recorded-tests.md) for the full recording workflow, sanitizer configuration, and migration guide. + #### Live Test Resource Infrastructure **1. Create Toolset Bicep Template (`/tools/Azure.Mcp.Tools.{Toolset}/tests/test-resources.bicep`)**