mcp-microsoft/docs/bug-bash/installation-testing.md

Installation Testing Guide

This guide covers testing Azure MCP Server installation across different platforms and IDEs.

Table of Contents

• Prerequisites
• Platform-Specific Testing
• IDE Installation Testing
• Verification Steps

Prerequisites

Before testing installation, ensure you have:

• Azure Subscription - Access to an Azure subscription

Platform-Specific Testing

Windows Testing

Installation Steps

1. Install Prerequisites

   # Check if Node.js is installed
   node --version
   
   # If not installed, download from https://nodejs.org/
   

2. Install via NPM

   npm install -g @azure/mcp@latest
   

3. Install via NuGet

   dotnet tool install Azure.Mcp
   

Verification

# Check Azure MCP version
azmcp --version

# Verify Azure CLI
az --version

# Test basic command
azmcp server start --help

Things to Test

• Installation completes without errors
• Binary is accessible from command line
• Extension loads in VS Code
• Server starts successfully
• Memory usage after installation: _____ MB
• Installation time: _____ minutes

macOS Testing

Installation Steps

1. Install Prerequisites

   # Check if Node.js is installed
   node --version
   
   # Install via Homebrew if needed
   brew install node
   

2. Install via NPM

   npm install -g @azure/mcp@latest
   

3. Install via .NET Tool

   dotnet tool install Azure.Mcp
   

Verification

# Check Azure MCP version
azmcp --version

# Verify Azure CLI
az --version

# Test basic command
azmcp server start --help

Platform-Specific Checks

• Intel Macs: Installation works on x64 architecture
• Apple Silicon: Installation works on ARM64 architecture

Things to Test

• Installation completes without errors
• Binary has correct permissions (executable)
• Extension loads in VS Code
• Server starts successfully
• No "unidentified developer" warnings
• Memory usage after installation: _____ MB
• Installation time: _____ minutes

---

Linux Testing

Installation Steps

1. Install Prerequisites

   # Check if Node.js is installed
   node --version
   
   # Ubuntu/Debian
   sudo apt update
   sudo apt install nodejs npm
   
   # Fedora
   sudo dnf install nodejs npm
   

2. Install via NPM (Alternative)

   npm install -g @azure/mcp@latest
   

3. Install via .NET Tool (Alternative)

   dotnet tool install Azure.Mcp
   

Verification

# Check Azure MCP version
azmcp --version

# Verify Azure CLI
az --version

# Test basic command
azmcp server start --help

Things to Test

• Installation completes without errors
• Binary has correct permissions
• Extension loads in VS Code
• Server starts successfully
• No dependency conflicts
• Memory usage after installation: _____ MB
• Installation time: _____ minutes

---

IDE Installation Testing

VS Code

Versions to Test

• VS Code Stable (latest)
• VS Code Insiders (latest)

Installation Testing

1. Install Extension

   • Method 1: Via Extension Marketplace
   • Method 2: Via Installation Link

2. Verify Installation

   1. Open Command Palette (Ctrl+Shift+P / Cmd+Shift+P)
   2. Run "MCP: List Servers"
   3. Verify "Azure MCP Server" appears in the list
   4. Click "Start Server"
   5. Check Output window for startup logs
   

Things to Test

• Extension installs without errors
• Extension appears in Extensions list
• Server starts successfully
• Tools appear in GitHub Copilot Chat
• Tool count matches expectations
• Configuration changes are applied
• Server restarts successfully
• Logs are visible in Output window

Performance Checks

• Extension activation time: _____ ms
• Memory usage (extension): _____ MB
• Memory usage (server): _____ MB
• CPU usage during idle: _____ %

---

Visual Studio 2022

Versions to Test

• Visual Studio 2022 Community
• Visual Studio 2022 Professional
• Visual Studio 2022 Enterprise

Installation Testing

1. Install GitHub Copilot for Azure Extension

   1. Open Visual Studio 2022
   2. Go to Extensions → Manage Extensions...
   3. Switch to the Browse tab in Extension Manager
   4. Search for "GitHub Copilot for Azure"
   5. Click Install
   6. Restart Visual Studio when prompted
   

2. Verify Installation

   1. Open a solution or project
   2. Open GitHub Copilot Chat
   3. Select Agent Mode
   4. Click the tools icon to view available tools
   5. Search for "Azure MCP Server" to filter results
   6. Verify Azure MCP tools appear in the list
   

Things to Test

• GitHub Copilot for Azure extension installs successfully
• Extension appears in Extensions list after restart
• GitHub Copilot Chat opens correctly
• Can switch to Agent Mode
• Tools icon shows available MCP tools
• Can search/filter for "Azure MCP Server" tools
• Server responds to commands in Agent mode

Performance Checks

• Extension load time: _____ seconds
• Memory usage: _____ MB
• CPU usage during idle: _____ %

---

IntelliJ IDEA

Versions to Test

• IntelliJ IDEA Ultimate (2024.3+)
• IntelliJ IDEA Community (2024.3+)

Installation Testing

1. Install GitHub Copilot Plugin

   1. Open IntelliJ IDEA
   2. Go to Settings/Preferences > Plugins
   3. Search for "GitHub Copilot"
   4. Install GitHub Copilot plugin
   5. Restart IDE
   

2. Configure Azure MCP Server

   1. Go to File → Settings (or IntelliJ IDEA → Preferences on macOS)
   2. Navigate to Tools → AI Assistant → Model Context Protocol (MCP)
   3. Click the + icon to add a new MCP server
   4. In the "New MCP Server" dialog, enter:
      - Name: Azure MCP Server
      - Command: npx (or azmcp if installed via dotnet tool)
      - Arguments: -y @azure/mcp@latest server start
   5. Click OK to save
   6. Restart AI Assistant if prompted
   

3. Verify Installation

   1. Open GitHub Copilot Chat
   2. Verify Azure MCP tools are available
   3. Test a simple command (e.g., "List my Azure subscriptions")
   

Things to Test

• GitHub Copilot plugin installs without errors
• Plugin appears in Plugins list
• MCP configuration dialog is accessible
• Can add Azure MCP Server configuration
• Server configuration saves successfully
• Azure MCP integrates with Copilot Chat
• Tools are accessible in chat
• Server responds to commands

Performance Checks

• Plugin load time: _____ seconds
• Memory usage: _____ MB
• CPU usage during idle: _____ %

UI-Based Mode Toggling Testing (Distribution-Specific)

Different distribution mechanisms provide different ways to configure server modes. Test the UI controls for each distribution method.

Distribution Method 1: VS Code Extension

Objective: Test VS Code Settings UI for mode toggling

Available Settings:

1. Azure Mcp: Enabled Services

   • Location: Settings → Extensions → Azure MCP
   • Purpose: Filter which Azure service namespaces are exposed
   • Type: Multi-select list
   • Options: storage, keyvault, cosmos, sql, etc.

2. Azure Mcp: Read Only

   • Location: Settings → Extensions → Azure MCP
   • Purpose: Enable read-only mode (blocks write operations)
   • Type: Checkbox
   • Default: Unchecked (read-write mode)

3. Azure Mcp: Server Mode

   • Location: Settings → Extensions → Azure MCP
   • Purpose: Choose tool exposure strategy
   • Type: Dropdown
   • Options: single, namespace (default), all

Test 1: Change Server Mode via UI

Steps:

1. Open VS Code Settings (Ctrl+, or Cmd+,)
2. Search for "Azure MCP"
3. Locate "Azure Mcp: Server Mode" dropdown
4. Select "namespace" (default)
5. Note current tool count in Copilot Chat
6. Change to "all"
7. Restart server: Command Palette → "MCP: List Servers" → Azure MCP → Start/Restart
8. Check tool count again

Verify:

• Setting UI is accessible and clear
• Dropdown shows all three options: single, namespace, all
• Default is "namespace"
• Changing mode requires server restart (prompt shown)
• Tool count changes after restart:
  • namespace: ~40-50 tools
  • all: 100+ tools
  • single: 1 tool

Test 2: Toggle Read-Only Mode via UI

Steps:

1. Open VS Code Settings
2. Search for "Azure MCP"
3. Locate "Azure Mcp: Read Only" checkbox
4. Check the box to enable read-only mode
5. Restart server
6. Test read operation: "List my storage accounts"
7. Test write operation: "Create a storage account"
8. Uncheck the box to disable read-only mode
9. Restart server
10. Test write operation again

Verify:

• Checkbox is visible and labeled clearly
• Default state is unchecked (read-write)
• Enabling requires server restart
• Read operations work in both modes
• Write operations blocked when enabled
• Write operations allowed when disabled
• Clear error messages when operations are blocked

Test 3: Filter Enabled Services via UI

Steps:

1. Open VS Code Settings
2. Search for "Azure MCP"
3. Locate "Azure Mcp: Enabled Services" setting
4. Click "Add Item" button
5. Add "storage" to the list
6. Add "keyvault" to the list
7. Restart server
8. Check available tools in Copilot Chat

Verify:

• Setting shows "Add Item" button
• Can add multiple service namespaces
• Can remove items from list
• Empty list = all services enabled
• Non-empty list = only specified services enabled
• Tool count reflects filtered services
• Test prompt for allowed service: "List storage accounts"
• Test prompt for filtered service: "List Cosmos DB accounts"

Test 4: Combine Multiple Settings

Steps:

1. Set "Enabled Services" to: storage, keyvault
2. Set "Server Mode" to: namespace
3. Enable "Read Only" checkbox
4. Restart server
5. Test various operations

Verify:

• All three settings applied correctly
• Only Storage and Key Vault namespaces exposed (~2 tools)
• Read operations work for allowed services
• Write operations blocked by read-only mode
• Filtered services not accessible

Test 5: Settings Persistence

Steps:

1. Configure settings: mode=namespace, readOnly=true, services=[storage]
2. Restart VS Code completely
3. Check if settings persisted
4. Open Settings UI and verify values
5. Start MCP server
6. Verify configuration is applied

Verify:

• Settings persist across VS Code restarts
• Settings shown correctly in UI after restart
• Server starts with saved configuration
• No need to reconfigure

---

Distribution Method 2: NPM Package

Objective: Test CLI flag-based mode configuration with NPM-installed package

Installation:

npm install -g @azure/mcp@latest

Test 1: Server Mode via CLI Flags

Test namespace mode:

azmcp server start --mode namespace

Verify:

• Server starts successfully
• ~40-50 namespace-level tools exposed
• Tools grouped by service

Test all mode:

azmcp server start --mode all

Verify:

• Server starts successfully
• 100+ individual tools exposed
• Each operation has dedicated tool

Test single mode:

azmcp server start --mode single

Verify:

• Server starts successfully
• Exactly 1 tool exposed
• Internal routing functional

Test 2: Read-Only Mode via CLI Flag

Enable read-only:

azmcp server start --read-only

Verify:

• Server starts in read-only mode
• Read operations allowed
• Write operations blocked

Combine with mode:

azmcp server start --mode namespace --read-only

Verify:

• Both flags applied correctly
• Namespace mode + read-only restrictions

Test 3: Namespace Filtering via CLI Flags

Filter to specific services:

azmcp server start --namespace storage --namespace keyvault

Verify:

• Only specified namespaces exposed
• Other services filtered out
• Can specify multiple --namespace flags

Test 4: Tool Filtering via CLI Flags

Filter to specific tools:

azmcp server start --tool azmcp_subscription_list --tool azmcp_group_list

Verify:

• Only specified tools exposed
• Automatically switches to all mode
• Can specify multiple --tool flags

---

Distribution Method 3: .NET Tool (DNX)

Objective: Test CLI flag-based configuration with .NET tool installation

Installation:

dotnet tool install --global Azure.Mcp

Test 1: Server Mode via CLI Flags

Test namespace mode:

azmcp server start --mode namespace

Verify:

• .NET tool version works identically to NPM version
• All mode flags functional
• Tool exposure matches expectations

Test 2: Configuration File Support

Create config file (azmcp.config.json):

{
  "serverMode": "namespace",
  "readOnly": true,
  "enabledServices": ["storage", "keyvault"]
}

Start with config:

azmcp server start --config azmcp.config.json

Verify:

• Config file loaded successfully
• Settings from file applied
• CLI flags override config file if specified

---

Distribution Method 4: IntelliJ IDEA Plugin

Objective: Test MCP Server configuration dialog for mode configuration

Installation:

1. IntelliJ IDEA → Settings/Preferences → Plugins
2. Search "Model Context Protocol (MCP)"
3. Install plugin
4. Restart IDE

Configuration Dialog Fields:

Based on IntelliJ's MCP configuration dialog:

• Name: Display name for the MCP server (e.g., "Azure MCP Server")
• Command: Executable command (e.g., npx or azmcp)
• Arguments: Command-line arguments including mode flags
• Environment variables: Key-value pairs for configuration
• Working directory: Optional working directory path

Test 1: Configure via MCP Server Dialog

Steps:

1. Navigate to File → Settings
2. Go to Tools → AI Assistant → Model Context Protocol (MCP)
3. Click the + icon to open "New MCP Server" dialog
4. Enter the following values:
   • Name: Azure MCP Server
   • Command: npx
   • Arguments: -y @azure/mcp@latest server start
5. Leave other fields blank for default configuration
6. Click OK to save
7. Restart AI Assistant or IDE if prompted

Verify:

• Dialog opens correctly with all fields
• Can enter server name
• Command and arguments fields accept input
• Configuration saves successfully
• Server appears in MCP server list
• Server starts when AI Assistant is opened

Test 2: Configure Namespace Mode via Arguments

Steps:

1. Open MCP Server configuration dialog
2. Edit existing "Azure MCP Server" entry
3. Update Arguments field to:

   -y @azure/mcp@latest server start --mode namespace
   

4. Click OK and restart
5. Test tool availability

Verify:

• Arguments field accepts mode flag
• Server restarts with new configuration
• Namespace mode applied (~40-50 tools)
• Tools grouped by service namespace

Test 3: Configure Read-Only Mode via Arguments

Steps:

1. Edit "Azure MCP Server" configuration
2. Update Arguments field to:

   -y @azure/mcp@latest server start --mode namespace --read-only
   

3. Click OK and restart
4. Test read operation: "List my storage accounts"
5. Test write operation: "Create a storage account"

Verify:

• Read-only flag accepted in arguments
• Read operations work
• Write operations blocked
• Error messages displayed

Test 4: Configure Service Filtering via Arguments

Steps:

1. Edit "Azure MCP Server" configuration
2. Update Arguments field to:

   -y @azure/mcp@latest server start --namespace storage --namespace keyvault
   

3. Click OK and restart
4. Test allowed service: "List storage accounts"
5. Test filtered service: "List Cosmos DB accounts"

Verify:

• Multiple namespace flags work
• Only specified services exposed
• Allowed services functional
• Filtered services not accessible

Test 5: Configuration Persistence

Steps:

1. Configure server with specific mode and flags
2. Close IntelliJ completely
3. Reopen IntelliJ
4. Navigate to MCP settings
5. Verify configuration still present
6. Start AI Assistant and test

Verify:

• Configuration persists across IDE restarts
• No need to reconfigure
• Server starts with saved settings

//TODO: Add VS, CLaude, Docker instructions.

What to Report for UI Issues

When logging UI configuration issues, include:

• Distribution method (VS Code ext, NPM, .NET tool, etc.)
• IDE/Platform version
• Screenshot of settings UI
• Expected vs actual behavior
• Steps to reproduce
• Settings file content (if applicable)
• Server startup logs
• Whether restart was performed

Verification Steps

After installation on any platform/IDE, verify:

1. Server Status

# Check if server is running
# In VS Code: Check Output > MCP: Azure MCP Server
# Look for: "Server started successfully"

2. Tool Discovery

# In GitHub Copilot Chat (Agent mode):
# Ask: "What Azure MCP tools are available?"
# Verify: Tools are listed

3. Authentication

# Login to Azure
az login

# Test authentication
# Ask Copilot: "List my Azure subscriptions"

4. Basic Functionality

# Test a simple command
# Ask Copilot: "List my Azure resource groups"
# Verify: Resource groups are returned

Related Resources

• Main Bug Bash Guide
• Troubleshooting Guide
• Installation Guide
• Report Issues

Next Steps: After completing installation testing, proceed to Scenario Testing.