testrail – Coding skill for AI agents

TestRail Integration

Bidirectional sync between Playwright tests and TestRail test management.

Prerequisites

Environment variables must be set:

• TESTRAIL_URL — e.g., https://your-instance.testrail.io
• TESTRAIL_USER — your email
• TESTRAIL_API_KEY — API key from TestRail

If not set, inform the user how to configure them and stop.

> The TestRail MCP server is not auto-registered (issue #978). pw-testrail

> was removed from the plugin's .mcp.json because it failed to connect for every

> user (the plugin ships no node_modules). The testrail_* MCP tools used below,

> and the /pw:testrail command, will fail with "tool not found" until it is

> enabled manually — see the Integrations section of the plugin's CLAUDE.md

> (cd integrations/testrail-mcp && npm install, then register the server in your

> own user/project MCP config). Setting the env vars alone is not sufficient.

Capabilities

1. Import Test Cases → Generate Playwright Tests

/pw:testrail import --project <id> --suite <id>

Steps:

1. Call testrail_get_cases MCP tool to fetch test cases

2. For each test case:

• Read title, preconditions, steps, expected results
• Map to a Playwright test using appropriate template
• Include TestRail case ID as test annotation: test.info().annotations.push({ type: 'testrail', description: 'C12345' })

3. Generate test files grouped by section

4. Report: X cases imported, Y tests generated

2. Push Test Results → TestRail

/pw:testrail push --run <id>

Steps:

1. Run Playwright tests with JSON reporter:

   npx playwright test --reporter=json > test-results.json
   

2. Parse results: map each test to its TestRail case ID (from annotations)

3. Call testrail_add_result MCP tool for each test:

• Pass → status_id: 1
• Fail → status_id: 5, include error message
• Skip → status_id: 2

4. Report: X results pushed, Y passed, Z failed

3. Create Test Run

/pw:testrail run --project <id> --name "Sprint 42 Regression"

Steps:

1. Call testrail_add_run MCP tool

2. Include all test case IDs found in Playwright test annotations

3. Return run ID for result pushing

4. Sync Status

/pw:testrail status --project <id>

Steps:

1. Fetch test cases from TestRail

2. Scan local Playwright tests for TestRail annotations

3. Report coverage:

   TestRail cases: 150
   Playwright tests with TestRail IDs: 120
   Unlinked TestRail cases: 30
   Playwright tests without TestRail IDs: 15
   

5. Update Test Cases in TestRail

/pw:testrail update --case <id>

Steps:

1. Read the Playwright test for this case ID

2. Extract steps and expected results from test code

3. Call testrail_update_case MCP tool to update steps

MCP Tools Used

| Tool | When |

|---|---|

| testrail_get_projects | List available projects |

| testrail_get_suites | List suites in project |

| testrail_get_cases | Read test cases |

| testrail_add_case | Create new test case |

| testrail_update_case | Update existing case |

| testrail_add_run | Create test run |

| testrail_add_result | Push individual result |

| testrail_get_results | Read historical results |

Test Annotation Format

All Playwright tests linked to TestRail include:

test('should login successfully', async ({ page }) => {
  test.info().annotations.push({
    type: 'testrail',
    description: 'C12345',
  });
  // ... test code
});

This annotation is the bridge between Playwright and TestRail.

Output

• Operation summary with counts
• Any errors or unmatched cases
• Link to TestRail run/results