Matrix Strategy for Multi-Environment Testing

Matrix Strategy in GitHub Actions

Matrix strategy allows you to run jobs across multiple configurations simultaneously, perfect for cross-browser testing, multi-platform builds, and testing against different dependency versions.

Basic Matrix Concept

A matrix creates multiple job runs from a single job definition by iterating over configured values.

name: Matrix Testing

on: [push]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node: [18, 20, 22]
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js ${{ matrix.node }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test

This creates 9 jobs (3 OS × 3 Node versions).

Cross-Browser Testing with Playwright

Real-world example for testing the blog across browsers:

name: Cross-Browser E2E Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        browser: [chromium, firefox, webkit]
        shard: [1, 2, 3, 4]
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Install Playwright Browsers
        run: npx playwright install --with-deps ${{ matrix.browser }}
      
      - name: Run E2E tests
        run: npx playwright test --project=${{ matrix.browser }} --shard=${{ matrix.shard }}/4
        env:
          E2E_BASE_URL: ${{ secrets.E2E_BASE_URL || 'http://localhost:4321' }}
      
      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report-${{ matrix.browser }}-${{ matrix.os }}-shard-${{ matrix.shard }}
          path: playwright-report/
          retention-days: 30

Including and Excluding Combinations

Control which matrix combinations run:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest, macos-latest]
    node: [18, 20, 22]
    include:
      # Add specific combination
      - os: ubuntu-latest
        node: 22
        experimental: true
    exclude:
      # Skip Windows with Node 18
      - os: windows-latest
        node: 18
      # Skip macOS with Node 18
      - os: macos-latest
        node: 18

steps:
  - name: Run ${{ matrix.experimental && 'experimental' || 'stable' }} tests
    run: npm test

Dynamic Matrix from JSON

Generate matrix from repository file or API:

jobs:
  prepare:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
      
      - name: Set matrix
        id: set-matrix
        run: |
          MATRIX=$(cat .github/test-matrix.json | jq -c .)
          echo "matrix=$MATRIX" >> $GITHUB_OUTPUT
  
  test:
    needs: prepare
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJson(needs.prepare.outputs.matrix) }}
    
    steps:
      - name: Test ${{ matrix.browser }} on ${{ matrix.device }}
        run: echo "Testing ${{ matrix.browser }} on ${{ matrix.device }}"

.github/test-matrix.json:

{
  "browser": ["chrome", "firefox", "safari"],
  "device": ["desktop", "mobile", "tablet"]
}

Testing Multiple Dependency Versions

Test against different package versions:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        astro-version: ['4.0.0', '4.5.0', '5.0.0']
        playwright-version: ['1.40.0', '1.45.0', 'latest']
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install Astro ${{ matrix.astro-version }}
        run: npm install astro@${{ matrix.astro-version }}
      
      - name: Install Playwright ${{ matrix.playwright-version }}
        run: npm install @playwright/test@${{ matrix.playwright-version }}
      
      - name: Run tests
        run: npm test

Parallel Test Sharding

Split tests across multiple runners for faster execution:

name: Parallel Test Execution

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4, 5, 6, 7, 8]
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run shard ${{ matrix.shard }}/8
        run: npx playwright test --shard=${{ matrix.shard }}/8
      
      - name: Upload blob report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 1
  
  merge-reports:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Download all reports
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
      
      - name: Merge reports
        run: npx playwright merge-reports --reporter html ./all-blob-reports
      
      - name: Upload merged report
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/

Real Blog Testing Matrix

Complete example for the blog:

name: Blog E2E Test Matrix

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM

jobs:
  e2e:
    runs-on: ${{ matrix.os }}
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest]
        browser: [chromium, firefox, webkit]
        test-suite:
          - name: smoke
            pattern: 'tests/**/*.smoke.spec.ts'
          - name: regression
            pattern: 'tests/**/*.spec.ts'
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Install Playwright
        run: npx playwright install --with-deps ${{ matrix.browser }}
      
      - name: Build site
        run: npm run build
      
      - name: Run ${{ matrix.test-suite.name }} tests on ${{ matrix.browser }}
        run: npx playwright test ${{ matrix.test-suite.pattern }} --project=${{ matrix.browser }}
        env:
          E2E_BASE_URL: 'http://localhost:4321'
      
      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ${{ matrix.test-suite.name }}-${{ matrix.browser }}-${{ matrix.os }}
          path: |
            playwright-report/
            test-results/

Conditional Matrix Values

Use conditions to customize matrix behavior:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest, macos-latest]
    include:
      - os: ubuntu-latest
        upload-coverage: true
      - os: windows-latest
        upload-coverage: false
      - os: macos-latest
        upload-coverage: false

steps:
  - name: Run tests with coverage
    run: npm run test:coverage
    if: matrix.upload-coverage
  
  - name: Upload coverage
    if: matrix.upload-coverage
    uses: codecov/codecov-action@v4

Matrix Output Aggregation

Collect results from all matrix jobs:

jobs:
  test:
    strategy:
      matrix:
        browser: [chrome, firefox, safari]
    steps:
      - name: Run tests
        id: test
        run: npm test
      
      - name: Set status
        id: status
        run: echo "result=${{ steps.test.outcome }}" >> $GITHUB_OUTPUT
  
  summary:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Generate summary
        run: |
          echo "## Test Results" >> $GITHUB_STEP_SUMMARY
          echo "All matrix jobs completed" >> $GITHUB_STEP_SUMMARY

Best Practices

1. Use fail-fast: false - Continue other matrix jobs if one fails
2. Set timeouts - Prevent hanging jobs
3. Cache dependencies - Speed up matrix jobs
4. Limit matrix size - Balance coverage vs. execution time
5. Use sharding - Parallelize large test suites
6. Name artifacts clearly - Include matrix variables in names
7. Combine with required checks - Ensure all matrix jobs pass

Common Patterns

# Mobile + Desktop testing
strategy:
  matrix:
    device:
      - { name: 'Desktop Chrome', viewport: '1920x1080', browser: 'chromium' }
      - { name: 'iPhone 13', viewport: '390x844', browser: 'webkit' }
      - { name: 'Pixel 5', viewport: '393x851', browser: 'chromium' }

steps:
  - name: Test on ${{ matrix.device.name }}
    run: |
      npx playwright test \
        --project=${{ matrix.device.browser }} \
        --viewport-size=${{ matrix.device.viewport }}

Monitoring Matrix Performance

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - name: Record start time
        id: start
        run: echo "time=$(date +%s)" >> $GITHUB_OUTPUT
      
      - name: Run tests
        run: npx playwright test --shard=${{ matrix.shard }}/4
      
      - name: Calculate duration
        run: |
          END=$(date +%s)
          DURATION=$((END - ${{ steps.start.outputs.time }}))
          echo "Shard ${{ matrix.shard }} took ${DURATION}s" >> $GITHUB_STEP_SUMMARY

Key Takeaways

✅ Matrix strategy runs jobs across multiple configurations
✅ Perfect for cross-browser and multi-platform testing
✅ Use fail-fast: false to continue on failures
✅ Combine with test sharding for faster execution
✅ Include/exclude specific combinations as needed
✅ Name artifacts with matrix variables for clarity
✅ Monitor performance across matrix dimensions