Mutation testing — injecting bugs to verify tests actually catch them

Advanced5 min read·eng-17-015

Concept

Mutation testing — a technique that INJECTS BUGS into your code (mutations) and then runs your test suite to see if any tests catch them. Tests that don't catch mutations are considered weak or missing.

What a "mutation" is: A small, targeted code change that represents a real type of bug:

Change + to -.
Change > to >=.
Change true to false.
Remove a return value.
Delete a conditional check.
Change === to !==.

The mutation testing loop:

Take a source file.
Create a "mutant" — a copy with ONE small change (e.g., change > to <).
Run the test suite against the mutant.
If a test fails: The mutant is KILLED. Good — your tests caught the bug.
If all tests pass: The mutant SURVIVED. Your tests are missing coverage for that code path.

Mutant survived = your tests are weak: If changing > to < in a comparison doesn't break any test, your tests don't verify that the comparison is correct.

PHP tool: Infection — the PHP mutation testing framework. composer require --dev infection/infection then ./vendor/bin/infection.

Mutation Score Indicator (MSI): killedMutants / totalMutants. 80%+ is a common target.

Mutation testing vs code coverage: Coverage tells you WHICH lines were run. Mutation testing tells you WHETHER those lines are tested effectively. 100% line coverage can have 0% mutation score if the assertions are weak.

Code Example

bash

# Install Infection
composer require --dev infection/infection

# Run mutation testing
./vendor/bin/infection --threads=4

# Output:
# 35 mutations generated
# 28 mutants killed
# 7 mutants survived (MSI: 80%)

php

<?php
// SOURCE CODE being tested
class PasswordStrengthChecker
{
    public function isStrong(string $password): bool
    {
        return strlen($password) >= 8
            && preg_match('/[A-Z]/', $password)
            && preg_match('/[0-9]/', $password)
            && preg_match('/[^a-zA-Z0-9]/', $password); // requires special char
    }
}

// WEAK TEST — high coverage but low mutation score
class PasswordStrengthCheckerTest extends \PHPUnit\Framework\TestCase
{
    public function test_strong_password(): void
    {
        $checker = new PasswordStrengthChecker();
        $this->assertTrue($checker->isStrong('Pass1234!')); // tests the "true" path
        // But never tests the "false" path variants!
    }
}

// Infection generates mutants like:
// Mutant 1: strlen($password) >= 8 → strlen($password) > 8
// Mutant 2: strlen($password) >= 8 → strlen($password) >= 9
// Mutant 3: return true && ... → return false && ...
// All these mutants SURVIVE (no test catches them) — MSI is low!

// STRONG TEST — kills more mutants
class PasswordStrengthCheckerTest extends \PHPUnit\Framework\TestCase
{
    private PasswordStrengthChecker $checker;
    protected function setUp(): void { $this->checker = new PasswordStrengthChecker(); }

    // Strong passwords
    public function test_strong_password_passes(): void { $this->assertTrue($this->checker->isStrong('Pass1234!')); }

    // Each rule tested individually:
    public function test_7_chars_is_weak(): void      { $this->assertFalse($this->checker->isStrong('Pass12!')); }
    public function test_8_chars_is_minimum(): void   { $this->assertTrue($this->checker->isStrong('Pass123!')); }
    public function test_no_uppercase_is_weak(): void { $this->assertFalse($this->checker->isStrong('pass1234!')); }
    public function test_no_number_is_weak(): void    { $this->assertFalse($this->checker->isStrong('Password!')); }
    public function test_no_special_is_weak(): void   { $this->assertFalse($this->checker->isStrong('Password1')); }
}

// Now Infection's mutants are killed:
// Mutant: >= 8 → > 8 → caught by test_8_chars_is_minimum
// Mutant: >= 8 → >= 9 → caught by test_8_chars_is_minimum
// Mutant: special char check removed → caught by test_no_special_is_weak

// infection.json5 configuration:
/*
{
    "source": { "directories": ["app"] },
    "minMsi": 80,       // fail CI if MSI drops below 80%
    "minCoveredMsi": 90,
    "threads": 4
}
*/

PreviousRegression — a bug reintroduced by a change; regression tests catch themVocabulary — Testing NextCI/CD — Continuous Integration and Continuous Deployment defined preciselyVocabulary — DevOps & Infrastructure