Originally, ColorMatrix supported four matrices:

0. Rec.709
1. FCC
2. Rec.601
3. SMPTE 240M

With four matrices, a flattened conversion table was indexed using:

1

source * 4 + dest

That was correct for a 4×4 table.

In ColorMatrix 2.6, Rec.2020 was added as a fifth matrix. The coefficient array was expanded to five entries, and the conversion table consequently became 5×5. The matrices are stored in this order:

1
2
3
4
5

0 = Rec.709
1 = FCC
2 = Rec.601
3 = SMPTE 240M
4 = Rec.2020

The coefficient-generation code iterates through all five source matrices and all five destination matrices, incrementing one flattened table index after every pair. In other words, the table is now laid out with a row width of five.^1

But the constructor still contains:

1

modei = source == dest ? -2 : source * 4 + dest;

It was never changed to multiply by five.^2

What happens with a Rec.601 → Rec.709 conversion

The mode parser correctly translates^2:

1
2

Rec.601 -> source = 2
Rec.709 -> dest   = 0

ColorMatrix then calculates:

1
2

buggy index = 2 × 4 + 0
            = 8

But in the new 5×5 coefficient table:

1
2

index 8 = source 1, destination 3
        = FCC -> SMPTE 240M

The correct index for Rec.601→Rec.709 is:

1
2

correct index = 2 × 5 + 0
              = 10

So the result is:

GetFrame() then loads its working coefficients directly from yuv_convert[modef], so once modef has incorrectly become 8, the wrong matrix is genuinely applied to every frame.^2

The bug was introduced with Rec.2020 support

The repository history identifies a February 22, 2018 commit named “add Rec.2020 support.” That commit changed:

1

YUV_COEFFS_LUMA_COUNT

from 4 to 5, but the same diff shows that the old:

1

source * 4 + dest

calculation was left unchanged. ^3

That explains the chronology:

• ColorMatrix 2.5’s four-wide table and source*4+dest were internally consistent.
• ColorMatrix 2.6 expanded the table to five columns without updating all indexing logic.
• Your February 2020 Hybrid version was already using code descended from this broken 2018 release.
• Current Hybrid still carries the affected DLL.

The proper source fix

The principal correction is:

1
2
3

modei = source == dest
    ? -2
    : source * YUV_COEFFS_LUMA_COUNT + dest;

However, changing only that line is not sufficient for the current GitHub master.

The hints/D2V path also retains old four-wide row offsets:

1
2
3

4  + dest
8  + dest
12 + dest

Those need to become five-wide calculations. A safer replacement is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

int ColorMatrix::findMode(int color)
{
    int src;

    if (color == 1)
        src = 0;                    // Rec.709
    else if (color == 4)
        src = 1;                    // FCC
    else if (color == 5 || color == 6)
        src = 2;                    // Rec.601
    else if (color == 7)
        src = 3;                    // SMPTE 240M
    else
        return inputFR != outputFR ? -2 : -1;

    if (src == dest)
        return inputFR != outputFR ? -2 : -1;

    return src * YUV_COEFFS_LUMA_COUNT + dest;
}

The stale hint offsets and old SIMD mode-number groups are visible together in the current source. ^2

SIMD dispatcher also needs updating

Under the corrected five-wide numbering, the existing SIMD groups must change to:

1
2
3
4

conv1: 1, 2, 3, 16, 17
conv2: 5, 8, 10, 13, 15
conv3: 7
conv4: 11

Alternatively, a corrected build could initially force the C route until SIMD dispatch is repaired.

This matters because the correct Rec.601→Rec.709 mode becomes index 10. The current SIMD dispatcher does not recognize mode 10, returns NULL, and the YV12 processing code immediately calls the returned pointer. A one-line index fix in an SIMD-enabled build could therefore turn the color error into a crash. ^2

The CTS2 diagnostic mapping also needs to be updated or, preferably, replaced with labels generated from:

1
2

source = mode / YUV_COEFFS_LUMA_COUNT;
dest   = mode % YUV_COEFFS_LUMA_COUNT;