It begins with synesthesia analog clocks. Replacing each number is its color. Non-synesthetes are so familiar with the format of clocks that they can still read them, and over time are brainwashed into also associating numbers with color. Then, digital clocks will be produced with changing blocks of color.
"Periwinkle: white-dark green?! Oh shit I'm gonna be late!"
Colorblind people, unable to roll with the times, will be late for work constantly and get laid off, left to decay huelessly into the past.
At stoplights you go on 6, yield on 12, and stop on 3. (Did you know Drake is really running through the light green with his woes?)
Mathematicians will carry around palettes and sell their calculations as art. We will display formulas on the walls above our bunkbeds. And through all of this we have learned not only to associate numbers with colors, but colors with numbers. So when I describe the sunset to you I might say "1-1-12-12-10-3" and your eyes will fill with tears, because that is a particularly stunning sky.