category: til date: ‘2018-06-04’ featured_image: 3 kernelspec: display_name: Python 3 (ipykernel) language: python name: python3 language_info: codemirror_mode: name: ipython version: 3 file_extension: .py mimetype: text/x-python name: python nbconvert_exporter: python pygments_lexer: ipython3 version: 3.9.7 redirect: makeitpop-intro tags:

• python colormaps title: Introducing makeitpop, a tool to perceptually warp your data!

Introducing makeitpop, a tool to perceptually warp your data!"¶

When I was in graduate school, I tended to get on my soapbox and tell everybody why they should stop using Jet and adopt a “perceptually-flat” colormap like viridis, magma, or inferno.

Surprisingly (ok, maybe not so surprisingly) I got a lot of pushback from people. Folks would say “But I like jet, it really highlights my data, it makes the images ‘pop’ more effectively than viridis!”.

Unfortunately it turns out that when a colormap “makes your data pop”, it really just means “warps your perception of the visualized data so that you see non-linearities when there are none”. AKA, a colormap like Jet actually mis-represents the data.

But what does this really mean? It’s difficult to talk and think about coor - especially when it comes to relating color with objective relationships between data. Rather than talking about colormaps in the abstract, what if we could visualize the warping that is performed by colormaps like Jet?

In this post I’ll show that this is possible! Introducing makeitpop.

What does makeitpop do?¶

Makeitpop lets you apply the same perceptual warping that would normally be accomplished with a colormap like Jet, but applies this warping to the data itself! This lets us get the same effect with a nice linear colormap like viridis!

For example, let’s take a look at the image demo in matplotlib. In it, we create two blobs that are meant to be visualized as an image. We’ll visualize this with our old friend viridis.

Hmmm, not too bad...but it’s a bit boring, no? Why can’t we make it snazzier? I know, let’s use Jet!

Oooh now that’s what I’m talking about. You can clearly see two peaks of significant results at the center of each circle. Truly this is fit for publishing in Nature.

But...as you all know, this data only looks better because we’ve used a colormap that distorts our perception of the underlying data.

Let’s illustrate this by making it pop!

Excellent! We’re using a nice, perceptually-flat colormap like viridis, but we’ve attained an effect similar to the one that Jet would have created!

Now let’s visualize all three next to each other so that we can see the total effect:

Let’s see it in the real world¶

Thus far I’ve been using toy examples to illustrate how makeitpop works. Let’s see how things look on an actual dataset collected in the wild.

For this, we’ll use the excellent nilearn package. This has a few datasets we can download to demonstrate our point. First we’ll load the data and prep it:

Next, we’ll create a “popped” version of the data, where we apply the non-linear warping properties of Jet to our data, so that we can visualize the same effect in linear space.

Now, I’ll plot the results for each.

• First, we’ll see the raw data on a linear colormap. This is the way we’d display the data to show the true underlying relationships between datapoints.
• Next, we’ll show the same data plotted with Jet. See how many more significant voxels there are! (/s)
• Finally, we’ll plot the “popped” data using a linear colormap (viridis). This accurately represents the underlying data, but the data itself has been distorted!

As you can see, different kinds of results show up when your perception of the data is affected by the colormap.

How does this work?¶

OK, so what is the black voodoo magic that makes it possible to “make your data pop”?

It all comes down to your visual system. I won’t go into a ton of detail because Nathaniel Smith and Stefan van der Walt already gave a great talk about this, however here is a lay-person’s take:

When we use color to represent data, we are mapping a range of data values onto a range of color values. Usually this means defining a min / max for our data, then mapping data values linearly from 0 to 1, and finally mapping those values onto RGB values in a colormap.

Implicit in this process is the idea that stepping across our space in the data equates to an equal step in our perception of the color that is then chosen. We want a one-to-one mapping between the two.

Unfortunately, this isn’t how our visual system works.

In reality, our brains do all kinds of strange things when interpreting color. They are biased to detect changes between particular kinds of colors, and biased to miss the transition between others.

Jet uses a range of colors that highlight this fact. It transitions through colors such that linear changes in our data are perceived as nonlinear changes when we look at the visualization. That’s what makes the data “pop”.

Perceptual “delta” curves¶

You can determine the extent to which a colormap “warps” your perception of the data by calculating the “perceptual deltas” as you move across the values of a colormap (e.g. as you move from 0 to 1, and their corresponding colors).

These deltas essentially mean “how much is the next color in the colormap perceived as different from the current color?” If your colormap is perceptually flat, the delta will be the same no matter where you are on the range from 0 to 1.

Let’s see what the deltas look like for Jet:

Oops.

As you can see, Jet does not have a flat line for perceptual deltas. Each “jump” you see above is a moment where Jet is actually mis-representing differences in the data. For shame, Jet.

Now let’s see what this looks like for viridis:

Ahhh, sweet, sweet linear representation of data.

In case you’re curious, here are the “perceptual deltas” for several colormaps. In this case, I’ve centered them and scaled each by the variance of the largest colormap, so that they are easier to compare.

We can even warp 1-dimensional data!¶

Let’s see how this principle affects our perception with a different kind of visual encoding. Now that we know these perceptual warping functions, we can get all the data-warping properties of jet, but in one dimension!

Here’s a line.

Ew. Boring.

Now, let’s make it pop! We’ll loop through a few colormaps, applying its color warping function to the y-axis of our line as we step through it.

As you can see, data looks much more interesting when it’s been non-linearly warped! It looks particularly striking when you see it on a 1-D plot. This is effectively what colormaps such as Jet are doing in 2 dimensions! We’re simply bringing the fun back to 1-D space.

Let’s see how it looks on some scatterplots. We’ll plot the raw data in the background in grey, and the “popped” data in front in color. Notice how some colormaps distort the y-values more than others.

So what should we do?¶

The reason that I wrote this blog post (and made this silly package) is to illustrate what we’re really doing when we use a colormap like Jet, and to highlight the importance of using a perceptually-flat colormap. Sure, we want to choose the visualization that best-makes our point, but a colormap like Jet is actively mis-representing your data. You’d never consider changing the raw data values so that an effect popped out, and you’d never alter the y-values of a scatterplot so that something shows up. Well, this is perceptually what you’re doing when you visualize 2-D data with Jet.

Here are a few things to keep in mind moving forward:

• Don’t use Jet
• If you review a paper or are an editor for a journal, consider asking authors to use a perceptually flat colormap (this is usually just a matter of changing cmap='viridis'!)
• Be aware of the effects that color has on the point you’re trying to make. Perceptual warping is just one of many potential issues with choosing the right color.

Wrapping up¶

I hope that this post has been a fun and slightly informative take on the nuances of colormaps, and the unintended effects that they might have.

So, tl;dr:

• Jet (and many other colormaps) mis-represent your perception of the data
• Perceptually flat colormaps like Viridis, Magma, Inferno, or Parula minimize this effect
• You can calculate the extent to which this mis-representation happens as you move along the colormap
• We can then use this function to distort data so that the data itself contains this mis-representation
• But doing so would be super unethical, so in the end you should stop using jet and use a perceptually-flat colormap like viridis.

If you’d like to check out the makeitpop package, see the GitHub repo here. In addition, all of the examples in this post are runnable on Binder! You can launch an interactive session with this code by clicking on the Binder button at the top of this page!

Addendum: Ok but how does makeitpop actually work?¶

In this section I’ll describe the (admittedly hacky) way that I’ve written makeitpop. As I mentioned before, all the makeitpop code is on GitHub and Pull Requests are more than welcome to improve the process (I’m looking at you, “histogram matching” people!)

Here’s what makeitpop does:

1. Collects a list of the “perceptual deltas”. These are calculated from the equations given in viscm, which was released as a part of the original work that created viridis.
2. Centers each colormap’s deltas at 0.
3. Scales each colormap’s deltas by the largest variance across all colormaps. This is to make sure that warping the data is done with the relative differences of each colormap in mind.
4. When makeitpop is called, the function then:
   1. Scales the input data linearly between 0 and 1
   2. Calculates the point-by-point derivative for linearly spaced points between 0 and 1 (the derivative is the same for all points here).
   3. Multiplies each derivative by the scaled perceptual deltas function, plus an extra scaling factor that accentuates the effect.
   4. Adds each value to the scaled input data.
   5. Un-scales the altered input data so that it has the same min/max as before.

There are plenty of ways you could do this more effectively (for example, by matching empirical CDFs and using linear interpolation to map the delta function of one colormap onto the delta function for another colormap). If you’d like to contribute or suggest something, feel free to do so! However, I’m just creating this package to highlight an idea, and think this approach gets close enough with relatively little complexity.

Some extras¶

Here’s a viz that will let you visualize how different colormaps distort data. We’ll show a gradient of linearly-spaced values, both using a warping colormap such as “jet” and a linear colormap like “vidiris”. Then, we’ll “pop” the data and re-visualize with viridis.