Picking by Color

In which you learn an alternative method for determining what object is under the mouse.

Inverse transformations, raycasting, and collision detection involve some heavy mathematics. Sometimes all you want to do is figure out what object was clicked on, and you don't need an exact location of the mouse in 3D space. In such cases, you may be able to take this very different approach to the problem of picking: render each object in a unique color and, on a mouse event, use the color under the mouse to identify the clicked object.

Click on the spheres in this renderer to change which is selected:

This renderer has two different render methods and two different shaders. The default method shades the spheres with diffuse lighting. The other shades the spheres with a unique shade of red. Toggle the checkbox to see this second method. No lighting is applied; the shade of red must be constant across a sphere.

On a mouse event, the scene is rendered in red. WebGL's readPixels method is called in order to get the color of the pixel under the mouse, and the pixel's red component is mapped to the index of the sphere. All together, the listener might look something like this:

window.addEventListener('pointerup', event => {
  // Use redness under mouse to determine clicked sphere.
  renderRed();
  const pixel = new Uint8Array(4);
  gl.readPixels(event.clientX, canvas.height - event.clientY, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);
  const redness = pixel[0];
  selectedIndex = convertRedToIndex(redness);

  render();
});

The Uint8Array is a type provided by JavaScript for holding raw bytes. The methods renderRed and convertRedToIndex are not provided as they will vary depending on how many objects you have. Each object must be assigned a unique color. If you have more than 256 objects, you'll need to use more than just the red channel.