Stylized rendering part the first: shadow extraction

In which some hacks are better than others, you can create perfectly flat shading by reversing the polarity of the neutron flow, and there is still life left in the grizzled old gunslinger known only as “shadow mapping.”

As you may recall, last year I showed my first attempt at creating a stylized rendering process in UE. It was mostly successful, but required some ridiculously horrific hacks, for which I may never be forgiven.

The current version still uses a lot of hacks, but they’re much less evil. One of the worst ones I used back then was the method of extracting shadows. It’s essential to this process to be able to treat shadows and lights separately: because the shapes that various elements make on screen is much more important than whether the shadow and light are coming from the same direction.

This is tricky to do in UE. In theory, there’s no reason UE couldn’t just write the results of shadow map calculations to a separate buffer, but that’s not what the renderer is designed to do. Fixing that directly would require forking the engine and creating a custom shading model, and while that may well be the right solution in the long run, it’s not something I want to do right now.

The solution I used last year was to set up two identical lights. One casts shadows, and one does not. Because a shadowed light would be created by multiplying the results of the light by the results of the shadow map, you can get just the shadow back out again by dividing the final result by the result of the light (luckily, UEs response to a divide by zero is to leave the pixel black, rather then error, or this wouldn’t work).

Not only is this solution an offense against the laws of god and man, it also sucks. Since there is no shadow data past the light’s terminator, the shadows are incompletely extracted, and you have to be very careful with the shadow direction to use them at all. I’m sure that Dante failed to include a circle of hell for this exact situation only because he could not conceive of an act so utterly depraved.

Luckily, I’ve since come up with a much better solution, one so simple that I smacked myself on the forehead for not thinking of it sooner. If the shadow is being combined with the light, and we want just the shadow, the best thing we can possibly do is make the value of the lit surface 1 for every pixel. That will give us a pure shadow pass, as multiplying anything by 1 is, well, that thing. We can do that very easily by simply pointing every normal along the surface directly at the light!

For point lights, that’s just the inverse of the method I showed last time for generating spherical normals: instead of having a normal that points away from a point, we make one that points towards it. That’s just a matter of reversing the polarity of the neu I mean order of the vector subtraction used to generate the new normal.

For directionals, it’s even easier--the forward vector of the light is just the reverse of the normal we want, so all we have to do is multiply it by -1 to get the correct normal:

I’m publishing the point or direction vector using the same method I used to publish the location of Jinx’s head. In this case, though, I don’t have to create an event track in Sequencer to publish the light’s location each frame. I can simply have the shadow-pass render element find any Little Bird lights that belong to it and publish their values. This function is called right before the render element renders the shadow pass:

Once again, we discover that the grizzled veteran known as the shadow map still has enough life in those old bones to shoot all the bad guys. (I’m imagining here that the shadow map is played by either Clint Eastwood or Sam Elliot.)

Shadow bias values are extremely effective here at making this shadow map usable, since they let us push the shadow away from the surface so that it doesn’t happen right at the terminator. Normally it’s not a problem for the surface to be fully shadowed at the terminator, because if you’re projecting the shadow from the same direction as the light both terminators will be in the same place, so the light won’t get suddenly cut off at the shadow projection’s terminator. But here we’re projecting shadows from a different direction, so pushing the shadow away from the terminator is important.

Approximate terminator of the shadow-casting light. Because this is a pure shadow pass and shadow map bias is being used heavily, shadow information continues past the terminator.

By contrast, a ray traced shadow is always perfectly accurate, and therefore would allow us no wiggle room at the terminator (The terminator is played by Arnold Schwartzenegger, obviously).

This also has the effect of simplifying the shadow shapes. Ray tracing is great for photorealistic uses but ironically all the old cheats I thought were dying out turn out to have a second career in nonphotorealistic rendering.

Next time we’ll take a look at the Little Bird background projection system.