Ok so, explanation on how I’m gonna theoretically do cel shading (I know general shader stuff but I’m entering into an area that if I take the wrong step here its gonna be like 2fps)

So, my game is completely pathtraced besides certain objects being cel shaded. Due to this, there’s no default lighting values I can clip the shading to and have it look natural. I need something that can dynamically decide what is gonna be the light and shaded parts of an object every frame.

1. First is to render the object in the path tracer without any textures.
2. Next is to, per object, render out a mask of just the mesh.
3. Third is to then use the mask on the original rendered image, and figure out the average color to use as the seperator between the light and dark spots (This can be additionally adjusted before being sent to the shader)
4. Lastly is to push this final color to the cel shader. The shader will then check the pathtraced pixels again and determine whether its lighter or darker than the final color, allowing it to be dynamically cel shaded.

Theres still a few additional things to implement. One is additionally checking for the average color value in the bright and dark areas after determining the midtone so that the light and dark areas can be tinted correctly. This doesn’t technically need to be done in the same frame, nor does the average color need to be exactly checked every frame- as this won’t change too often. Due to noise in the pathtracer, its also best that these checked values be Lerped over time.

So this is just my notes to take before my brain explodes. I have no clue if this is gonna tank performance on implementation. If this is gonna be on like 100 objects then I’m gonna have to downsample textures, schedule each object for checks on different frames, ect. I also gotta completely rewrite my ShaderGraph Min-Max shader in standard hlsl with these additional parameters. Whew its gonna be a lot.

Fun fact, this dynamic cel shading was stupidly easy to do for Min-Max. I’d just sample the nearest reflection probe at its lowest LOD to get the average midtone.