Shader download link. Read at the end of the post for instructions.
The visual highlighting system has been an ongoing challenge throughout the development of Software Inc. It seems very simple on the surface, and in the end it was, but I had to read up on shaders to finally land on a working solution.
The main problem was showing highlighted objects through other objects. Initially I used a modified version of the cartoon outline shader that comes with the Unity standard image effects asset. It doesn’t show through objects, but it creates a nice outline by offsetting vertices out from their normals and using front face culling. This has discontinuity problems with flat faces, since the edges will break off when offset, but it’s not really that noticeable.
In alpha 6, I decided that selected objects needed to be visible, because I often found myself confused by what furniture I had selected, when I changed floor. I went with the easy solution of just rendering the same mesh at half opacity with depth testing set to greater, so it only renders when something is in front of it. All in the same shader as the outline. However, it still looks confusing as you can see below. You can see how the floor of the room renders green through the tables.
The final solution required using the stencil buffer, which I hadn’t used before. I couldn’t easily draw the outline on top of everything otherwise, since it would require only depth testing each object with itself. Basically you render the entire mesh including outline to the stencil buffer, without actually drawing anything on screen. Then you remove the mesh itself from the stencil buffer by rendering only the mesh. Finally, you render the result of the two stencil passes without depth testing, so it appears above everything else. This is done per mesh, so they are drawn independently of each other, meaning there is no over-shadowing and no post processing involved, like blurring to increase the outline width.
Actually putting it to use is a whole other story, but basically each mesh I want to highlight has a highlight tag and when it is highlighted an extra shared material get’s added to its MeshRenderer, which uses the outline shader. Each object which can be highlighted inherits from a Selectable class, which in turn inherits from GameObject. A SelectorController class keeps track of highlighted Selectables and tells them if they need to traverse their transform hierarchies to add the highlight material to the highlight tagged meshes.
So to use the shader, simply add a material with the shader to the MeshRenderer of the object you want to highlight, on top of the material(s) it already uses.