The idea of discarding the blue channel of a normal map to get some free texture space is not new by any means. However, every now and then threads appear where people seem to struggle to get the result look like they want it to be. There are also some tutorials giving misleading informations.
The main reason to abuse the normalmap blue channel is that you get a greyscale map for free (in terms of memory), e.g. specular/height/emissive/variation etc. This will cut down the memory footprint or/and amount of texture lookups, depending on your setup, preferences, workflow.
A special technique where an extra texture channel is very beneficial is gradient mapping as described here: http://artisaverb.info/DitchingDiffuse.html
Texture and Material Setup
First of all create a tangent space normal map as usual (make sure it is normalized). Then replace the blue channel with a greyscale map of your choice and save the texture.
On UDK import make sure to set the compression to TC_Default. This will ensure a good quality for the blue channel (TC_Normalmap is radical on this part).
After importing the texture you need to uncheck "SRGB" in the texture properties (keep this in mind for the texture creation of the blue channel map).
Besides it is necessary to transfer the normal map from [0,1] to [-1,1] range. This can be done by changing the "Unpack Min" values for the red and green channel from 0 to -1. This setting does not have an instant effect, but can be triggered by applying changes to the material in which the texture is being sampled.
The material setup is pretty simple. Add a component mask node to mask off the blue channel and plug this into a DeriveNormalZ node to reconstruct the missing normal map channel.
Use the blue channel for whatever it is intented to. In my example it holds and ambient occlusion/cavity map which simply gets multiplied by the diffuse map.
Old UE3 builds might neither offer the unpack max/min options nor the DeriveNormalZ node. For those you can to do the [0,1] -> [-1,1] conversion (x*2-1) as well as the z component derivation (z=sqrt(1-(x²+y²)) manually in the material: