Thanks for your answers. I think my mistake is trying to use a depth map for an organic (non-flat/non-uniform) surface. Judging from the tons of tutorials for 3d engines that only ever cover flat walls, flat floors or hard surface models, this seems to be what displacement maps are supposed to be used for. In my case, I tried to use the depth pass to make the bumps of a stone wall more believable than the normal map alone does. However, the wall model already is pretty bumpy, so the parallax effect ends up just stretching/warping the texture on very angular surfaces, and no visual displacement effect is created:
[
]
no height map:

If I set the depth to a very low level (around .05 as you suggested), nothing changes visually, if I crank it higher, distrotion occurs.
However, by using the cavity map as height map, so only the recesses of the model get a displacement effect, I was finally able to get little more depth all around. Thanks again for your input!