The answer to this question depends where you start from and where you want to end up.
Axis Communications carried out a very interesting exercise last year to answer this very question and the results were quite revealing. In a new installation (i.e. Cat 5 is installed as part of the building process) IP is always more cost effective and on other occasions for camera counts above 40 cameras, IP is still more cost effective.
We have also seen this in a number of projects we have undertaken with partners because these savings are brought about by such things as more efficient cabling (One cable to carry everything rather than multiple cables.), more cost effective use of storage (no wasted channels and scalability etc.) and the strength of using the larger IT market forces – Cat-5 cable for instance is a lot cheaper than Coax as far more of it is produced.
Again this all comes down to planning the correct system and either utilising new technologies to their best advantage or understanding how best to migrate existing infrastructure to meet the end users needs for functionality, future proofing and budget.
See comments below for a discussion thread that discusses the Axis report kindly posted by Carl Lindgren on a previous Controlware blog post and reproduced here.