|
First published: Australian Financial Review Section: Review
In the Dark
Henry Ergas
Last year was not a good year for the business of equipping and arming Australia's defence forces − at least in PR terms. Not only did long−troubled projects like the guided missile frigate upgrade and the replacement of the RAAF's air defence command and control system make headlines for all the wrong reasons, but several more recent defence projects went off the rails.
The armed reconnaissance helicopter project slipped 18 months behind schedule and the much touted Wedgetail Airborne Early Warning and Control aircraft project will now be delivered a full two years late. And in a most extraordinary turn of events, the federal government openly canvassed cancelling the Sea Sprite helicopter acquisition after having spent more than $1 billion of taxpayers' money on the project. Its fate hangs in the balance pending the outcome of a review.
With so much bad news, and with memories still fresh of the Collins submarine program and the Jindalee Operational Radar Network project, it is perhaps inevitable that defence procurement has become synonymous with very large amounts of money being spent for questionable returns.
Public concern over the way Australia buy arms for the Defence Force is understandable, and appropriate. After all, equipping the Australian Defence Force (ADF) with new weapons systems will cost taxpayers over $5 billion this financial year, with a further $3.7 billion going to repair and maintain existing equipment. Together, the acquisition of new equipment and its maintenance, repair and upgrading (called "sustainment") account for nearly half of Australia's defence outlays, and absorb nearly 1 per cent of gross domestic product. The economic stakes associated with the performance of the Defence Materiel Organisation (DMO), which purchases and sustains weapons systems for the ADF, are substantial on any scale.
But it is more than money that is at stake. As a nation, we invest in defence to deter potential aggressors and, when necessary, to allow the ADF to fight and hopefully win. The men and women of the ADF who risk their lives in combat deserve the best equipment available. Excess costs and unnecessary delays in equipping the ADF can only compromise that goal and undermine our security.
Given this, it would seem reasonable to expect a healthy and well−informed debate about our defence procurement arrangements. But that is far from being the case.
Until the pioneering efforts of the Australian Strategic Policy Institute's Mark Thomson, there was little, if any, useful analysis of defence budgetary and economic issues in the public domain. Even with those efforts, we are still very far from having the depth of economic analysis − within Defence or outside of it − that is associated with the work of, say, the RAND Corporation and similar think tanks in the United States. Virtually no economic studies of weapons programs are undertaken in Australia, and those that are, are far from systematic and in any event, remain confidential.
As a result, public perceptions − and many aspects of public policy − are driven by newspaper headlines. But those headlines, no matter how sensational, do not provide a sensible basis for understanding and improving the way we buy and maintain weapons. For one thing, they ignore the many successes − such as the continuing timely delivery of new Abrams tanks for the army and the recent ahead−of−schedule delivery of a new fleet oiler for the navy: good news is no news when it comes to defence projects. But the real problem is there is little or no appreciation of the complexities and subtleties entailed in defence procurement.
Buying advanced weapons systems is not like buying laundry soap or paperclips − rather, it is an undertaking of exceptional difficulty, beset by uncertainties and risks. Contemporary weapons systems are among the largest and most sophisticated engineering projects our societies undertake, involving millions of interdependent parts, each technically demanding in its own right and then needing to inter−operate effectively and reliably under combat conditions.
Experience and common sense suggest the costs and time required to successfully complete projects of this kind will be difficult to predict. As a result, it is simply unrealistic to expect each project to be completed on time and to budget.
Commercial software projects provide a useful point of comparison − and one that is highly relevant, given the increasing role software plays in advanced weapons systems. According to one study, an astonishing 84 per cent of all software projects do not finish on time, on budget and with all features installed. The same study found that for those projects that were not cancelled, costs exceeded expectations by an average 189 per cent. These results are broadly confirmed by a study of technically complex software projects, which found that over 50 per cent failed completely and were abandoned.
Forecast errors are also common in large infrastructure projects. Even though road construction is a relatively standardised activity, a study of a large number of road projects found that costs were routinely underestimated, with the average gap between initial estimate and final result being in the order of 15 per cent. Forecast error on major rail projects (which admittedly tend to be more complex) was even greater, with an average underestimate of costs in the order of 40 per cent, while demand was overestimated by an average of 105 per cent, so that actual use was, on average, less than half that initially estimated. It is unsurprising that 40 per cent of the large engineering projects examined in one recent study were found to have "performed very badly", with fewer than half the projects surveyed ultimately meeting most of their stated objectives. That technically challenging projects should sometimes fail is unsurprising; but what does need probing is why the difficulties are so often severely underestimated at the outset.
Why is it costs and time are usually underestimated, rather than being underestimated in some cases and overestimated in others? Three factors seem to be involved.
First, overconfidence, which manifests in unduly optimistic forecasts, appears to be an inherent feature of managerial decision−making. While "gilding the lily" can arise from the self−interest that proponents of projects have in getting things under way, that is not the whole story. Equally important, though more subtle, is the fact individuals with high levels of self−confidence (not only relative to others, but compared to their own abilities) tend to do better in organisations, including in terms of rising to positions of authority. This embeds an "optimism bias" into even the most stringent commercial decision−making processes.
Second, as previously ignored difficulties emerge, the "tyranny of sunk costs" comes into play. Projects which would not have been undertaken if their total costs had been known at the outset, are not cancelled because the benefits of completing the project are thought to be greater than the marginal cost of completion. Total costs mount as each evaluation concludes spending a (relatively) little bit more will make the exercise
worthwhile.
Third and last, as the time taken to resolve problems causes project timetables to stretch out, pressures arise to adapt the systems being developed to take advantage of new technologies and to provide expanded functionality. This rework inevitably increases total project costs, especially in systems that rely on large numbers of closely integrated subsystems.
These features, which seem common to technologically complex projects, are greatly accentuated in the weapons acquisition process.
This is partly due to the technical characteristics of advanced weapons systems, notably their sheer complexity and need for extensive, real−time interoperability. But it is also because these systems, as well as being extremely complicated, are also − and perhaps uniquely − required to operate effectively in the face of hostile actions aimed at destroying them. Indeed, the performance of a weapons system is only usefully defined relative to the capabilities likely to be arrayed against it. This has important consequences. To begin with, it introduces significant additional complexities into the design stage. Understanding how a system will behave under combat conditions is extremely challenging and has results that are inherently difficult to predict. Equally, because weapons systems are designed to be employed in combat, they are of little use if they cannot evolve as adversaries' capabilities increase. The systems must, in other words, be capable of being modified in line with changes in the technologies and strategies that will be used against them.
The life cycle of weapons systems can be forbiddingly long. Developing a new system can take eight to 15 years, with even longer lags in individual cases. The Air Warfare Destroyer, for example, has been in planning since 1999, will enter production in 2009, will be in production through to about 2015, and will remain in the fleet for 30 or more years. That circumstances will change over that period is inevitable. These very long planning and deployment periods create a risk that a system will become obsolete even before it comes into service. Dealing with that risk involves allowing some degree of system redesign during development and acquisition. It also involves providing scope for extensive modification during the system's operating life. As a result, for most technically complex weapons systems, it's an illusion to believe that
specifications can ever be set in concrete before the procurement process is well under way. But the fact specifications are inevitably open−ended creates substantial difficulties for the process of buying and modifying weapons.
By their nature, contracts for complex weapons systems cannot exhaustively specify the full range of contingencies that will arise. Rather, significant elements will be determined only in the course of contract life, through the interpretation, addition, modification or deletion of contract conditions. This exposes the buyer − in Australia's case, DMO − and the seller to considerable risks.
From the buyer's perspective, the risk is that sellers will play what US defence economists call the "get well" game − in which the seller incurs losses in securing the initial contract, including by bidding artificially low for any early stages of work, but then uses changes in contract conditions to inflate costs and profits. Once the work is under way, the buyer is more often than not locked in to the chosen supplier. This gives the supplier a degree of market power that shortcomings in the original contract, and changes in requirements, allow it to exploit.
Risks arise for the seller too. In practice, the seller is dealing with a monopsonist − that is, a sole buyer. Once the seller has incurred significant costs in developing a system − costs it would not be able recoup should the project be cancelled − it too is vulnerable to being "held up". Specifically, the buyer may force changes that materially reduce the profits the seller might have secured, even though they do not drive the seller to the point where continuing is no longer financially viable.
The negotiations that inevitably occur during the life of weapons contracts are therefore fraught with risks and tensions. Each side has incomplete information as to the costs and benefits accruing to the other, and limited scope to credibly convey or signal that information. The fear of being exploited, or of forgoing gains one might have made by pushing that bit harder, cannot but colour and complicate the negotiating process. These conditions hardly make for efficient outcomes, at least relative to an idealised standard of what would be "first best". Rather, they are more likely to and often do, result in what seems like a mess, as costs escalate, delivery falls behind initial promises and adverse Audit Office reports catch the attention of politicians, media and public alike.
Can anything be done to avoid these messes, or at least minimise their extent? Long experience shows anything approaching the "first best" is simply unattainable −and touted solutions are partial at best. The 1987 defence white paper The Defence of Australia saw the solution as lying in a commitment to fixed−price contracts, competitively let. The commonwealth would thus be protected from unnecessary fiscal risk, with any cost overruns falling on whichever firm had signalled, through its bidding for the contract, that it was best placed to bear them.
Competitive bidding has a lot to recommend it. But the belief that fixed−price contracts is a panacea for the problems of defence procurement was not properly thought through.
In practice, it is difficult, if not impossible, for the commonwealth to simply walk away from major defence acquisition programs. As a result, there is an element of cost risk that the commonwealth cannot avoid − an element made all the greater by the fact that it is the commonwealth itself that often seeks modifications to
the systems it is acquiring.
At the same time, it's not usually efficient for the entirety of development risk to be borne by the seller. Rather, there are types of risk (such as those associated with volume uncertainty and with some changes in requirements) that are substantially under the control of the buyer. Even for other risks, such as technical failure, the commonwealth may have options for hedging that are not available to the seller. In such circumstances, forcing all risk onto sellers merely increases costs.
Finally, competitive allocation of fixed−price contracts creates pressures for the exhaustive definition of system specifications, often in highly detailed functional form (so as to allow the tender process to work). These detailed specifications then impede trade−offs from being made as new information comes to light, creating contract friction and further inefficiency. In the worst case, inflexible adherence to system specifications can result in technological opportunities being forgone and a second−best capability delivered as a consequence.
The 2003 Kinnaird review of defence procurement, partially recognising these deficiencies, brought in a two stage process, in which the commonwealth spends money in an initial stage to reduce technical and commercial uncertainty before making a firm commitment to purchase. Additionally, the options examined in the first stage must now include a "military off the shelf" option, relative to which Australian−only modifications need to be justified.
These proposals are eminently sensible. The paradox of defence procurement is that projects are easiest to cancel when they are in their early phases, but that is when there is the least information about their prospects. It makes good sense to spend resources so as to acquire additional information upfront, to avoid a lot of pain down the track.
But it's also important to recognise these proposals' limitations. The Kinnaird first stage has many similarities to design competitions, in which two or more firms are funded to undertake development up to the point where a "go/no−go" decision can be made, and, if the project proceeds, a winning approach selected.
However, firms participating in design competitions tend to focus heavily on enhancing the quality of the system, and ensuring it offers tangible performance benefits to each constituency involved in the decision. Cost control, in terms of acquisition and through−life support, are neglected, because costs estimates tend to be more difficult to test than statements about performance. As a result, while the two−pass approach may help, the improvements it provides could be disappointingly limited.
This is not to disparage the reforms that have come out of the Kinnaird review − far from it. Although implementation of these reforms is still under way, there are clear signs that the establishment of DMO as an effectively independent agency within the Defence portfolio, with independent financial responsibility and a clear charter to acquire and support equipment for Defence on a quasi−commercial basis, is yielding substantial improvements. But more needs to be done for the full potential benefits to be obtained.
To begin with, Defence − and the government − need to be serious about considering military off−the−shelf options.
Developing, or substantially modifying, advanced weapons systems will always be a highly costly and risky undertaking. Australia should engage in that undertaking only when rigorous analysis shows both that imported, off the shelf options are clearly inferior in terms of military value and that the incremental benefits of an Australian−developed system would outweigh the risk adjusted costs. A considerable strengthening in Defence's ability to undertake analyses of this type, and more independent scrutiny of those analyses that are carried out, should be a priority.
Greater use of imported military off the shelf systems would reduce the burden on our processes for buying weapons, as well as the cost to taxpayers. Inevitably, however, there will be cases where we have to develop Australian−based solutions to meet the demands of our geography, the likely operational roles of the ADF and our commitment to defence self reliance.
The Collins Class submarines are a classic example of a domestic project driven by such considerations. Moreover, there is a substantial component of maintenance work that is most efficiently carried out locally, both for cost reasons and to ensure equipment can be serviced in the event of conflict.
The question becomes what can be done to improve the performance of procurement arrangements in managing Australian−based solutions. Three areas stand out.
First, there is further scope for DMO to act as a smart and demanding buyer. Nowhere is this clearer than with information.
In negotiating with suppliers, DMO should be well placed to benchmark costs and outcomes across weapon production programs, and use that information to get performance improvements. Even though programs differ, there are functions that are common to many projects that can be compared. Expanding benchmarking would be an important step in reducing the information asymmetry between DMO and its suppliers.
Second, incentives can be used more effectively. Defence contracts involving many hundreds of millions of dollars often have incentive and control clauses that are simple, if not basic, compared to those used in commercial project finance. While contractual complexity has costs of its own, greater use could be made of mechanisms that escalate rewards and penalties in line with performance of both the project and the project relative to comparators. All too often, projects persist because the costs of budget overruns are widely spread (and not obvious to those who bear them), while the benefits of persisting with the project are highly concentrated. Only the proper design of incentive arrangements can reduce this risk.
Lastly, accountability must be strengthened. In his classic study of major British procurement failures former chief economist David Henderson emphasised the "unimportance of being right" − the fact that the British system, where decision−making by committee spread responsibility widely, gave few rewards to those associated with long term success, and few penalties for those involved in projects that had plainly failed. Admittedly, the long lead times involved in determining success or failure − which often can be judged only 10 or more years into project life −greatly complicate performance assessment, but good outcomes cannot be achieved without aligning control over decisions, responsibility for managing risks and accountability for results.
There are many elements to securing accountability, the first being to have realistic standards of performance. One element, however, is especially weak in the Australian policy process, and that is well−informed and rigorous review.
This has both an internal and an external component. Relative to the United States and the UK, our Defence establishment seems weak in its systems analysis capabilities, which are indispensable for informed assessments of choices and for measurement of performance.
External scrutiny of defence expenditure also needs to be strengthened. There are many difficult economic issues in the way we select the weapons we want to buy and manage their acquisition and maintenance that simply go unanalysed. The result is not merely to undermine the scrutiny of substantial outlays, but also to reduce our ability to learn from experience and make improvements.
More informed analysis of, and debate about, procurement policies is all the more important because buying and maintaining advanced weapons systems is going to get harder in the years ahead. There are four trends that suggest that the pressures on defence procurement will become ever more acute.
The Defence budget will be squeezed by the steadily rising cost of weapons relative to other goods and services. This is partly the result of technological factors − as is occurring with stealth aircraft − where the search for greater systems performance chases ever−diminishing returns. It also reflects the costs associated with meeting threats that though long present, are becoming more important, such as asymmetric warfare. An improvised explosive device may cost only a few hundred dollars, but protecting against it is complex and expensive. And, as weapons costs rise, governments tend to buy fewer units of each system, so that unit costs rise even more rapidly than total costs.
Second, the global defence market is becoming less competitive. In the 1980s, when the Collins submarines were being planned, seven shipbuilders were approached to meet the program's requirements; of those, only two remain active. A similar trend to concentration has occurred in most of the complex systems, with a result that worldwide, procurement has tended to involve ever fewer players. In the United States − the world's largest market for defence equipment − the five largest suppliers of arms now account for overone−third of all defence purchases, up from some 20 per cent during the later years of the Cold War.
Moreover, a high share of the sales these firms make is now "sole sourced", that is, non−competitive; with sole sourcing contracts rising from about 45 per cent of the total in the mid−1990s to 65 per cent today. A less competitive arms market internationally will make it more difficult to ensure we get "value for money".
Third, and turning to a more local factor, we face a very substantial − and as yet unfunded − maintenance requirement, as a wide range of new systems come into use. The costs of maintaining these new systems do not appear to have been adequately factored into projections of required spending, but at some point, those costs will need to be funded.
Finally, the current phase of strong economic growth and continuing budget surpluses, which have removed any real constraints on Defence spending, will at some point come to an end. When that happens, the government will need to be far more rigorous in setting spending priorities than it is now.
The sooner these forces and their implications are understood, the better prepared we will be for the choices to be made. It is crucial the debate about defence procurement be realistic and well−informed. There is still a long way to go.
**Henry Ergas has consulted to the DMO and assisted the Minister for Defence in one defence industry policy review. These are his own views.