Government Software and Software Development20sep99: Software consultant A: Open Source for government software?
The current model
Public software projects should have clearly defined specification, tender and implementation phases. The client writes a specification; the specification becomes part of the tender documentation and the winning contractor implements the specification. That's the theory.
But the theory doesn't work.
Rapid application development
Traditionally specifications can be harder to write than the software itself.
However, with modern tools, good developers can implement prototypes from simple informal descriptions to help the client see what is possible. This method of informal prototyping has recently been given the label "Rapid Application Development" to give it some respectability. Looking on one of the foremost internet search engines for "rapid application development" we get
"AltaVista found about 32952 Web pages"and one of the first pages found says
"Cyberscreen provides a very rapid application development cycle, allowing you to go from the database to a basic application in a matter of seconds - yet it is powerful enough to quickly and efficiently handle the most complicated of database manipulation techniques."Propriatry packages are not 'the' answer but show that software development can proceed in a more informal manner giving the opportunity for greater interaction with the client and reducing the reliance on an expensive specification process.
This does not address the problem of accountability. Openness is increasinly required for public contracts. Can this be satisfied without a detailed specification ?
The open source approach ...
In recent years an interesting feature in software development is the rise of the open source movement. The best known projects are Linux and Apache.
Linux started as one man's implementation of the Unix operating system to become fastest growing non-Microsoft operating systems with countless press releases. At the time of writing the latest headline from Inter@ctive Week is "IBM introduces first Linux laptop." Apache is the most used web-server in the world.
Open source projects publish the source code of their software for public use and are developed by a looser community than with the conventional client/contractor model. This "community" is led by a core team but can have thousands of participants who can input ideas. Such projects rely on the internet to control the information flows.
The open source movement is a counter-culture of its own but recently the 'suits' have taken much more interest. It has, after all, produced some spectacularly sucessful software. One leading evangelist is Eric Raymond who's best known article is "The Cathedral and the Bazaar" (see www.tuxedo.org/~esr/writings/cathedral-bazaar/).
Here is a quotes from a smaller note (found at www.tuxedo.org/~esr/faqs/hacker-howto.html) (Note: Raymond uses the word "hacker" in its original meaning - not the meaning associated with "hacking into" websites):
"I can't give complete instructions on how to learn to program here -- it's a complex skill. But I can tell you that books and courses won't do it (many, maybe most of the best [open programmers] are self-taught)."For a conventional approach to software development, this can be a problem: There is no official accreditation "Problem Solver: Class 1". For the open source movement this is less of a problem - gifted participants work their way into the inner circle.
But a second quote from Eric Raymond highlights an apparent problem:
"Like most cultures without a money economy, [open programming] runs on reputation."Few participants in open source software are directly paid because, although the software is useful and can be used extensively, it is freely available so it is difficult to generate an income stream to pay the participants.
... applied to government contracts?
Software written for public bodies is often created to do a specific job and, in most cases, the publication of the source code in a working form would not be a problem.
It should be perfectly possible to adapt the open source model for suitable public 'contracts'. In the case of, say, an interactive database system for a governament department, core staff would be salaried or employed as contractors. Other participants would probably initially come from potential users of the system or software workers within the department.
Participation in the project could also come from a wider community: Other government departments might take an interest; Job seekers might try to work their way into the project or students might consider the participation in a large real-life project a worthwhile learning experience.
If this sounds fanciful remember that the open source movement is scoring some goals while 'traditional' software development in the government sector at least, is helplessly scoring own-goal after own-goal.
Many of these own-goals are still well hidden!
23mar00a: From Computing magazine: Government locked in
Whitehall waives Andersen repaymentThe UK Treasury has abandoned hopes of recovering millions of pounds in compensation for delays in the new national insurance system because it needs to preserve the relationship with the system's developer, Andersen Consulting.
In a decision which illustrates the huge potential for outsourcer 'lock-in', paymaster general Dawn Primarolo said that the concession is required because Andersen's co-operation "is essential to the delivery of future changes to the system".
"I am satisfied taking action against Andersen would prejudice the partnership relationship now established between it and the Inland Revenue," she told MPs last week.
In effect, government plans for major reforms to the national insurance, pensions and welfare systems are now dependent on Andersen's good will... Full article
10apr02a: Computer programmer A: My experience in the computer games industry
(Computer programmer A is in his late 20's)
Highly specified, safety-critical programming
I did a masters degree before going into flight simulators. The work used SSADM, up-front design and very thick specifications for projects. I HATED it. I think this approach might be necessary for safety-critical applications, but productivity (eg. lines of code per programmer per day) was necessarily low. But the work on flight simulators gave me an "in" for the games industry.
The games industry consists of about 10 big games publishers world-wide (eg Electronic Arts) and lots of software houses: There are two dozen in London alone and, in the UK, more in Guilford and Scotland.
Software houses usually have the ideas for the games. They approach the publishers to find one that is interested.
The ususal contract will centre round a 20 page design document, which won't cover all of the technical issues, but will deal with game play issues, features etc. Also, monthly milestones are specified for the project.
The contract is more like a gentleman's agreement
A full spec would be huge but this is not done because the customers (ie. publishers) will change their minds all the time. Some publishers will try and write technical solutions into the specifications but the software teams ignore these. Some publishers have tried to write havy up-front specs but rarely persue the because they change their mind too often. The real professional publishers keep the specifications light-weight and the contract is more like a gentleman's agreement.
I have heard reports of full-specification products being developed but I don't believe them.
On average a game will take about a year to develop
Specifications change monthly
Payment under the contract is by royalties on sales but substantial advances are given, which are paid when monthly milestones are met. But as the customers can change their minds and redefine the game on a monthly basis, the milestones have to be readjusted (renegotiated). Changes amounting to, say, 5% of the specification per month is reasonable. Typically a customer may not chage for three months then want a huge rewrite.
Half the projects are cancelled.
The publisher can withhold payment if milestones are not met and can pull the plug on the project at anytime. Advances are not repaid. In the industry about half the projects are cancelled. Cancellations may be because the market has changed or because of technical difficulties. On cancellation, negotiations are usually conducted by external third parties.
Track record more important than qualifications
Before a software house gets a publishing deal it must have a track record or a limited technical demo. For individuals it is all track record. There are no qualifications yet but they may be coming as some colleges are trying courses. Graduate computer scientists do come into the industry. They tend to have too much formal methodology and are not so interested in the end result. They usually change their ways or leave the industry.
Most of my colleagues write in C++ but I write in C. Object-oriented stuff like inheritance not used much because "you have to get dirty to get performance".31aug02a: Faxfn: An Academic's Joke
There were three undergraduates:
They were each given the task of writing a program to add up the numbers 1 to 10.
First let me apologise for the advert in Private Eye - it was over the top but the faxfn advertising department thought it might catch the eye.
The question "Can PRINCE make large software projects work?" is not an idle one. There is a serious conflict between
The Cabinet Office published an interesting paper which points out some of the problems associated with software projects particularly large ones(Successful IT: Modernising Government in Action, May 2000).
This means that the specifications of later steps are contingent on the evaluation of earlier ones. The consequence of this are :
This does however leave a serious problem for software procurement because a software project becomes a continuously changing stream of small steps. But how can such a project be put out to tender or seriously considered for Treasury approval?
The Cabinet Office document hints that PRINCE ( which is the property of the Office of Government Commerce )may be of some use in these circumstances. So I would like to ask this question: Can the PRINCE methodology resolve this dilemma and make the specification of large government projects stable enough to use for contractual purposes?14oct02a: IT Manager in Small Local Authority: One nice thing about PRINCE
The trouble with PRINCE is the way it is taught. It is taught more as a qualification than something that is really useful. But one nice thing about PRINCE is that it provides a common language - everybody that has had PRINCE training knows what the acronyms mean.
But it does not enable specifications to last very much longer. In my experience specifications do not remain unchanged for even a month.14oct02b: Executive in Government Organisation: Our experience with PRINCE
We have used PRINCE2 for 3 years on more than twenty projects. The biggest one was projected to be £10m (including hardware costs) but it was stopped a third of the way through. There was nothing wrong with the methodology. The programme did not deliver because it was deemed not to be appropriate to deliver it. It was stopped at the point of the functional specification. It could be said that it helped us see more clearly that the project needed to be stopped.
Does PRINCE work? Like Christianity it is a set of principles to which to adhere - and these can be learnt on a five day course. A follower of Islam may have a different set of principles but a good Muslim would lead much the same sort of life as a good Christian. If fact, we are considering moving from PRINCE to another methodology.
PRINCE does tend to slow down spec changes simply because the existence of formal procedures slows down changes in specification.
We use PRINCE for any project where significant expenditure is involved. Many projects need legal traceability. PRINCE gives this.
Specifications (ie. projects) should only last six months and employ a maximum of six people22oct02a: Geoff Beacon: "The Economist" takes an interest.
Earlier I asked "Can PRINCE make large software projects work?" and highlighted the conflict between
they also mention the role of the Office of Government Commerce and "Gateway Review" (which incorporates PRINCE.)
"But the main source of optimism [that large government projects can be made to work] is an almost mystical belief in something called the Gateway Review process to prevent future Government IT fiascos. Gateway was devised by the Office of Government Commerce, which oversees purchases of government departments."
Faxfn has posted two responses to the PRINCE/Gateway question. These responses do not represent an industry consensus but it is interesting to see that both responses highlight the need to keep projects small. Not surprisingly, the biggest "success" of PRINCE was its use in justifying the cancellation of a large project - "it helped us see more clearly that the project needed to be stopped".
The Economist piece has some praise for the use of open standards.
The new NHS-wide email service, for example, being built by EDS, is a web-based service similar to hotmail. The use of open standards will also allow central managers to set, say, the format of electronic patient records, while allowing decisions about implementation to be made on a local level.
Importing any sort of openness into government software projects will be welcomed by the Open Source movement (see "Open Source for government software?" above). But they should go further than this and actually put the source code from the project into the open source domain. Then it could be checked for validity by the legions of clever nerds who can be found on, say, www.slashdot.org
Amongst these nerds we would certainly find informed opinions about the "value for money" question that the "Specification for Tendering" model is failing to resolve.
Public accountability and transparency should require source code for projects like the NHS-email service to be put in the public domain. We are able to see the brickwork on new public buildings. It should be just as easy to see the detailed work on a major public IT project. A major benefit of an Open Source approach would be that it would be easier to break projects down into smaller parts and much easier to avoid supplier lock-in. (See "Government locked in" above).
12nov02a: Geoff Beacon: Does Government software fail because the Treasury can't cope?
This is to continue the discussion of methodologies which might make large government software projects manageable. Previous entries discuss whether this is feasible, but we have yet to see evidence that large software projects under current methodologies can actually be made to work.
The Government, through the Cabinet Office and the Office of Government Commerce, seem to be convinced that a solution exists (the PRINCE2 methodology and the Gateway Review process). I, for one, am sceptical. This debate has now reached The Economist magazine. It comments (see above):
"... the main source of optimism [that large government projects can be made to work] is an almost mystical belief in something called the Gateway Review process to prevent future Government IT fiascos. Gateway was devised by the Office of Government Commerce, which oversees purchases of government departments."
The question I wish to ask is this: Is there any reason why the Government Commerce makes such extravagant claims for their methodology (Gateway and PRINCE2)?
The Cabinet Office report "Successful IT: Modernising Government in Action", published in May 2000, gave an analysis of the problems of software procurement. Under the heading "Making large projects more manageable" they accept that there is a case for breaking down large projects:
Governments in the UK and abroad and the private sector have recognised that an effective way to reduce risk is to break large projects into smaller, more manageable components.
However, they give the following as evidence
Evidence: A large insurance company undertook a project in which the staffing grew to such an extent that the management overheads were not worthwhile and communication became difficult. The company now insists that senior management can see 'both ends of the tunnel' at all times. This is achieved by making all projects modular, for example, letting them run for no more than 18 months and involving no more than 50 staff.
A project lasting 18 months with 50 staff is well over the limits advocated for modern software development methodologies, which many experienced software writers put at a maximum of six staff for a project, lasting at most six months (see above Executive in Government Organisation: "Our experience with PRINCE" by an Executive in a Government Organisation).
Clearly, the authors of the Cabinet Office report are not suggesting projects be reduced to the size advocated by the gurus of modern software development methods such as Agile Programming (see The Agile Alliance) or Extreme Programming(XP). These are methodologies, where small teams of programmers work closely with users during the course of software development.
One commentary ( Computerworld Software Topics) says
A move to agile programming, in fact, strikes the most sensitive of nerves: honest communication. For years, corporate IT and users have worked separately. Lightweight methodologies bring everyone together face to face and keep them there.
However, there is a clear reason why Agile Programming and similar approaches would be rejected by the authors of the Cabinet Office report. The stumbling block is that they insist on the benefits of a project being understandable before a project starts and these benefits alone, should be the measure of success. They state
6.1 A project or programme is only successful if it delivers the benefits for which it was initiated.and
Recommendation 15: A post-implementation review must be undertaken of all projects or programmes and benefits realised assessed against projected benefits outlined in the original business case or subsequent amendments.
The conclusion drawn from these points is that more control from the Centre (The Treasury?) is needed:
Money for large projects or programmes is authorised by HM Treasury for the realisation of a specific set of benefits contained within a business case. Therefore, the Centre has a key role to play in monitoring the extent to which those benefits are, in fact, delivered.
Clearly any project that starts with a business case to be put to the Treasury, cannot be small. Both the scale of such a project and the "projected benefits only" rule means there is little scope for the customer and programming team getting together to "work things out as they go along". The authors of this report do, however, allow for this possibility in another way - the use of prototypes. Under the heading "Purchase of preparatory work" they say
5.18 During the pre-contract phase, one process that can help to firm up requirements is the use of prototypes. ... Most are throwaway systems that are built as cheaply as possible to help to clarify requirements or prove a concept. Due to the short-term nature of these prototypes, they are unlikely to be suitable for procurement under a PFI arrangement.
The question the authors do not answer is "What should be done if the prototypes actually work?" One of the methods of Agile Programming is to progress through a series of small stages, which should be production quality code and should deliver business benefit. Typically, these stages will cost significantly less than the £100,000 allocation the agency set for each prototype.
But, of course, projects of this size are too small to go up for Treasury approval. So my question is this:
Are we stuck with large projects with rigid specifications because the
Gateways and fantasy PRINCEs are unlikely to overcome this handicap.
Kent Beck claims lightweight methodologies, such as his, force developers to be honest, because users work closely with them and can see what is happening. With Government software projects, we should go that step further and let the public have the access to see what is happening. This could be simply achieved by using the existing mechanisms of the Open Source movement.
29nov02a: Various gurus: "conclusion ... upfront designs were often not very good".
Just found via slashdot.org:
artima.com have an interview with software development guru Martin Fowler here. He says
But in my view, extreme programming's practices of continuous integration, testing, and refactoring actually make evolutionary design work, and more effectively than planned design. Planned design's weakness is that creating a well-planned design is actually really tough.and
It's interesting that many of evolutionary design's major proponents, like Kent Beck and Ward Cunningham, are stunningly good designers. But they came to the conclusion that their upfront designs were often not very good. They tended to over-engineer things, adding unnecessary complexity to their designs. They would often make a design flexible in areas that didn't need flexibility and inflexible in areas that did need it. So they've adopted an approach where, by applying a set of disciplines, evolutionary design works instead. As a result, they have created better designs, and at a faster rate. I think 80 percent of the time evolutionary design works for me as well. And I have the arrogant opinion that I am an above average designer, so I think evolutioary design would be even more useful for a wider range of people.
09dec02a: Geoff Beacon: Assignment for the Treasury: Study the games industry.
The Gershon Report and Gateway Review
In 1998 Peter Gershon was appointed to review civil procurement in Central Government. His report led to the founding of the Office of Government Commerce. Peter Gershon has become its first Chief Executive. The OGC has subsequently developed the comprehensive Gateway Review process, which has five review phases and about thirty stages (see the Gateway Review Leadership Guide). It species a procedure for govenment procurement, identifying the roles of people involved (eg. "review team leader", "project team member", "Senior Responsible Officer"). It is a serious heavyweight process.>
While Gateway Review is not solely aimed at IT projects, they are one of its major areas of application: "This process ... will meet the requirements of the Gershon Report on government procurement and the Cabinet Office Report Successful IT: Modernising Government in Action."
The Gershon Report was initiated for serious reasons. In 1999, the Select Committee on Public Accounts considered 25 large public software projects, which have failed seriously over the previous decade. They included
Readers of the computing press and even the broadsheets could name other projects that have failed, been cancelled or are seriously late and over budget: London Ambulance, DVLA, Air Traffic Control etc. Large successful software projects have been more difficult to identify.
Software Development in the US
In the United States, the situation was much the same but there, some systematic work has been done in recording the failures. The CHAOS report in 1994 by the Standish Group, which surveyed a sample of 365 respondents and representing 8,380 applications, says
"The Standish Group research shows a staggering 31.1% of projects will be canceled before they ever get completed. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg."The successful projects were 16% of the total. By the year 2000, they report elsewhere , the percentage of successful projects had risen to 28%. During this period the abandoned projects had reduced from 31% to 23%. Some of this improvement may be put down to better project management.
Small projects good. Large projects bad.
In as much as it is a better tool for the management of big projects with fixed specifications, Gateway Review may have a role, especially in stopping large rogue projects. (See the use of PRINCE, the precursor to Gateway, above: "Our experience with PRINCE"). However, the biggest problem with the Gateway Review process is the illusion that it can be used to make very large IT projects work.
In a later article in Software Magazine by Jim Johnson, chairman of the Standish Group says
"Most of these new [successful] projects are well within The Standish Group's criteria established in "Recipe for Success, 1998," which limits project duration to six months and project staff to six people."This article was based on information from the company's paper, "Extreme CHAOS 2001."
Large projects necessary for PFI.
It is true that Successful IT: Modernising Government in Action." does mention the problem of project size. It discusses breaking down the implementation large projects into smaller stages:
"Governments in the UK and abroad and the private sector have recognised that an effective way to reduce risk is to break large projects into smaller, more manageable components."
This does not, however, reject the "very large project with a fixed specification". The smaller component is a part of a larger whole. Indeed, they discuss small projects only in terms as prototypes. Under the heading "Purchase of preparatory work" the authors of the report say
"During the pre-contract phase, one process that can help to firm up requirements is the use of prototypes. ... Most are throwaway systems that are built as cheaply as possible to help to clarify requirements or prove a concept. Due to the short-term nature of these prototypes, they are unlikely to be suitable for procurement under a PFI arrangement."This is the key:
Large projects are necessary because that is what is required for "procurement under a PFI arrangement". (But, of course, large monolithic projects are very difficult to make work.) They go on:
"To encourage the bidders to follow this approach, each has been paid £100,000 from [a UK government] agency's own funds. The agency views this as an acceptable price for the risk reduction it offers."No answer is given to the obvious question as to what should be done if the prototypes work or form the basis of a workable system. However, elsewhere the authors say
"Again using Microsoft Office as an example, one could consider the development of Word from its first release through to Word 97 to be part of the incremental development of that product. Microsoft did not attempt to build all the functionality of Word 97 into the first release of Word; they created a simple version with a usable set of facilities, which was then built on to create later versions."
Lightweight software developmentSome would regard a £100,000 prototype as a good start to a development process, which may or may not go further. But the advocates of lightweight software development methodologies (See the manifesto of the Agile Alliance .) would argue that this first stage must have a business benefit.
A process of linked stages does have its dangers. The authors of Successful ITcite this exampleA government body has contracted with a single supplier for the installation and management of a desktop infrastructure under a Private Finance Initiative (PFI) arrangement. This is the first stage in what they intend to be a long-term relationship.This sounds to close for comfort to the lock-in reported in Computing magazine (See above: "Government locked in").
The twentieth stage of the thirty Gateway Review stages is identified as "Development and Testing". It comes after stage nineteen "Project Board approval to award contract". This is the stage at which serious computer code will get written to a tight specification. This is far from the approach of the lightweight methodologies cited above. With these, code is written by a small software team that is in very close contact with customer representatives who themselves understand the business requirements. Coding starts early in the project and the software is reworked vigourously - to improve performance, internal consistancy and the usability to the customer. The time to an intial release delivering measureable business benefit is short - a few months at most.
Where did they go wrong?
One purpose of the Gateway Review process is to develop a tight contract suitable for employing external suppliers under, for example, a PFI arrangement. This contract is likely to contain a rigid specification. The Treasury are keen to make PFI work in the IT field but PFI, as currently envisaged, requires large projects with fixed specifications. We know this is a recipe for disaster. If the Treasury want the benefits of PFI and Successful IT, it must square this circle.
Why did Peter Gershon get it wrong? Perhaps in some areas he was right, for example, with non-IT projects. There are also areas in IT where incremental development is not thought possible. In certain industries, rigid specification is deemed absolutely necessary. Aircraft and defence systems provide examples of these. It is noteworthy that Peter Gershon's previous experience at ICL (now Fujitsu Systems), Marconi and BAE Systems is within this type of environment.
Gateway forbids good software development
The Gateway Review Process makes successful software development methodologies, such as Rapid Application Development, Agile Programming and Extreme Programming, impossible. It also makes it impossible continuously to modify the specification. Continuous modification of specifications enables the development process to respond to unexpected difficulties or opportunities for enhancement. It is worthy of note that that Joel Spolsky, a manager on the Microsoft Excel project said that his specifications changed on a daily basis. He also says
Specs Need To Stay Alive. Some programming teams adopt a "waterfall" mentality: we will design the program all at once, write a spec, print it, and throw it over the wall at the programmers and go home. All I have to say is: "Ha ha ha ha ha ha ha ha!"(Found on joelonsoftware.)
Do computer games have an answer?
The piece "My experience in the computer games industry" (above) describes an interesting picture. The market in the development of computer games is not a perfect one but it has more to commend it than the rigid centralised planning of the Gateway Reviews. Coupled with an oligopoly situation amongst the suppliers, which occurs because the Gateway Review Process is such a barrier to entry, we have a recipe for the IT disasters to continue.
The computer games industry has several developers on the supply side (the software houses) and several on the consumer side (the publishing houses). They have mechanisms for starting projects, stopping them, and continuously retargetting them during development. They keep the projects small compared to Gateway Projects. They are successful because they truly are curtailed by market forces.
14dec02a: IT consultant in a large local authority: Successful IT projects are small.
(Autobigraphical note :This consultant worked for the Inland Revenue for more than a decade. She has also been in her present post with a large local authority for more than a decade).
Large Inland Revenue project failed
"I worked on a large IT project for the Inland Revenue, the "Shedule D" project in the early 1970s. It died completely after three or four years and mountains of effort."
Successful projects are small
"Successful projects are small projects. Large projects can be made into an amalgamation of small projects. I think it may be possible for parts of these to be outsourced as long as the deliverables are clearly identified."
A good record of project completion - but no outsourcing
"My local authority does not outsource as such, but we do bring in consultants and contractors. We make them work very closely with us to our own standards. My local authority has a very good record of completing projects. We deal with the consultants and contractors as individuals or SMEs (Small to Medium sized Enterprises). We tend not to use the big firms."
We are now committed to PRINCE - but I havent used it yet
"PRINCE? Personally I have never used it but the local authority now has a committment to it."
08jan03a: Geoff Beacon: Treasury rules exclude successful software development methods?
More advances in Agile Programming
Martin Fowler and Pramod Sadalage have an interesting article, Evolutionary Database Design here. It is aimed at software developers and was a discussion topic on Slashdot ("News for Nerds. Stuff that matters") But it also has some consequences for theories of procurement. The authors say
"In the last few years, we've seen the rise of a new breed of software methodologies, the agile methodologies. These make some new and significant demands on database design. One of the most central of these demands is the idea of evolutionary design. On an agile project you assume that you cannot fix the requirements of the system up-front. As a result having a detailed design phase at the beginning of a project becomes impractical. The design of the system has to evolve through the various iterations of the software. Agile methods, in particular extreme programming (XP), have a number of practices that make this evolutionary design practical."and
"One of the tenets of agile methods is that people with different skills and backgrounds need to collaborate very closely together. They can't communicate mainly through formal meetings and documents. Instead they need to be out talking with each other and working with each other all the time. "(My emphasis)
Agile Programming and procurement
The rise of this "new breed of software methodologies" poses this problem: If Agile methodologies are the way forward for software development how can a procurement process be formulated that can incorporate them.
Specifically, this is a problem for the Treasury: If "you cannot fix the requirements of the system up-front" how can you apply the Gateway procurement process? In addition, the Gateway Review is a lengthy process with some 30 stages. Stage 20 is "design and implementation" which comes some time after the requirements of the system are fixed. With a changing personell between stages, it is difficult to see how the particpants can be "be out talking with each other and working with each other all the time".