Why can't the IT industry deliver large, faultless projects quickly as in other industries?
After watching National Geographic's MegaStructures series, I was surprised how fast large projects are completed. Once the preliminary work (design, specifications, etc.) is done on paper, the realization itself of huge projects take just a few years or sometimes a few months.
For example, Airbus A380 "formally launched on Dec. 19, 2000", and "in the Early March, 2005", the aircraft was already tested. The same goes for huge oil tankers, skyscrapers, etc.
Comparing this to the delays in software industry, I can't help wondering why most IT projects are so slow, or more precisely, why they cannot be as fast and faultless, at the same scale, given enough people?
Projects such as the Airbus A380 present both:
Major unforeseen risks: while this is not the first aircraft built, it still pushes the limits of the technology and things which worked well for smaller airliners may not work for the larger one due to physical constraints; in the same way, new technologies are used which were not used yet, because for example they were not available in 1969 when Boeing 747 was done.
Risks related to human resources and management in general: people quitting in the middle of the project, inability to reach a person because she's on vacation, ordinary human errors, etc.
With those risks, people still achieve projects like those large airliners in a very short period of time, and despite the delivery delays, those projects are still hugely successful and of a high quality.
When it comes to software development, the projects are hardly as large and complicated as an airliner (both technically and in terms of management), and have slightly less unforeseen risks from the real world.
Still, most IT projects are slow and late, and adding more developers to the project is not a solution (going from a team of ten developer to two thousand will sometimes allow to deliver the project faster, sometimes not, and sometimes will only harm the project and increase the risk of not finishing it at all).
Those which are still delivered may often contain a lot of bugs, requiring consecutive service packs and regular updates (imagine "installing updates" on every Airbus A380 twice per week to patch the bugs in the original product and prevent the aircraft from crashing).
How can such differences be explained? Is it due exclusively to the fact that software development industry is too young to be able to manage thousands of people on a single project in order to deliver large scale, nearly faultless products very fast?
Interesting question. I'm tempted to say the quality of the average worker in the software industry is less skilled and qualified than, say, civil engineering where every engineer has completed a rigorous and intensive degree and likely gained his charter too. Furthermore, the complexity space of large software (eg. an OS, MS Office) is probably much greater even than an aeroplane. Certainly many more places to fail! And a final important point: most software more or less works, even if was poorly written and highly buggy...certainly the cost of failure is normally much less than an aeroplane!
Any recent mode of transport is full of software, if you're interested in the software quality of aerospace systems then have a look at the DO-178B and DO-178C certifications
Find someone who actually works in any of those other industries (but not in PR) and ask them about "large faultless projects". I can virtually guarantee that you'll earn pained laughter.
Primary problem is that every new project usually have new, untested things to solve.
The realisation of a software project takes seconds or minutes; it's what happens when you click "compile" in your IDE. On the other hand, *programming is design*. How long did it take to design the A380?
That TV program is a hype. They only telecast successful projects. Any channel will make programs for viewers attention.
That's two successful projects picked from the aviation industry and compared to all the unsuccessful projects from the software industry, without even giving specific examples.
@ThorbjørnRavnAndersen - I don't disagree with your comment, however, it does raise the interesting follow on question of why then, does software development concern itself so much with notions of reuse (components, cohesive classes, OOP etc.) if we essentially have to solve different, new and untested problems each time?
@CraigTP a guess would be that the world changes fast enough for the basic premises to settle. If you keep writing DOS programs - which Windows will run - you will provide suboptimal solutions in todays world.
"Computer programs are the most complex things that humans make." - Douglas Crockford
Software development exists since.. about 30 years? Maybe 40. Engineering has done virtually ever since! A lot of expierence could be gathered through all the time.
Why are you surprised? Engineers in general have several millennia head start on software engineers.
I read somewhere saying that software engineers often underestimate the time required to complete a project. The delays you experienced may simply be the fact that the people set un-reasonable deadlines when in fact it took just as long. Just a theory, not confirmed fact by any means.
A seasoned PM once told me that in a factory/assembly line like setting, each person/team at every phase has a different expertise. They can't necessarily change something there and then and must be 'escalated back up'. Software - almost everyone on the team can spend sometime, understand code they haven't written and change without necessary regard to downstream repercussions. Not so with other industries
A quick review would have convinced you that the A380 is certainly not the success you think. Faultless? As recent as 2010 it had a major recall because of a major fault in the Rolls Royce engines.
Go over to YouTube and search for episodes and clips of the "Seconds From Disaster" series. All isn't perfect in other industries.
If human got a Airbus, human is happy with it. If human got a patient journal system, it's wrong implemented. I feel it also a result of consumers nature. Differently touched "the canine is to low to roof", engineers say "go and sit". A softwareengineer "of, your probably right, we fix?"
Ironic that you should pick the A380 as an example of a successful, on time project. The A380 was two years late entering service due to wiring issues, and now all aircraft need to be re-worked to fix cracks in the wings. It's beginning to sound exactly like your typical IT project!
@Gavin Coates: quoting from my question: "With those risks, people still achieve projects like those large airliners in a very short period of time, and **despite the delivery delays**, those projects are [...]".
It's easy, just *redefine* success to be the same level in all situations. In the airplane, when your tray table isn't exactly well aligned, you don't bother to complain. If an icon isn't exactly aligned in your software, you'll make much more noise and complaining about it, and so on...
I downvoted due to the begged question: anyone who's ever lived in an area with major civil works projects or followed the news on projects like the ones you mentioned has reason to question whether they're that much different. A better question would compare some large projects which did and did not procede efficiently to compare the approaches - there's a potentially interesting discussion about strategies for managing complexity.
The formal launch was in 2000, but the article said development started in 1994. That's 11 years. Imagine writing and testing code for 11 years before shipping.
'imagine "installing updates" on every Airbus A380 twice per week...' Imagine enemy robots constantly probing the plane for vulnerabilities while untrained pilots push buttons at random. I bet you'd need regular patches.
@MainMa - The A380 project was far from developed in a "very short period of time". Officially offered for sale in Dec. 2000, and entered service in Oct 2007. That's 7 years in development. Initial studies began in 1988 before it was even decided it was feasible to do, with the project first announced in 1990. I'm a big fan of the A380, but it certainly isn't a project designed in a short period of time, or a good example of a successful project (in terms of being on time and on budget).
Paradoxically, because too many people in the IT industry don't know the answer to this question. Neither ask it.
This article is a better answer than most of the comments posted so far: http://www.drdobbs.com/tools/building-quickbooks-how-intuit-manages-1/240003694
Management can decide to release software that fails key tests. But you can't release an air plane that fails physics.
You talk that it's a given that projects that aren't software related are always more successful. That isn't necessarially true. Both the A380 and the Dreamliner had delays and cost overruns, and the F35 program is currently looking like a serious screwup, especially the B variant. Structural and mechanical engineering disciplines are more mature than those for software, that's certainly a factor, but there are plenty of projects that didn't go well that went nowhere near software.
Actually, in IT industry there is too much change in small amount of time. This change doesn't allow anyone to master a specific topic fully. For example, if you master Silverlight, you will be told that this is not going to last long, its already dead or it will be superseded by say HTML5 or whatever. So, we should make the stable and promote innovation, rather than killing each others technologies. Also, IT is still is newer than say Automobile industry.
This is a great answer for this kind of comparison: http://www.stevestreeting.com/2011/06/16/why-software-engineering-is-a-misnomer/
I love the answer Dijkstra gave more than thirty years ago: To the economic question "Why is software so expensive?" the equally economic answer could be "Because it is tried with cheap labour."
The reason those construction projects can be designed and built much faster is due to the quality of the software systems that allow the design to be modelled and tested. Weaknesses and design faults are likely to be caught during a simulation. It's unfortunate that the software industry still hasn't implemented these ideas with unit testing to a great degree
Large airliners are generally more reliable than software? Really? Which one usually needs more maintenance?
Well Airbus A380 was not all about hardware? Software was also involved. :)
it is just because software building has no strict 'coding' process to follow and has no way of 99.99996% (6 sigma) to be tested like factory, it's more like implementation of human thoughts than a real stuff. and human is just human.
agreed the premise of the question (that big projects are less problematic outside of software business) is highly questionable & arguably quite false. this question relates to human nature and is nearly anthropological/ cultural. in a word ***politics/ bureacracy*** which is common/ intrinsic/ inescapable to all large human projects and even smaller ones. another basic recent case study is the boeing dreamliner which you conspicuously dont mention and is now world famous for its delays.
The answer is due to the complexity, number of details, and heterogenous nature of IT projects (the project must satisfy many criteria, handle many different scenarios,) and the fact that many IT projects are built from scratch rather than using well-developed code that's already in existence. Every time you build a new airplane, you're not re-inventing the wheel. And the software that runs on airplanes only has to do a limited set of very specific, predictable things, unlike IT services which are trying to satisfy a wide variety of things many of which are not known in advance.
It took decades of hard lessons costing lots of lives to get aerospace where it is. Early on american rockets were routinely blowing up on the launch pad. And I think of the 3 Apollo I astronauts burning alive. These tragedies forced us to learn what it meant to design and build true quality. Challenger and Endeavor loses taught us - twice - the problems of organization culture and management reality distortion. Software just hasn't killed as many people - yet. Oh - NASA's software bug tolerance is about 1 in 10^6. No software startup would ever implement their process.
This question is based on a flawed premise: the A380 was delivered years late, was billions of dollars over budget, and had flaws that had to be corrected post-production. We can find similar issues with bridge, skyscraper, highway, railway, etc. construction, too.
https://youtu.be/G2X_7ojZwtU Cynefin. Software (complex, some pieces complicated) due to its cognitive and creative aspects is not manufacturing (complicated, sometimes simple).
@radarbob: "Challenger and Endeavor loses" - Challenger and Columbia were lost. The Endeavour is fine and resting in a museum, according to Wikipedia.
- 8 years ago
Ed Yourdon's Death March touches upon a number of these meta type questions.
In general, the software industry lacks a lot of the following, which gets in the way of large projects.
Standardization and work item breakdown.
- This has certainly gotten better, but the design constructs still aren't there to break out a big system. In some ways, software can't even agree on what's needed for a given project, much less being able to break things down into components.
- Aerospace, building construction, auto, etc.. all have very component-driven architectures with reasonably tight interfaces to allow fully parallel development. Software still allows too much bleed through in the corresponding areas.
A large body of successful, similar projects. The A380 wasn't the first big airplane that Airbus built. There are a lot of large software applications out there, but many of them have suffered dramatically in some aspect or the other and wouldn't come close to being called "successful."
A large body of designers and builders who have worked on a number of similar and successful projects. Related to the successful project issue, not having the human talent who has been there, done that makes things very difficult from a repeatability point of view.
"Never" building the same thing twice. In many ways, an airplane is like any other airplane. It's got wings, engines, seats, etc.. Large software projects rarely repeat themselves. Each OS kernel is significantly different. Look at the disparity in file systems. And for that matter, how many truly unique OSs are there? The big ones become clones of a base item at some point. AIX, Solaris, HP-UX, ... herald back to AT&T System V. Windows has had an incredible amount of drag forward through each iteration. Linux variants generally all go back to the same core that Linus started. I bring it up, because the variants tend to propagate faster than the truly unique, proprietary OSs.
Really bad project estimation. Since the repeatability factor is so low, it's difficult to project how large it will end up being and how long something will take to build. Given that project managers and Management can't put their hands on the code and actually see what is being done, unrealistic expectations regarding timelines get generated.
QA / QC is not emphasized as heavily as it could or should be for larger projects. This goes back to having looser interfaces between components, and not having rigid specifications for how components should work. That looseness allows for unintended consequences and for bugs to creep in.
Consistently measurable qualifications. Generally, people speak of the number of years they've worked in X language or in programming. Time in is being used as a substitute for caliber or quality of skill. As has been mentioned many times before, interviewing and finding good programming talent is hard. Part of the problem is that the definition of "good" remains very subjective.
I don't mean to be all negative, and I think the software industry has made significant strides from where we've been. Forums like this and others have really helped promote conversation and discussion of design principles. There are organizations working to standardize on "baseline" knowledge for software engineers. There is certainly room for improvement, but I think the industry has come a long way in a reasonably short period of time.
It was difficult to pick an answer to accept among several very good answers, but I finally select this one despite the fact that it doesn't have the highest votes. In fact, both this answer and the one by m3th0dman precisely *why* there is such specificity in IT industry, helping to understand *what* to do in the future to close the gap. Compared to the answer by m3th0dman, this one seems much more detailed.
+1 but might I just add that since software exists in the realm of the mind, it has *almost infinite* possibilities, whereas every plane every built must contend with the finite requirements of reality.
+1 for "QA / QC is not emphasized as heavily as it could or should be for larger projects." Would you fly in an airplane that was built "quick and dirty"? What about using a bridge that was built "faster" without any "needless" testing to delay things?
Very well answered. As an interesting example--imagine if a large plane was designed and implemented by a bunch of people without process or company history--people that just got together and formed a business to build a plane on the scale of a 747 from scratch. That's how 90% of the software projects I've seen are done. The other 10% with experienced architects and companies with history and process seem to be a lot more successfull. For a counter-example look at the development process behind software that causes people to die when it fails.
@MainMa you should not choose the highest voted answer, but the one that helped you the most.
@Thorbjørn Ravn Andersen: I know and I already did it before, but I still find it useful to explain why I disagree with the votes where there is a considerable gap between the highest voted answer and the one I accept.
I think you accepted the wrong answer. Danny Woods was right, failures in other industries are just as frequent, and programming isn't building it's design. Construction in the software world is done by the compiler and is virtually free. Your original question was very telling _Once the preliminary work (design, specifications, etc.) is done on paper_ the design and specification work of a physical structure is an EXACT specification of how to build the structure. The only thing matching that in the software world is code. Code is design, compilation is construction.
@MikeBrown - I view the high voted answers as complementary. What I felt MainMa was getting at is why large software **projects** have high failure rates. My reference to Yourdon is partial evidence to this being a large topic of discussion within software engineering. Focusing the conversation on design vs. build | compile ignores all the other SDLC aspects within a project.
But the question itself is flawed. While we do need to turn a critical eye toward our own shortcomings. Acting as if the software industry is the only one that has unsuccessful projects is ludicrous. To say "once all the design has been done" is telling as well. Even if you don't agree that programming is a design activity, how often do you see detailed, iron-clad design done up front for software? People give fuzzy specs on software because they anticipate being able to change the software if the specs weren't right without caring how those changes might impact the entire solution.
@MikeBrown - I think the question ignores the challenges other industries have faced and sometimes resolved. And I do agree that other industries have had their own spectacular failures. For me, the question focused on why software projects had such an abysmal successful completion rate. Programming is definitely design, but a software project is much more than just programming. I've worked on projects with tight specs and those with loose specs. What amount was done upfront depended upon the criticality of the system.
@MikeBrown - d*mn community wiki status... ;-)
The answer is surprisingly simple: those 'other industries' do have a high failure rate. We're just comparing the wrong things. Writing software is often called 'build', and so we compare it to the manufacturing or construction phases in other industries. But if you look at it, it's not construction at all: it's design. Software designs are written in code, and building is done by computers, whether by compiling software or directly interpreting it on the fly.
Many designs in other industries either take way longer than originally estimated, cost way more, or simply never see completion. Sound familiar?
So, what are we doing when we're planning software? Well, we're still designing, but at an earlier stage.
In software, there's no manufacturing line of note. Building the final component is (comparatively) cheap, and replication of that final product is both perfect and effectively free--you copy the build artifacts.
Even in the industry the OP mentioned, Aerospace, the Boeing 787 Dreamliner and JSF F-35 have both had significant delays. Last week a carpark collapsed in one of the major shopping centres in Sydney. Sydney has very rigorous building standards but mistakes happen.
A thousand times this. Even the question's schedule link shows that the project was actually in development from 1988. The source code is the design: http://www.developerdotstar.com/mag/articles/reeves_design.html
Many large projects like the F-35, Hubble Telescope, etc were late because of the software side of development. The Hubble actually sat in storage for years because the software was not ready.
@MrLane:In the real world it works like this. A schedule is set for when the hardware is supposed to be done and the software is supposed to be done. The hardware designers provide an ICD to the software team so the sw team can write their code without the hardware. The hardware slips its schedule, by a lot and changes its interfaces to work around the hw issues, frequently without notifying the sw team. Finally, the hw sort of works and is delivered, way late. Of course the sw doesn't work because of the myriad of unexpected hw "features". Because it is cheaper to fix hardware problems in...
software, the sw team now has to incorporate the changes to the ICD and come up with workarounds for buggy hardware. So in addition to the hw being delivered way late and now the sw team is also fixing buggy hardware, who gets the blame for being late? Well, the software team isn't done yet. It is the software that is late. Everyone always forgets about the electrical, mechanical and systems engineering schedule slips that sw depended upon and then that forced sw to be rewritten and have extra requirements. All they see is that the sw team is still coding. Thus, the software is late.
To point out some figures:
- Change of requirements after implementation started; for example when the first Airbus A380 started to be created in the factory I cannot believe that if someone wanted 200 more seats, those would be put there; but in a large software project even after the programmers started development 5 more types of users can be added.
- Complexity - large software projects are extremely complex; probably the most complex projects human kind designed and developed;
- Not enough resources are spent in architecture and design phase;
- Field immaturity - software engineering is relatively a young discipline compared with other engineering sisters; this has two implications: a) Not so many heuristics and good practices; b) Not so many very-experienced specialists;
- Lack of mathematical proof - in most of the cases mathematical formal methods are not used to prove that a piece of software works as required; instead testing is used. This does hold true in other engineering fields which rely more heavily on mathematics; the reason of this is as complexity;
- Rush - many managers have unachievable deadlines; so quality of code is placed second, because of the deadline.
Answering strictly to the question - I tend to believe that others have very high expectations (especially in delivery time) from programmers and do not understand exactly how difficult programming large software is.
Formal mathematical proof in software, besides the fact that it is often practically impossible to do right, is ultimately nothing more than writing the program twice (once in the actual programming language, and once in the formal-proof specification language) and verifying that both versions do exactly the same.
I agree upon the fact that formal methods are practically impossible to be applied upon large projects. The difference is that a program described by a formal specification (Z, VDM) is correct due to mathematics; but a program written in a typical programming language needs to be tested.
tdammers, there are tools that can help you write both at once: Coq supports "program extraction" to extract a program in OCaml or Scheme from a certified Coq program.
"Change of requirements after implementation started". I call this the "moving the toilet problem". You are building a house, are putting finishing touches on the bathroom, and the owner wants the toilet in a different place. You give them the cost estimate. They balk, saying "why so much, I just want the toilet 8 feet away?". You then explain that you have to install new plumbing, rip open walls/floors, etc. to be able to move to toilet. Late changes are always expensive.
I'd say testing an airplane is actually much more difficult than testing a software. With airplane, all the mathematic magic you conjured ends up useless when you figured that the software simulator or the wind turbines you created doesn't really reflect the way things works when you're up there. Formal proof in software is only a hard problem, compared to formal proof in engineering which is impossible.
#1 in your list is the most important IMO, i'd add two more things to it: -Lots of big changes can be made in a short timeframe (for example 'lets switch the communication protocol!'), that wont break stuff in the short term, but many of these make the project very hard to manage in the long term. - The changes in the environment where the software runs can change drastically over a short time. While the basic premises for an airplane will stay the same (must fly in storms, must land on solid runways,..), a software can totally break, if the new verison of the OS comes out for example.
To expand on what sydd said, there's a limit to how much duct tape and bondo you can put on a project before you get shut down and have to start all over. In software, duct tape and bondo are largely invisible even to your fellow engineers unless they deeply inspect your code, and project managers have no idea of what the difference even is compared to good design, so they'll demand more of it to make up time. Big software is often more body shop than mass manufacturing.
#3 is a good point, I've seen too many projects which start without a (good) software architecture plan. This results in a higher occurrence of unknowns which brings all kinds of problems.
In 2009 a software OS kernel was mathematically proven to be correct. It took 30 man years to verify 7500 lines of code. DO the math - how long to verify something useful - e.g. Linux - is bug free, even assuming linear complexity etc it ain't going to happen any time soon. (refer http://phys.org/news173086905.html)
@tdammers, writing a program and proving that it is correct is not the same thing. It is possible to write a correct program which will be *impossible* to prove that it is correct.
@Alexey: What I'm saying is that an automated formal proof of correctness requires a definition of 'correct' behavior in a formal language. But then, defining the behavior of a piece of software in a formal language is exactly what programming *is*.
The proof does not have to be automated, it can be done "by hand" in plain language, but probably there are some logical systems that are better suited for this. (I have learned recently about Hoare logic, for example.) Defining a behavior is not enough to implement it. Consider the problem of factorizing a given integer as the product of primes, for example.
no "Change of requirements after implementation started" LOL - see http://globalprojectstrategy.com/lessons/case.php?id=23 they didn't add seats, but instead a two year delay after manufacturing started due to having to redesign as the requrements for the electrical systems not being copied correctly from one computer to another.
The premise of the question is a bit flawed. Both the A380 and the Boeing 787 were delivered years late.
In the case of the A380 much of the delay was caused by the French and German units of Airbus using different and slightly incompatible versions of CATIA design software. This incompatibly manifested itself as wiring harnesses that didn't quite fit the airplane.
There wasn't anything wrong with CATIA, which is the most widely used aircraft design software, but someone somewhere dropped the software configuration ball.
The Boeing 787 also suffered from software related delays, but most of its problems were more traditional new airplane problems like weight control and delivery of substandard parts by suppliers.
Both the A380 and the B787 had to modify their wing designs after the initial aircraft had structural issues.
Large complex projects are difficult for humans in all fields.
Agreed. I think there's a false attitude that software is "produced more sloppily" than physical stuff because bad software ends up with fixes which are very visible, plus everyone has to deal with broken software. If a plane is a piece of crap and they have to do some work on it you don't send it back, it's just something the mechanics complain about amongst themselves in most cases, unless it's a *huge* defect. And those happen too.
I think the question still stands even though the examples are flawed. It is statistically proven, that software projects have much bigger final costs and take much longer that predicted.
It should be noted that the design and implementation of a commercial airliner system inherently includes the completion of a massive, and massively complicated, IT project, one that *has* to be fully and correctly functional.
Highway construction would qualify as large scale projects. Those are frequently completed on-time or ahead of time. Building construction is another example of large scale that can complete on-time or ahead of time without that being an exceptional thing.
And given that the A380 had a major recall as recent as 2010, I wouldn't call it "flawless" even then.
Also, LOTS of concept airplanes have been funded and canceled over the years, particularly military contracts. Airplanes are not good examples at all, because we don't hear about many of the (classified) failures until many years or decades later.
Or sometimes we do hear of the failures, and it's heart breaking - like the Lockheed Martin X-33 project.
Both A380 and Boeing 787 are planes. Planes have large and complex software installed in them that interacts with the things that the "other industries" are building. Who knows, maybe their were years late mainly because of software issues.
Skyscraper guy here. Not sure if I can answer your question but I can surely shed some light into various items in the thread. Buildings do indeed occur very fast. A major constraint is locale (regulations). But in general it takes 3 to 10 years for a tall building from start to finish.
I think comparing a new building with a new software project is not very accurate. A new building is closer to a new version of a kernel or OS. In this respect software development is much faster. We never build from zero as this will be a high risk task the client would never sign up for. Most design and development in buildings is derivative of proven and completed projects.
From personal experience only one in ten projects ever get built. The process is development-driven rather than design-driven, so the moment something like the price of steel goes up the whole project is out the window, or designed, as design is the cheap component of the process.
Design takes a month for concept to schematic, two to six months to design development, another six months to details and construction documents by a team of architects, planning consultants, structural engineers, wind engineers, services engineers, quantity and cost consultants, surveyors, accessibility engineers and the list goes on...
The argument of virtual versus physical is not very accurate. We also work mainly with virtual tools, and moreover we are both time- and scale-remote from our final product. In most cases we can not even test aspects of buildings in mockup scale and we use simulation to try predict what may come about.
Complexity-wise there are differences, but overall it is maybe about the same. We not only have interrelated units and multiple levels of tiered abstractions and interfaces but also people are very much specialized in small parts of the process that make communication very difficult.
As for the argument of design versus development, I think both processes are design-built. It sounds academically nice to keep these separated but it is not possible to design if you don't know how things work. You just increase the risk of failure.
Overall my (potentially wrong) estimation as per OP's question is that programming is more of an art than engineering. Why would people take pleasure and even do it for free, find expression and elegance in it? Computer science is also (as on the tin) more of a science than engineering. Why would you try to prove algorithms instead of just patching existing parts together and work to make things just work? Not sure if this makes any sense; I'm not a software guy.
One aspect that strikes me with software design and development is about the medium itself. Computers make human-centric work very unnatural. Everything is so very explicit and there are few tolerances. It's hard to mentally work your way around this, and some get away with it by dumping complexity within. If nothing else this may have something to do with it?
+1 Thanks for the insight. "to design if you know know how things work" -> " to design if you don't know how things work" ?
Hey builder, I suggested some edits to this post, I think you have excellent points, I just tried to clean up some grammar.
I definitely agree with the point about programming being more of an art than engineering. I often find creativity as a core aspect in software design.
I disagree with the assertion that a large software project and a tower have similar complexity - based on my experience working as both a structural engineer and a software engineer, I'd say software complexity is much higher. There's probably a raft of reasons for this, but the one I'd suggest is that there's a whole lot of wiggle room in engineering; the upper bound of construction design is almost always given by cost, and that's a soft constraint. Software needs to be *exact*, since computers don't deal with ambiguity well. Slab not working? Add a shitload of steel, she'll be right.
Some people do design and build buildings for pleasure. Don't tell my employer, but now that I think of it some of my software is like the *Sagrada Familia:* Idiosyncratic, too many ornaments, never quite finished; but of interesting design, fun to build and use and still standing.
Then how long did the design of those took? Year? Two? Ten years? The design is the most complex part of building something, the construction itself is easy.
Based on this article, it is slowly being understood, that software development is mostly design process where design document is the source code itself. And the design process is totally different from the production process. It requires experienced people and is impossible to plan, because even small requirement changes can result in huge changes in the overall architecture of the project. This understanding is the basis for agile methodologies that focus on improving code quality as the final design document and taking testing and debugging as parts of the design process, just like they test airplane models in wind tunnels.
The construction itself is handled automatically by compilers. And thanks to that, we are able to build whole products in a matter of minutes.
The reason why software projects are finished with huge delays and inflated costs is because managers still think they can estimate, predict and plan such a design process. This backfires more often than it is actually valid. They still think that by tying people into a rigid construction process they can somehow increase quality even though end result is mostly opposite with increased costs and missed deadlines.
"This understanding is basis for agile methodologies". Exactly. The waterfall method makes sense for skyscrapers: you want every detail in the blueprint to be right before you pour the foundation. But if tearing down and rebuilding skyscrapers were free and near-instantaneous, like compiling code is, you'd probably use an iterative process.
@NathanLong: Especially if they came out with new forms of concrete every three years, and somebody figured out how you could run multiple elevators in a single shaft, and suddenly steel wasn't cool anymore and everyone decided to build their frameworks out of carbon fiber. Stuff like that goes on *all the time* in the software industry.
"software development is mostly design process where design document is the source code itself" you just opened my eyes. Thanks.
@TMN Imagine if while building a skyscrapper, they'd change the color palette of the interior because it doesn't look right. That apply for any component of the building. Trying to run multiple elevator on a single shaft is one thing, but you'd have to test 20 elevator from different supplier before even trying to hook all of them to the shaft...
@LoïcFaure-Lacroix: *Engineers* might test 20 different elevators, devs would simply use the one described in the blog post where they first heard about it. Then when the building opened and there was a problem with the elevators, they'd discover that the company that built them no longer existed. When they tried to get replacements from a different supplier, they'd discover their original elevators used a non-standard shaft...
@TMN No, this is an example of bad dev. A bad engineer could also take the cheapest supplier without looking at the detail of what they're using. A good dev on the other hand would take into account the following things before choosing a component. Project alive? Easily maintainable if things need fixes? Well documented? Are there other good alternatives? Is it flexible enough or too complicated? All of the question a dev could ask himself before choosing a component are weight for making a decision.
Imagine, in the middle of the design of the Airbus A380, someone piped up in a meeting and said, "Heh, could build it as a triplane?" Others joined in saying, "Yeah, yeah. A triplane. More wings are better." The next thee years is spent turning the A380 design into a triplane. At another meeting, someone says, "A triplane? That's old. We want a biplane. Just remove one of the wings."
Or imagine, in the middle of a bridge construction project, someone says, "Heh, we just made a deal with a shipping company. They need the bridge to be another 40 feet higher, because their ships are much taller. Fix it. Thanks."
These are but some of the reasons why software projects, big and small, end in failure at an alarming rate.
This is exactly what happens. The management types or the clients change their mind every 10 seconds which just frustrates the developers. I quit my last job because of crap like this
As someone with a mechanical engineering background working in IT, I've often wondered about the reasons of the low success rate in IT.
As others in this thread, I've also often attributed the failures to the immaturity of IT, the lack of detailed standards (yes I'm serious, have you ever checked the standard sheet of a simple bolt?) and the lack of standardized components and modules.
Other industries, like building construction or ship building also have much more "beaten paths": knowledge and experience of a particular solution prototype, which - in customized form - is re-used again and again. Ever wondered about why buildings, ships or airplanes of different size and purpose somehow look so similar? (there are exceptions to the rule of course...)
That is because those prototypes are well researched, well understood, generally used and have a proven track record. Not because it couldn't be done any other way. In IT standardization is rarely the case: (large) projects tend to re-invent components, doing research and delivery at the same time and with the same people!
These inevitably lead to one-off products, which are expensive to develop and service, are error-prone and fail in unpredictable ways under uncertain conditions. And because of this, of course, these products are much quicker obsolete, written down and replaced at equally great costs with only slightly better ones. What IT needs is the equivalent of the industrial revolution, which turned middle-age artisans into efficient factories.
That said, there are factors that make IT truly unique however. As opposed to those other mentioned industries, IT is truly ubiquitous: it is used in every aspect of our modern life. So it's a small miracle IT achieved this much progress and is capable of delivering the results it does.
+1. Good example for the standardization (standard sheet of a simple bolt). In IT, rare are the components which are normalized. Take registration forms: every website reinvent their own, and few are the developers who know how their registration form behaves with unicode, with empty strings, with strings too long, etc.
@MainMa: poor example, do you create your buttons, textboxes, option boxes, option boxes from divs? No, you reuse the browser's form elements; and the browsers used the Operating System's form elements.
I were rather speaking about the internals, not the controls. Take some random website. Can you use Chinese characters for the password? Can you use 25-characters passwords? What will happen if you put a whitespace in password or user name? All this could be normalized, but it's not, and every person is reinventing the wheel for every project, often badly, i.e. no hashing and/or salting, or passwords limited to sixteen characters (example: MSN), etc.
Changing IT from "artisans" to "factories" may not be possible. Factories are executing a process which has already been created. Workers in a factory execute their process with little or no human thought. Many factories have replaced humans with robots due to this fact. In software you are not executing a process, but creating one. Creating software would be more akin to designing the factory and it's processes rather than running the factory. Although software creation could benefit from standards, it cannot fundamentally become a factory.
@ArseniMourzenko it's a bad comparison to compare "data sheets for bolts" (i.e. tools, equipment) to "registration forms standards". Registration forms would be more like "a roof" or "a front door" (indeed, there are a zillion ways to make those). What a bolt compares to is more like a processor pipeline behaviour. It's nowhere close to what we need: reliable OS (with _rigorously_ defined characteristics, not "data may hit the disk depending on the mount options used") and ditto development tools (they need to be able to check 90% of the code for standard properties)
I'm afraid that I disagree with your statement.
Airbus and Boeing are two examples of companies that build planes. How many companies that build planes are there? Very few, if you would compare it to how many companies build software.
It is equally easy to screw an airplane project as to screw a software project. If only the entry barrier was so low in the aircraft-building industry as it is in the software industry, you will certainly see many failed aircraft projects.
Look at cars; There are high-quality manufacturers that build very durable and highly advanced automobiles (think Land Rover, Mercedes) and there are ones that build cars that won't last a year without having to repair them (think Kia or Cherry). This is a perfect example of an industry with slightly lower entry barrier, were you start to have weaker players.
Software is no different. You have lots of buggy products, on the other hand, Windows, Office, Linux, Chrome, or Google Search are very high-quality projects that were delivered on time and had similar quality level as an aircraft.
The other point that many people miss is how much maintenance goes into maintaining a car, a tanker or an aircraft that we just take as a fact of life. Every plane has to undergo a technical check-up before every take off. You have to check-up your car every several k miles and do so regular stuff like change oil, change tires.
Software needs that too. If only people spent as much time on diagnostics, prevention or auditing software's state and quality as they do with mechanical/physical products, I would expect way less statements like these. Do you read your application's logs each time before you launch it? Well.. if it was an aircraft you would have to ;-)
Windows has often not been delivered on time (Longhorn, aka Windows Vista, was originally supposed to ship in 2003). I don't know about Office, but most of the other software projects you mentioned explicitly don't have timelines, or their release schedule is so frequent that they include whatever features are ready in the release, and push everything else off until it's ready.
This is a better example of high quality software: 420,000 lines and bug free: the software that powered the Space Shuttle. Mind you, there was enormous costs associated with getting this sort of quality.
It took something like a decade for Windows to get to a decent quality state, too.
Digital building blocks can be 1 or 0. There is no inbetween.
A mechanical design has a level of tollerance. I can put one less than perfect rivet into a bridge and it will most likely will not fall down, however, in code even just once instance of putting a 0 where a 1 should be can fail the entire program.
Due to the logical and interative nature of computing, any, even very small changes, can lead to drastic failure.