DoxPara Research
02-Mar-1999 / Dan Kaminsky Core Competencies: Why Open Source Is The Optimum Economic Paradigm for Software
Executive Summary

Open Source Software has proven itself to be inordinately successful in providing high quality software in a method that, on its face, defies standard business reasoning. This article clarifies OSS in the language of economics such that the true reasons behind its success may be understood by everyone, not just the coders. The following summary, written by Jack Carrol, well explains the contents of this essay:

Open source software development is a barter economy with some very unusual economic properties. The original author of a piece of OSS gets paid, but not in money. The author gets paid in software. So does everybody else who contributes a feature or a bug fix. This form of payback is very real, for all of its intangibility. The return is huge -- everybody who contributes to the growing program ends up with the use of a much more capable and higher-quality product than they could afford to develop on their own, and they get a product which has been carefully enhanced to meet their exact needs. The actual developers aren't the only contributors and beneficiaries, either. Everyone who contributes documentation, distribution services, mutual tech support, or publicity participates in this barter economy.

Some other highlights:

  • Source Licenses, where businesses pay to receive access to the codebase, are a far different creature than Open Source Software, as they end up placing fewer eyeballs against more bugs--in essence, it devotes less resources to more weaknesses.
  • OSS means outside engineers see your code before it crashes, while it remains in the development cycle--and these engineers are likely to assist in the fixing of your bugs, saving you substantial amounts of money.
  • Closed source code development forces sites that require custom features to choose between software that is too expensive to produce or code that is too unstable to trust, with significant diversion from core competencies unavoidable due to the massive redudant effort that is necessary for any complex project.
  • Closed Source software providers are actually less likely to provide timely code fixes due to their organizational structure, and the "insurance premium" that managers justify paying out large amounts of money to a single supplier with actually ends up creating a support monopoly, in the sense of the supplier knows that it´d be too expensive for you to leave and absolutely impossible to find anyone else who could patch the code. LinuxCare can be founded, MicrosoftCare never can be.
  • OSS merges the advantages of custom code with the advantages of Commercial Off-The-Shelf(COTS) products, essentially creating Modifiable COTS, minimizing the amount of inefficient labor that must be expended to create a product.
  • OSS allows hiring managers to truly understand the skills of an individual who has coded under an Open Source license, thus increasing both his hirability and his salary range--OSS developers, quite frankly, are more bankable, especially since they have the ability to prove themselves worthy of any position, not just one that fits the position they´ve taken at a given company.
  • Specialization, and Ricardo´s Theory of Comparative Advantage provide significant backing behind a free and open code architecture.
  • When one examines the rest of our society, it quickly becomes clear that when stability matters, openness is the only way to achieve it. Stability matters for business. Period.
Introduction

Open Source Software works. A simple statement, but finally understood--code developed openly, without significant restrictions on redistribution, works better than code developed under almost any other software methodology. Business is finally beginning to accept us on our technical merits, although it still doesn´t yet understand why we´d work so hard on something we just don´t seem to be getting paid for. They know what we´ve done, just not why we´ve done it or how we´ve managed to do it so effectively. We have proved our paradigm--and considering how different it is from "established" software business practices, it truly is a paradigm--successful, and like any success story, there are many in the world who are desperate to understand why, so that they too may follow our road to success. They still have their doubts, and contrary to what some of us might think, they have good reason to doubt. Many in the industry we would drastically change have honest, good, and valid reasons for being doubtful of the seemingly siren call of Open Source. It is not enough for them to understand the empirical reality that OSS is blisteringly, frighteningly, and tremendously successful in its pursuit of speed, stability, quality, and soon, ubiquity. They must know why, in terms they understand--and, since it is always valuable to understand the perspective of users, so must we.

The Ghost Of Source Code Past

One of the primary reasons for experienced members of the IT community posessing a degree of wariness surrounding Open Source software is that it seems like nothing more than the old promise of source licenses. For a very long period in computing, publishers made the source code to all applications commonly available and often licensed it to large businesses for mission critical applications. Such licensing caused serious problems if the business hired programmers to add the features they required to the code--how were the original authors supposed to support something they didn´t write? Needless to say, an environment that is difficult or impossible to support is difficult or impossible to keep reliable, so businesses were forced to forgo any site-specific code patches and only use the licensed code for reference purposes.

This, of course, wasn´t always businesses licensed the code for in the first place. Some had been sold on the idea of maximum flexibility and maximum reliability through the ability to hire programmers to customize everything down to the core. But the empirical reality was that once a site began making such modifications, they were required to hire additional programmers to help support something that nobody else could have the means to understand. The final results were so ugly--massive buck-passing, unacceptable downtimes--that the predictable and supportable comfort of closed source software simply became the new status quo.

This, of course, is the oft-ignored voice of experience´s Tulip Claim against Open Source Software: We´ve had access to the source before, and it made things worse. Why, then, should "Open Source" be any different?

The short answer: Open Source means outside engineers see your code before it crashes--and can do something about it.

The Evolution of Access

Linus Torvalds, creator of the classic Open Source success story known nigh-eponymously as Linux, is known also for what Eric S. Raymond named Linus´s Law of Software Engineering: "Given enough eyeballs, all bugs are shallow." The inverse often also holds true--"Given few enough eyeballs, all bugs are deep." Linus´s law explains the false promise of the source licensees. Having spent so much money obtaining the code they were to modify, pressure existed to keep the code out of as many eyes as possible. Exposing too much to the original authors might lead them to rip off company code and put it in the next version, so formal or informal limits to the lines of code that could be transferred were imposed. Worse off, even these limited code transfers would only occur if a site got desperate enough to actually seek external support. Source licensees were faced with the problem of fewer eyeballs facing more bugs--a recipe for disaster if there ever was one. Open source works far differently. While both methods grant access to the source code, open source demands that all code written be freely available, with limitations on the distribution of modified code forbidden. This seems ludicrous from a business perspective until one realizes the importance of sticking to one´s core competency. An architecture firm is not a software firm, and neither is a clothing store, but both can and do have needs for custom code that fits their needs, works correctly, and can be supported to the required degree. Engineering software in-house, either from scratch or by use of an costly license, requires expensive programmers to execute redundant labor to write inadequately tested and difficult, if not impossible to support code,. Worse, it requires businesses to divert more and more of their resources away from their core competencies as their systems remain unreliable. Outsourcing code saves much of the hassles of hiring and maintaining programmers, but brings in its own host of problems beyond the management tax while addressing few of the quality issues that any closed source development will incur. The fact is, testing and support requires manpower that is doubtful to be significantly less expensive for an outsourcing company vs. an in house department, meaning cheap testing is cheap testing no matter where you slice it. It is no wonder that closed source software producers enjoy this paradigm--there´s simply so much more control for the original authors when customers can´t put their own teams together to implement a feature or fix a bug they desperately need fixed.

The Best Code Money Can´t Buy

Large corporations tend to believe that their payments lead to software companies "looking out for them"--and, by corrolary, that free code is completely free of obligation and quality. Professor Manoochehr Ghiassi, chair of the department of Operation and Management of Information Systems at Santa Clara University, explains:

Managers would prefer to pay a company an "insurance premium" so that they may interact with a corporate support structure organized in a way they are accustomed to. This is why, historically, the Santa Cruz Operation(SCO) would win clients over the far less expensive BSD based systems.

It is therefore sadly ironic that most closed source software companies refuse to address security issues in their software until public exploits for those security flaws are released. Time and time again, we learn of an attack mailed to the programmers weeks before the exploit release, and time and time again we can only see something done about it some days(or weeks) after the vulnerability is exploited--in other words, nothing is done until somebody loses time, money, and productivity from a flaw. Dr. Ghiassi explains why this is:

Why do you think that [companies] delays the distribution of patches for so long? Writing code is the least expensive part of any fix. Distributing the new code to each customer is traditionally so expensive that delaying the release of patches until the next revision of the software is the only affordable solution.

A further issue, mentioned by Red Hat Software, is that quickly releasing new versions leads to resellers demanding their now-obsolete unsold product be exchanged for the newest release. Microsoft´s inclusion of the Critical Update functionality, which automatically checks the Internet for "important fixes" for Windows 98 is a development to be cautiously lauded for solving this very problem. (I say "cautiously", since the "trojan patch" that simultaneously attacks all machines is much simpler to execute when everybody patches simultaneously.) But, even with technological advances, Microsoft and most other companies can only justify crisis-speed responses after, and not before, the occurance of a crisis. The problem is not technical, just organizational. Someone must beg before they will respond.

This behavior is not surprising, given the minimal harm a client can do to their bottom line--this is especially true in the case of Microsoft. Large companies may want someone to sue, but amazingly software companies are practically untouchable. These closed source suppliers are legally absolved of any responsibility due to claims of artistic right--basically, software is like speech, and to mandate correctness would be akin mandating perfect grammar. These are valid claims, mind you, until the industry turns around and succumbs to the urge to patent such "art" suspiciously like a manufacturered product design. (This does little to enhance credibility and will probably eventually backfire in the face of the software industry. It´s not too late...yet.) As things stand, only the moral and technical consciences of programmers can supply a pre-emptive strike against insecurity.

Unsurprisingly, we only see such moral and technical consciences exhibiting themselves in the open source world, where code can freely travel from programmer to programmer. This ought to ring strange to businessmen--why would the uncompensated organization or individual be so much more likely to operate in the best interests of its clientele as compared to their compensated counterparts? There are a couple reasons.

Support Monopolies are a major cause of the existing situation. The closed source software paradigm gives users the choice of either putting up with whatever the original authors provide or ceasing usage of their software. Since the cost of switching from an installed application can be substantial, businesses are forced to accept the whims of a monopoly agency that "holds all the cards", to borrow a cliche. Eric S. Raymond, author of the industry classic The Cathedral And The Bazaar, euphemizes this relationship as "a bad strategic business risk". Whatever you call it, the effects shouldn´t be surprising--most monopolies behave with such disdain for their customers. The basis of the monopoly is that the authoring company under the closed source paradigm is the only company that can both see how the code really works and fix any problems. By switching to Open Source, the authoring organization still possesses a competitive advantage in providing support, but other individuals and organizations can still learn the platform and reduce the end user cost of receiving support. Linux, unlike the free Unixen of the past, has been able to parlay inexpensive computers and inexpensive network access into an amorphous but quite real user support base. Extremely technical questions are routinely answered, whereas many closed source outlets make it difficult to even find a way to submit bug reports--perhaps they believe such a link might constitute an admission that their software has bugs. Open source can be so successful providing support with no backing organizational structure since the large number of individual specialists are all available to answer those questions which are comparatively simple for them to answer. This support environment is self-scaling--the more users come in, the more users learn and are able to support eachother. However, many businesses do not feel comfortable dealing with such an undefinable support architecture--unlike closed source, these businesses can choose alternate support providers while still using the same software. LinuxCare is slated to handle the support needs for Dell, and IBM is also building Linux support systems. There can never be competition in the market for support of a closed source architecture--there could never be a MicrosoftCare--but open source can create support that lasts as long as there is demand, even if the original company goes out of business. It should be painfully obvious which platform truly provides better opportunities for support. Open Source´s reach goes beyond resolving the support monopoly. According to Steve Rostdedt of Lockheed-Martin, each business often has to choose: Either develop code that specifically fits the required specifications, or purchase code that is COTS, for Commerical Off The Shelf. Once purchased, the overall system is "hacked" such that the (very inexpensive) COTS solution can be used in place of its customized and very expensive replacement. Needless to say, there are definite problems with both approach. While custom code will function exactly according to specification, it can only do so at great cost to stability and the bottom line. COTS is cheaper and ostensibly more stable in a general environment, but often requires significant modification to systems surrounding it. In fact, these modifications often end up costing as much and reducing stability equal to the amount supposedly saved by going COTS--Steve tells stories of even the most staunchest COTS advocates scaling back, simply because they got burned by this so badly. The interesting thing, though, about Open Source is that it´s essentially Modifable COTS--a company can take advantage of software that does 90% of what´s needed and literally add in the remaining 10% without either starting from scratch or making do with out it. In that way, Open Source can absorb the best advantages of custom code while leveraging the economies of scale that COTS provides.

But how has Linux, through Open Source, managed to advance its code to the impressive state it stands at today? Linus´s Law, once again, states that with enough eyeballs, all bugs are shallow. The Open Source paradigm increases the supply of eyeballs available to scan and satiate the demand for stable and effective software. Interestingly, both the quantity of buggy and/or suboptimal code replaced is increased as the price per fix approaches zero. Zero? People can not live on zero dollars, nor can businesses survive on zero revenues. How can zero cost software make economic sense to anybody involved, except for those who receive it? In short: Trick question. The software isn´t zero cost, and even those who write the initial code receive update code in return. Consider: Bad code equals more work, while good code doesn´t. Badly written code must be repeatedly fixed, hacked around, modified, tweaked, ruled out, and, when the eventual upgrade occurs, checked extensively for "useful bugs" that the rest of the codebase depends on. The longer bad code goes undetected, and the longer it remains in a codebase, the higher the expense it creates in both time and money. The Open Source process decreases the detection period and shortens the amount of time and energy necessary for a corrective patch to be written--essentially, OSS removes much of the expense of bad code. There is another benefit of the time compression that should not go unnoticed. All code written for corporate usage goes through a limited development period before it is sent to production. It is in this period that it is, by far, most cost effective to find bugs and patch them. It stands to reason that the only thing better than bugs being found before they can cause a problem is bugs being found before they´re put in place to create one. Open Source´s unrelenting nature of code improvements is the only process known that can affordably do this. Finally, additional patches of necessary features are publically submitted, thus increasing the amount of useful code in the codebase. The sheer existance of Linux, with an estimated R&D expense of over two billion dollars covered entirely by volunteer labor, is proof of the viability of this argumentation. In essence, since the expense of developing software without OSS is thus higher than the cost of developing it in an open environment, OSS must be credited financially with reducing costs--in a sense, the process pays for itself in speed of quality development.

In the final analysis, business requires stable software. No amount of money can outweigh the systemic and malignant effects of downtime of corporate computers--everything from morale to public relations to bottom line income and productivity are slashed when the tools that the workforce depends on just aren´t there. The GNU Development Platform with which most Open Source software is written has finally reached the level of maturity that not only acceptable but best-of-breed solutions may be built with it. Apache, the open source web serving software, does not hold a commanding majority market share against the Microsoft marketing juggernaut merely because it´s free. Linux, the open source poster child, has not been the only Unix to see its share grow simply because of its Penguin mascot. Linux and Apache are stable under extreme load. If you accept the argument that downtime is extraordinarily taxing against the financial and human resources of any company, one must account the prevention of these downtimes as a dollar value win for the company. Open Source is loss prevention, pure and simple.

All Things Large And Small

Yet, while significant value is delivered to the company, what about the individual? Frankly, most of the stability concerns of the large company simply are not applicable to one person. If my computer goes down, I reboot it. Annoying, but it´s doubtful to affect third quarter profits. What economic rationale do I have to produce, debug, or support Open Source software? The simplest answer is that it directly affects my hirability and salary range. Human resources people don´t have much to work with under the closed source paradigm--A recommendation letter from a job the applicant no longer works at, claims of business duties verifiable only by asking a reference who can be sued for excessively contradicting the applicant, and an interview process that penalizes the meek and rewards the liars. Worse, anything that smacks of actual examples of what the applicant actually wrote at the previous place of employment risks serfdom-suits over the divulging of trade secrets. Open Source gives companies much more to work with. Instead of claiming, applicants can prove their skills providing support, can show their own source code, can provide evidence of their contributions to large software projects. This makes OSS developers more bankable, and thus worth more on the pay scale.

What is good for the developer can be very good for the development house. Untold millions have been made from designing web sites that run off of Unix and Apache. If stable code is key, the ability to prove one´s prowess using it--free or not--is extremely valuable. A company that not only develops sites with the code but also develops the code to its fullest degree strengthens its online presence considerably as the new code gains the appreciation and respect of the net community. This "saves" potential clients the trouble of worrying whether a certain supplier is a possible scam artist and paves the way for the organization.

Therein lies the answer to the question James Sullivan posed to me. "Suppose," he began, "suppose you could earn one hundred thousand dollars if you published your essay in a magazine. Would you still release it freely?" I paced for a moment, and replied, "If I lived in a world where my essay had a value of a hundred grand, then open source means that I´d be reading millions of dollars worth of the essays of others, and all parties involved would be rich." There´s a more subtle advantage for the individual as well. While one doesn´t have tremendous influence over what projects one is placed on in a corporation, the entire range of code, from graphical manipulation to network file serving to device drivers are open for the individual to improve in the Open Source landscape. Such availablity of resources allows individuals to experiment without risk, express without changing jobs, and specialize in whatever best matches his or her expanding skillset. Economic theory is greatly respectful of specialization as it is the means by which resources are best allocated. Ricardo extended specialization into the international realm in his theory of comparative advantage. Comparative Advantage holds that the global economy benefits most when each state produces that which it generates with the greatest efficiency. In the case of software, the goods are various tools and applications, the states are individual programmers, teams of programmers, or entire conglomerates, and the barriers are licensing policies that prevent access, modification, or redistribution of the original source code. While an organization may hold the ability to write certain code with the greatest level of efficiency, testing or debugging procedures may be comparatively less expensive by other programmers. Without large amounts of money providing incentive for otherwise suboptimal code to be used, the entire system shifts into a meritocracy, with each programmer and each organization providing the services they are most comparatively able to achieve. Just as Ricardo would predict, this minimizes the cost of designing software that otherwise may never have been created, thus maximizing the value of code at minimal cost.

Interestingly enough, though, most people who code Open Source Software have never heard nor thought of much of the previous argumentation, although there usually is a vague realization that somehow the code might get them a job. The true reasons many people work on Open Source Software are non-monetary, but assuredly not without reward. Some people patch open source code for the same reason others replace the shocks in a classic car. Others help support a widely used application because it was given to them freely and they receive human appreciation for their assistance. Finally, some people start their own projects because they need their computers to do something that they just won´t do yet. For a sense of progress, appreciation, or usefulness, people will circumvent the intermediary of money if the cost is low enough and the reward is great enough. However, open source development need not be unpaid--a single sponsor can wholly subsidize the full time professional activities of an OSS developer, and systemically the price per unit delivered of each bug fix will remain negligibly above zero. However, both the sponsor and the programmer will receive supplementary payments in the form of public appreciation, mindshare, good will, and perhaps influence in the future design of the whole product. In other words, the sponsor receives exactly what advertising is supposed to buy it by literally handing users something that cost the sponsor little but contributes significantly to the fulfillment of the target´s desires. It is advertising canon that targets are most likely to respond positively once they´ve been provided something desirable by the advertiser, and thus the cost expended by the sponsor in subsidizing the OSS software becomes money well spent.

There are further reasons why OSS ends up being more psychologically worthwhile. When one writes a patch at a large software company, the credit for that patch is eternally buried within the logs, kept only in case that code ended up flawed so that person can be disciplined accordingly. The overall corporate brand stays strong, but the individual is left having patched something "that shouldn´t have been wrong in the first place." The collaborative nature of Open Source is far different--a useful patch can often result in a appreciative email from the lead coder, and credit is proudly placed in the announcements of another version. The power of this should not be taken lightly--economic theory is very clear that money is only as good as what it buys, and psychology tells us that appreciation and recognition are arguably primary ends that individuals will invest significant energy to achieve.

Finally, the "self-scaling" nature of open source support also applies to open source development, as the more valuable a certain nascent project is, the more programmers will attempt to join. It is estimated only five to ten percent of the Linux kernel remains written by Linus Torvalds--It is a good thing to end up with code better than one could create by ones self! This additional support, plus the lack of a boss demanding x hours of work, mean an individual does not need to devote himself full time in order to be useful, this can occur--another advantage over closed source where an individual must devote themselves to a single organizational position in entirety. It is the demands of the closed source paradigm that provide the necessary contrast to understand the critical advantage of Open Source: Freedom, with a low barrier to entry. The Internet, as it exists now, almost completely eliminates the marginal costs of code development by facilitating discussion and data transmission among the various members of a development group. With an open operating system, open development tools, open networking subsystems, etc., new programmers do not need to spend anything more than their time to add value to the overall system. The importance of this can not be understated. It´s an economic truth that markets that are difficult to enter are almost always overpriced and underserved. Markets where competition is possible almost always create the best solutions--even Open Source developers compete for who can write the most elegant patch to a newly discovered flaw. There´s a good reason the computer industry has always favored open platforms over closed --witness the success of the entire x86 platform from serial port to PCI bus, the proliferation of HTML(which allows source code viewing of any and every page), etc. Openness works--it always has.

The Real Status Quo

Professor Larry Iannaconne of Santa Clara Univeristy writes:

[Open Source Software] requires some very special features of production and distribution (e.g., marginal costs of distribution are nearly zero, so that the socially optimal "price" is zero). On the production side, costs are substantial, but can be spread among numerous people--indeed, the comparative advantage argument may imply that costs should be so spread. The analogous activity is academic research, pursued with a relative high degree of collaboration and interaction, performed for little or no direct money pay, rewarded primarily through reputation, and (key) performed in tandem with complementary paid work (teaching and administration).

Such is the life of the programmer who decides to set out and publish code under the GNU Public License. The dirty little secret, it appears, isn´t so much that openness is the Hot New Thing but that openness has always been the standard. Consider--it is very expensive for a business´s headquarters to physically collapse, therefore blueprints are subject to public inspection and actually must conform to many regulations such that the building remains safe. Medicine too demands the rigorous trials of peer review to determine whether or not a certain procedure can be considered safe. Indeed, to protect our freedom, we demand of our government open laws, open debate of those laws, and open access to the entire political system.

Our education, workplaces, our lives, and our liberty all exist under the penumbra of Open Source. To deny the value of openness is to deny those protections we take for granted on a daily basis. The burden, if you will, is truly shifted to those who would keep their code secret in an attempt to aggrandize their own company at the expense of their customers to prove the value in their actions.

Access Archives
Mission
DoxPara Research exists as a repository for information security analysis, UI theory, and the miscellaneous writings of its founder, Dan Kaminsky.

Authorship

Writings
ZapMail Redux
RFID Security
The Absentee SIGGRAPH 2002 Review
Deaf and Dumb: A Critique
Speech Vs. Vision
Why Most Albums Suck
Tracing Smart Fridges
Password Rejected
Trinity Redux
Thoughts On Secure Deletion in 2001: Part 1
Thoughts On Secure Deletion in 2001: Part 2
On The Nature Of Data Shredding
Cryptography Doesn't Save Napster, and The War Over Parodies
Passfaces: An Intriguing Way To Authenticate
BugTRAQ-- Re: Security Hole in Win2K's FTP server

Security and Networking
Insecurity By Design: The Unforseen Consequences Of Login Script
TCP Chorusing in the Windows9x TCP/IP Stack
Vectorcast

Editorials
Core Competencies: Why Open Source Is The Optimum Economic Paradigm For Software
Mandatory Registration: Bad Business

User Interface Proposals
Analogous Key Arrays
Cluehunting