Jorge Luis Borges, The Library of Babel
Jorge Luis Borges' haunting short story provides a crucial insight into the problem of design: the vastness and order of "design space". Daniel Dennett elaborates in the idea quite a bit in Darwin's Dangerous Idea.
In abstract, the Library is simply the combinatorial space of all combinations of length P from a set of N possible symbols. Borges' genius is to have given concrete values to P and N, and to have superimposed a concrete spatial vision over the mind-numbing vastness of the resulting number (10 to the 1,823,342).
The Library encompasses every conceivable text, meaningful or not. The number of books which contain some meaningful text in some language is still unbelievably large, as is the smaller number of books consisting solely of meaningful text in one given language. For any given book, there is a still freakishly, unimaginably large number of books which differ from it slightly enough (a comma, a few letters) as to make no noticeable difference. For any idea, there exist a stupendous number of books expressing that idea - and as many expressing its exact opposite.
This is the metaphor appropriate for understanding what software development consists of. All conceivable programs exist somewhere in the Library of Babel, or a version of it slightly modified to allow for the ASCII encodings we favor over Borges' original alphabet. Nearly all of them are meaningless jumbles of characters, but an astronomical number are programs which would compile if given to some compiler. For any given set of requirements, there is a freakishly, unimaginably large number of programs which implement these requirements.
The writing of any program, like the writing of any text, is a non-random exploration through a molecule-thin slice of this vastness that is design space, which if it is successful results in finding one solitary point (one book, one program) the effects of which are appropriate when read in an appropriate context.
For an idea of what "appropriate" might mean, and of how strategies for searching points in this vast space are formulated, one must consider a third kind of design space; that which encompasses all possible strings of DNA. This is the design space which has been explored for millions of years by evolution under natural selection.
Dennett's book is a superb discussion of how to think about nature's strategies for exploring this immensity of design space, and how they relate to (in fact, build up) our own minds' strategies for exploring analogous spaces. To anyone involved in such explorations - any writer, any programmer, any designer - these ideas are simply indispensable.
-- GregGrappone