One way to look at Salon co-founder Scott Rosenberg's new book, "Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software," is as an attempt to tell the story of a specific software development project -- the effort by industry legend Mitch Kapor and a band of ace programmers to create Chandler, a kind of turbo-powered personal information management program that would dazzle users with its ability to enhance their productivity.
On that level alone, the book is successful, even though Chandler's development process has been bumpy, and some four years after the project began, it's still not finished. Like an anthropologist recounting the behavior of some hitherto unknown tribe, Rosenberg brings us inside the world of this particular group of software developers and documents their quest with detail and insight.
But the story of Chandler is also a platform for Rosenberg to explore much larger questions about the nature of software. Such as, why is it so hard to pull off a big project on deadline? What is it that software developers actually do? How is the art of writing software different from other forms of creative endeavor or technological construction?
With the years-past-deadline rollout of Microsoft Vista this past week, Rosenberg's questions (and answers) are especially timely. And given his long association with Salon, we are delighted to showcase them here, even as he dashes between speaking engagements at Microsoft and Google and Yahoo and writing Op-Eds for the Washington Post.
Disclaimer: Scott Rosenberg was responsible for Salon's hiring me 10 years ago, was my editor and boss for many years, and is a close personal friend. My daughter baby-sat his twin sons as I "interviewed" him. So I'm utterly biased and completely partial. But "Dreaming in Code" is still a darn good book.
Is it fair to summarize "Dreaming in Code" by saying that it is basically an attempt to answer the question: "Why is writing software so hard?"
Yes. That was the question I started with. It dates back to the experience at Salon that I start the book with. I had moved from being a journalist and an editor to being the managing editor of a Web site, and we had decided to build, on our own, some software for Salon: a new content management system. But as I tell it in the book, it turns out to be a software disaster, a classic death march situation in which everything went wrong and nothing worked as planned and when we deployed it everything broke.
And it was inconceivable to me, as it was happening, that this could be so. The fact that we were undertaking this thing that many people had done before us, we weren't the first, and that it could be so out of control, so unpredictable -- that threw me. I thought, OK, I don't know anything about this, I'm a writer and a journalist. I'm ignorant, but let me go educate myself. And so I did, and the first book I read was "The Mythical Man Month" [Frederick Brooks' classic study of the difficulties inherent in large software projects]. And I quickly realized that our experience was not unique at all. In a way it was the norm.
I've noticed that this week you've been in demand as a commentator, both in the Washington Post and on Marketplace, with respect to the launch of Microsoft Vista. Are you saying that there is a common thread between the problems that plagued our little Web site, and the delays experienced by the biggest software company in the world in rolling out its newest version of Windows?
Well, they're clearly hugely different undertakings. But there is probably more than one common thread. The one that comes first to mind is simply the incredible difficulty of estimating the time it takes to do this stuff, whether you are building a little content management system for a relatively modest-size Web company or whether you are building the operating system that will be used by three-fourths of the known universe. The difficulty in saying, A) How long will it take to do what you want to do? And B) When are you done doing what you want to do?
But you know, there are people in the field of software development who would look at Salon's story and say: "This is a very ill-disciplined project," and they would look at Chandler, the project at the center of "Dreaming in Code," and they would say, "That is a very ill-disciplined project," and they would look at Vista and say, look, they spent a couple of years trying to rewrite the file system and take on all these ambitious things -- "They were really ill-disciplined."
Can these people point to actual examples of disciplined projects?
Yes, they can, but usually they are extremely narrowly defined, well-known things -- situations where you have a very specific need, like an embedded system, something that is running on a single piece of hardware that is part of some larger system, and all you need to do is write the code for it and everything is specified in great detail. Those kinds of systems can come in on schedule.
But any time you look at a project that is trying to do something new, you have a different story. In the book this is what I call, in a sort of tongue-in-cheek way, Rosenberg's Law. I got to this point in my writing where I realized that everybody else in the software world has a law -- there's laws galore! -- and surely, if I'm going to contribute anything in this book, I have to find something that I can hang the label "Rosenberg's Law" on. But in fact, in the course of writing this story, it came to me in the middle of an argument that I was making that as long as you are not trying to do something new -- if you are doing something that has already been done -- then you have a frame of reference to estimate how long it is going to take, and to guess how many people are going to be required, and so on. And of course in other fields of engineering where we do the same thing over and over, the same is true. If we are going to build a house, there are always imponderables, and every site is different, but you have a rough idea: two-story house, and such and such a set of features, it is going to take a certain amount of time.
How do you define "new" in terms of a software project?
In every big software project today, there are always areas that are repeating stuff that other people have already done, and then there are the parts of it that are the new parts, that are the reason you are writing the software -- because if there weren't any new parts there would be no point in building the software. But it's a good question, because this is where the mind-set of developers affects the process and where projects often get into trouble. You're doing the project because there is this new feature or features that you need. The developers will often look at this and say, well, we know we have to do that, but then there is all this other stuff that needs to be done: For a content management system, you need an interface for putting in the text, you need a database, all these basic things. The developer might say, I could take this thing off the shelf that exists already and plug it in, but it's going to take almost as long for me to learn how to do that, or maybe even longer to learn it, than to write it myself.
And programmers, as I quote Larry Constantine in my book, programmers are programmers because they like to code -- given a choice between learning someone else's code and just sitting down and writing their own, they will always do the latter. And the programmer who says, it will be faster for me to write it, rather than to learn it, is usually correct. Except that what he will write, most likely, is something that will work but will not have its rough edges worked out, will not have the benefits of a piece of software that has actually been used for a few years, where the bugs have been found and the users have given feedback and have helped you figure out where the problems are. So what they will often be handing you at the end of that I-can-do-it-faster-myself thing is something that works, but that is kind of a mess in certain ways. Whereas the thing that you were going to pull off the shelf, maybe it will take the programmers a while to learn it, but once they learn it enough to hook it up to this project you are creating, what they are hooking up will probably have a lot fewer problems.
You always hear, with regard to software, that managing developers is like herding cats, and what you just said kind of alludes to that. Does that mean one of the problems with getting software projects done on time is the psychology of the people who are actually doing it?
That's a huge and fascinating subject. There is this raw fact in the field that the best way to create software would always be to have just one person write it. At Salon, our original content management system, the predecessor to the big one that was a disaster, was a much smaller system that one developer wrote. It was limited, but the process of creating it was much less painful.
So then we went from one developer to a team of a half dozen people. One person can keep it all in his or her head. But the moment you go to two people or beyond you introduce all of the possibilities for miscommunication between people. That is one of the things that readers of "Dreaming in Code" will get a sense of from the story. I have huge respect for the developers of Chandler, led by Mitch Kapor. They are all really good at what they do. There wasn't a lot of internal politicking, it was a pretty straightforward project in a lot of ways, and yet they had great difficulty getting on the same page, just on things like the terminology they were using.
There were some unique design concepts that they were trying to introduce to the world of personal information management, and just coming up with language to describe those things, in such a way that the software could be developed by multiple developers, turns out to be really hard. And that is less a function of the mentality of developers than it is of the insubstantiality of the product of software -- the fact that it is made up, it's layers of stuff that have been made up on top of other layers of stuff that have been made up, all the way down to the bottom.
Is there a sense of progress in the field? Do people know now how to do things better than they did in 1980?
I think there is a sense of progress, because clearly we have programs that do more extensive things today and that are used more widely by more people. And there are things that programmers never stop to think about today that they had to worry about 30 years ago. For better or worse, and there are some people who say it's worse, a beginning programmer today does not learn any machine code. That has been abstracted out of the picture, and a lot of even the lower-level stuff that is above machine code that programmers used to learn about they don't learn about today. There are some people who argue that that's a problem, but most of the time, it enables people to get more done faster.
But it is slow progress. It's linear progress.
You followed the Chandler project for four years. Was it frustrating to see the project get delayed beyond the time frame of your book deadline? Or was the fact that they didn't finish on schedule a fitting example of exactly what you were writing about?
I hope that readers won't feel dissatisfied or frustrated. I feel like it's still dramatic, but it's drama like "Waiting for Godot." I've talked to Mitch Kapor and the other people there, and I know they are a little frustrated with how that happened, because they feel like they've made a lot of progress since the end of my story. They are going to do what's called a "preview release" of Chandler in April. I'm excited about that -- I still want to use that program that they dreamed about building. But I felt that I had reached a point where, for the questions I had started with, I had as good answers as I was going to get. And the open-endedness of the book's story, I feel, is actually a reflection of the book's themes.
You recently spoke at Yahoo to more than a hundred people. Were they expecting you to help solve their software development problems?
I hope not! No, I don't think they were expecting that, and anyway, I certainly didn't solve their problems. When I started writing this book, I knew right off the bat that I needed to be really clear with people that I wasn't writing a how-to book, and I wasn't writing a book of advice. First of all, I don't think I'm really qualified, and second, the shelves are groaning with those books. That wasn't my goal.
My goal was really more to write something that, if you were a developer, then yes, you might find it interesting. But even more, if you had a relative who was always wondering, "What is it that you do all day?" you could hand my book to that relative and say, This is what my work is really like.
Shares