Over the past year, the information technology elite have started to dismiss Linux as a flash in the pan that tried and failed to dominate in a world owned by Windows. Woebegone Linux and open-source companies are scattered across the landscape like so much shrapnel. The stock prices of IPO high fliers VA Linux and Red Hat currently trade near half of their pre-IPO offering prices. Meanwhile, Windows XP gets the press and the plaudits.
But what's happening behind the scenes? In the early days of the open-source movement, Linux-based operating systems made their way into the business world through the back door, usually shepherded by an engineer who just wanted to get his or her job done in the most efficient way possible. That motivation hasn't disappeared, even if some of the companies that tried to capitalize on it are already distant memories. In fact, today, entire industries are making Linux-based operating systems central to their business.
Take, for example, the glamorous, and absolutely essential to modern entertainment, visual-effects industry.
Visual effects, known in industry parlance simply as VFX, are those bits of movie magic that make dragons fly and toys come alive. The companies that create those effects are end users of technology -- they don't create tools to sell for others to use. They make movies. They are technology consumers. One of the best-known members of this industry is Dreamworks Animation of Glendale, Calif.
Dreamworks' 2001 summer blockbuster "Shrek!" was rendered -- a technical term referring to the process of creating computer-generated animation -- using racks upon racks of PCs running Linux. In total more than 1,000 computers running Red Hat Linux were used in a single giant cluster, or "render farm."
"Dreamworks set an agenda two years ago," says Ed Leonard, head of Technology at Dreamworks Animation, "to migrate completely to Linux."
And Dreamworks is far from alone. Pixar and Industrial Light & Magic, two other giants in the special-effects world, are also either using Linux or investigating it. And many of the top developers of special-effects software are making sure that their products will work with Linux. It's a classic open-source success story: industry adoption not because some megacompany is pouring millions of dollars into marketing, but because the software gets the job done, cheaply and efficiently.
The Dreamworks production pipeline includes the Dreamworks-PDI studios that add effects to live-action feature films as well as Leonard's studio where animated films are produced. Each pipeline uses two distinct sets of computing systems: those used as back-end "render farms" to produce individual frames of the movie, and workstations used by artists and programmers to create models that are fed to the render farms.
"The desktop agenda was driven by Dreamworks Animation," says Leonard. "About a year and a half ago we deployed a pencil test animation system, where traditional animators would capture images from original artwork and replay them." Leonard says the project was interesting because Linux lacked some of the tools commonly available on their old favorite, SGI IRIX (famous for producing the special effects in "Jurassic Park"), such as multimedia code libraries.
The work Dreamworks needed to do wasn't exactly what Linus Torvalds had in mind when he invented his operating system "kernel" -- the core of an OS. Says Leonard, "We had to push the kernel in real-time ways that it wasn't quite ready for. We had to write software for driving cameras, image capture cards and audio cards." The studio then had the daunting task of making its existing in-house applications work with the new OS, a process that required managing several million lines of proprietary code.
"Dreamworks' work with Linux on 'Shrek!' has been held up as a poster child for the industry changes," says Visual Effects Society (VES) technology chair Ray Feeney, who also founded VFX studio and software maker Silicon Grail. But getting the industry over to Linux wasn't just a matter of having studios rewrite their own specialized (and studio specific) software. The industry had to work as a team to get application vendors to port a number of widely used applications. To do that, the studios needed to break away from the secretive environments that cloaked every production.
The event that marked the start of the changes for the industry was the VESTECH 2000 conference in Santa Barbara, Calif., where a special Linux summit was convened. The VES is the professional society for the industry. It includes numerous Academy Award winners for special effects and technology.
Representatives from 24 of the leading effects companies, including Dreamworks, Rhythm & Hues, Pixar, Industrial Light & Magic and Digital Domain, attended the Linux Summit. The membership determined that it was inevitable that the industry move away from SGI, with its high-cost hardware and questionable business future. As a group, they wanted to take advantage of the low-cost commodity hardware the Intel platform provided.
"Where we come from and where we're going are two different places," said Leonard. "Historically we purchased a large amount of SGIs. Those were amortized over several films. You'd want to get five years out of some of that hardware due to the expense of that investment. With the Intel-Linux strategy today, we're moving toward what we refer to as disposable computing. Now productions are generally two years long, and during that time the technology takes several steps ahead. Usually we anticipate to recoup a large portion of our hardware costs with every production. So with each new movie, we go out and purchase a new render farm. That lets us reset the benchmarks for each movie, letting us tell better stories."
But moving to Intel hardware meant getting third-party applications moved to new computers and finding a new operating system to run them on. Microsoft's NT was offered as one option, but Feeney says most members had already tried and rejected that path.
"There are lots of PC manufacturers which run [Microsoft's] software," said Feeney, dryly. "But there are certain advantages to using a Unix environment for the larger enterprise [large business] markets. Once upon a time there was a great focus on the part of Microsoft that the VFX industry would be the next realm they would conquer -- taking Windows from a consumer tool to the enterprise. They decided later that they would be better off spending their time elsewhere, like on the Web with Hailstorm and .Net. So they never bit on the enterprise market. They're off working on issues that don't solve the set of issues relative to the high-end effects industry."
Choosing Linux was a matter of resources. Studios already had a large amount of expertise on IRIX, a variant of Unix created by SGI. Moving to Linux would provide fewer headaches for existing staff. And it would require less work rewriting or "porting" already existing software. The porting issue extended to third-party application vendors like SideFX Software and Alias-Wavefront. Both companies were contacted by industry members regarding ports of their software to Linux. And both responded.
"The Linux agenda is a key one for us," says Chris Ford, senior Maya product manager for Alias-Wavefront. "The high-end film community uses this product. They need the robust, scalable environment that Linux offers and Alias-Wavefront knows we need to be there."
The company is making a heavy commitment, says Ford, but one driven by their customers.
Alias-Wavefront's Maya program is a de facto standard: a general-purpose graphics application that fits across the high-end film, episodic and location-based entertainment arenas. The appeal to the industry is the program's openness. Many VFX studios build their own tools, but they integrate them into the Maya architecture.
SideFX is actually credited with starting the move to Linux by the industry, having ported its Houdini modeler in April 1999, a full year before the VES 2000 Linux Summit. SideFX's CTO Paul Salvini says that his company ported the application even before there was hardware-accelerated video card support for it in Linux because someone needed to take that first step. "Part of the reason we considered doing the port was the interest shown by a number of effects houses for a Linux-based Houdini. The interest was 'provisional,' considering the lack of workstation-class hardware support that was available, but SideFX took these requests to heart and felt they could help move things along by providing the application side of the equation."
By combining their weight as an industry, the VES members were able to convince many other application vendors to port their software to Linux as well. Pixar's PRMan, Nothing Real's Shake, Silicon Grail's Rayz and Avid's XSI are just a few of the high-end, high-dollar applications already ported.
There have been some roadblocks on the way to fully adopting Linux in the special effects industry, however. Working with the open-source community hasn't always been a smooth ride.
Feeney says that the open-source community is very unforgiving with what he terms as "wrong behaviors." One of those behaviors has to do with the intersection of proprietary and open-source software.
"We believe very strongly that the infrastructure issues point directly to open source," says Feeney. "But there is also the fact that specialized intellectual property exists at the studios. So what we have to contend with is the involvement of the open-source community with respect to the shared proprietary concerns of the industry, without getting everyone upset that those proprietary pieces aren't given back to the community."
Fortunately, the problem may be one of volume over quality, as many of the fundamentalists in the open-source world also tend to be the most vocal. But a large portion of the open-source world expects that corporations have proprietary concerns and the two can find a middle ground. The solution will be in finding the right licenses for the industry to use.
The biggest issue -- as it is with many industries considering open source -- is what to do about the GNU General Public License (GPL), the prevalent license used on open-source software. Does the VFX industry know what the GPL is, what it means and how it relates to proprietary software? Feeney says probably not. "In our community, the open-source push is great when you need to reach consensus and standards." But, he adds, the industry relies on differentiation through specialized approaches. To them, the GPL doesn't appear to allow them to keep those differentiations. Or at least it doesn't make it clear how it can be done.
Still, Feeney says the VFX industry needs to approach the problem from the right point of political correctness. "It's important we get guidance from the open-source community about how best to approach them to get them interested in supporting the ongoing needs and requirements of the film industry."
Feeney says the industry wants to be marching in lock step with one another without having to build their own version of Linux. "We don't want to make a Hollywood branch of a [version of] Linux, for example ... To us that means adoption of single solutions which can be incorporated into a commercial package without the rules imposed by a specific license."
And Leonard says he'd still like to see the open-source community look toward entertainment as a partner in innovation, not just in recognition. "One of the hard parts of dealing with open-source is that it's still viewed as a bit of a hackers' world: As long as you're willing to hack at the code you'll get what you want." A reality for the VFX industry is that as a business they need to find a way to channel the talent in the open-source world so that they can get value from it.
"The VFX industry is willing to take risks," says Leonard. "Our solutions don't have to be wrapped with a pretty bow -- we're willing to work together to make these things work. I think the application for what we solve together with open source can be applied to a much larger community. Entertainment is a place where we can push the high end for innovative use of Linux. And it is a visible place where Linux can get momentum to use across broader industries."
License issues aside, the technical problems of getting Linux onto the desktop have not been simple. For early adopters such as SideFX and Dreamworks the solution was to get professional help: Hewlett-Packard's Ft. Collins graphics group.
At Dreamworks, Leonard laments that the thing that drove graphics card performance on Linux in the early days of the migration was the first-person shooter computer game Quake. Gamers who were fans of Linux and Quake hacked on Linux until Quake ran smoothly.
"If you were using something that Quake relied on, you were great," says Leonard. "If you were using something that Quake didn't care about you were probably in some serious problem." They faced problems like getting 24 frames per second 2-D playback with synchronized sound and making that work reliably and consistently. HP helped solve the problems.
Fortunately, open source tends to move moderately quickly for infrastructure pieces, such as video card drivers -- code that allows a video card to work with a particular configuration of hardware and software. But as Leonard notes, "This is sort of the best and bad news at the same time. The good news is that the technology changes really rapidly. That's also the bad news."
Today the studio faces the question of which technology to be on. "The sum of all this is that there are many pieces that must fit together to create a stable environment for two years," says Leonard. "That's a pretty challenging task when you're not going to a single source vendor like we did with SGI. The biggest challenge is creating stability in a very dynamic world. Choice is good, and choice is bad."
Leonard says that while the industry is willing to accept short-term solutions that are not open source because of the early stage of the migration, they want to see long-term strategies that do use open source. "We've said to vendors like HP, 'In order for us to partner, we really want to see you embrace Linux and open source.'" That gives the industry more flexibility in choosing hardware. The industry is driving open-source solutions from vendors.
The VFX industry has spent the last year doing some experimenting. "Not everybody has 25 people to dedicate toward a Linux initiative," notes Leonard. "The ILMs, the Dreamworks, the Pixars of the world have those kind of resources, and all are actively engaged in doing it. And as we do it, everybody benefits. As we figure out how to get drivers aligned with the hardware, they become available to other places that can't invest in the engineering to make it work."
In any case, Leonard is pleased with the results. "Today, I'm happy to say, all of these things have succeeded to the point where we feel confident to committing all of our pipelines to be 100 percent Linux for the desktop and the render farm."
Perhaps the most intriguing aspect of the VFX industry's adoption of Linux-based operating systems is that the cooperative environment resulting from the Linux Summits (a second event was held in 2001) has helped loosen the veil of secrecy that has plagued the industry. Leonard says creative content companies have always been very protective of their works. "The Dreamworks and ILMs of the world understand the separation between the technology as plumbing and the technology as discriminator. As we go forward, our companies are becoming better educated at an executive level about those things."
Pixar's VP of technology Darwyn Peachey says it's nice to see the openness that has evolved between the studios over the two years that they've been meeting to talk about Linux. "We're all competitors in one sense or another, but this has helped us share a little more than we historically have about our thoughts and plans. Not surprisingly, those problems are very common across the many studios. We're starting to see that if someone solves a problem that isn't central to the art we do, there is suddenly a feeling we can all benefit from it. That's encouraging. It's fun to see."
The old competitive nature that arose when two similar films were released at the same time is fading. "In fact," he adds, "it's hard to determine if one film hurts the other. If you go to a bunch of bad ones you won't want to go to any of them. So we all do a little better if the films are better."
"This is a fundamental shift in how we do business," says Leonard. "And there are two interesting parts to this. One is that we're moving towards commodity hardware in an open space. The other is the spirit of cooperation and community and infrastructure sharing that has never happened before. We were all very competitive companies that were not very interested in sharing at all. This Linux stuff is pulling us together."
Shares