Phoenix has the rather unenviable position of maintaining probably the largest body of assembly language source code in the world. At my last count, there were over 4K assembly-language source code files in an average deliverable and ~240K files in our TrustedCore code base.
On top of that, while TrustedCore is one product with hundreds of variants. The variants represent the different combination of features and silicon support that go into supporting a single platform. Phoenix itself maintains over 500 different variants on the basic TrustedCore (BIOS) product and our customers maintain thousands more. The average life-span of a single variant is about 24 months, from start of a silicon-support product, through the delivery of a typical mobile platform. Embedded and server platforms have a longer lifespan and desktop platforms have a shorter platform. In a way, TrustedCore is never shipped the same way twice.
Compound this problem by having a few hundred developers simultaneously working on the same product at the same time, from 14 different sites around the world, some of which have less total Internet bandwidth than the broadband connection that I use from my house.
In this series, I'll talk a little bit about Phoenix maintains this massive source code database in a fairly efficient manner, using a combination of standard and home-grown source code control tools.
Phoenix's source code is organized into directories and sub-directories. Some of these directories are for silicon (i.e. CHIPSET\INTEL\SOLANO). Some directories are for features (ACPI\STD32.600). Some directories are for individual platforms (BIGREF\TC\865\GBF.TC). In order to build TrustedCore, you go to the platform directory and type 'nmake' and that starts the build process for the entire BIOS. While each variant doesn't require every directory, many directories are used by most, if not all, products.
Each of the 14 different sites (India, Europe, Taiwan, Oregon, etc.) has a server with a complete copy of all of the source code and all of the source code archives. But each of the directories which are part of TrustedCore is "owned" by only one site.
For example: CHIPSET\INTEL\SOLANO is owned by our Beaverton office, ACPI\STD32.600 is owned by our Taiwan offce and BIGREF\TC\865\GBF.TC is owned by our west coast U.S. office. We use an internal utility WHOOWNS to return the owner of a directory.
K:\nb\source\NUBIOS\BIGREF\TC\865\GBF.TC>whoowns
Santa Clara
The site (or server) which owns a directory is the "master" or "owning" site. So, if you want to check in a change to files in that directory, you must change the archive on the owning site.
Normally, when I log into Phoenix's internal network, the default is to use the Milpitas (or Santa Clara) server. That means I will have fastest access to archives which are owned by Milpitas. If I want to check in files to a directory owned by another server, I have to change servers. We do this using an internal utility MAPDRIVE
C:\pv\dos>mapdrive twn
Site TWN assigned to Drive K:
Now, checking in to a remote server, like Taiwan, is slow for me, since I'm based in the U.S. I might do it for a file or two, but if I need to check in a lot of files or need to do a lot of work over a period of time, I might request that the ownership of the directory be changed from Taiwan to Milpitas. Once the ownership is changed, my access is much faster because I'm talking to a U.S. based server.
This is important, because it allows Phoenix to give very fast access to our massive source code base to any engineer around the world on a directory-by-directory basis.
So what about other engineers? What do they do? Well, lurking in the background is PSYNC, a Phoenix utility which monitors each directory on the owning server for changes in the archives. When it detects a change, it copies the source code archive and the most recent revision of the source file to every other server.
Most engineers just retrieve source code using the copy of the archive on the local server. Doesn't this lead to problems of potentially being out of date? Yes. But it usually doesn't lead to problems for three reasons:
- The synchronization process is fast, since the process is continuous and the number of archives changed per day is fairly low.
- Most engineers don't need access to the bleeding edge of development in every directory. They usually retrieve from a version label rather than from the tip revisions.
- Many of our sites are in radically different time zones. Many times, the Taiwan, China and India sites are starting just as the U.S. sites are finishing.
This distributed ownership of archives also gives one other benefit: we have a built in backup process. When a server dies, there are usually 13 other copies of every archive! And when we need to bring a new server on line, the synchronization process can automatically bring it up to speed by copying all the files from the other server.
It works remarkably well, has managed the transfer between different networks and runs using a fairly standard version control system.
Next time, in part 2, we'll talk about versioning and labels.
Tim
Comments