April 2006 Archives
Please, please, tell us that when you write "recycled" you really mean someone took the antiques home or gave them to a collector or a museum. Even to those who are not the biggest fans of Microsoft this stuff is true history. Sadly, those close to it don't always understand it as such. An "original" Mac (whether it be a 128K Mac or a Mac XL) once owned by Microsoft would be very valuable indeed. The birth of Excel 1.0, not to mention its predecessor, Microsoft's first GUI spreadsheet known as Multiplan, quite a significant milestone in human history if only because of the number of lives eventually touched. Oh, please, please say those antiques didn't actually get recycled... That would be really bad Karma. Could have even auctioned them off for the Bill and Melinda Gates foundation. There must be a lot of readers out there just shuddering to think of the tragic waste.When I found out they were headed for the trash I felt exactly the same way. So here's what we did: 1. We gave as many as we could to Microsoft's Museum but they didn't need or want every single one we had so... 2. We found loving homes for a bunch of the Macs with Microsoft employees who love Macs, some inside MacBU, others not. 3. We somehow found more storage space for the rest :) Fun Fact: At Mac World 2004 we had a special Microsoft Mac timeline at the booth with a few of the old Macs, all of which came from our Mac Nursery and Sandbox areas. So Drew Merkle the Macs are okay. :)
Today I'll stop my 3rd person perspective. I'm going to write a little bit more about what I do to help MacBU ship great software and provide some color around what's it's like to work on Mac software at Microsoft. Often when we have press events or special visits from our MVPs I'll give them a tour of the Mac Lab and explain what we do. They've always found it very interesting and so I thought I'd share a virtual tour of our Mac Lab. Let's get started:
We start with the door. The Mac Lab is about 2000 square feet of solid computers. The calendars you see down the side of the wall we use to mark team birthdays as well as special events. The CD on the door is an old Apple Software Restore CD from the last beige Mac Apple produced, the Power Macintosh G3.
The first area in the Mac Lab is what we call the Sandbox. This is where we keep all significant hardware configurations Apple has released that run our products. We'll use the Plasma display to, watch DVDs and play games, uh er, I mean, do important training presentations. ;-) It's actually very useful because everyone can be in front of a computer and still see the main screen and follow along. Often other groups at Microsoft (the games group, hardware drivers group and even the Windows media group) will come and schedule time in the Mac Lab to test their software on the different hardware configurations.
We have lots of Apple hardware. You can see here the old colorful iMacs along with some of the old iBooks. You can also see two of our Lab Technicians working on the backup systems, but more about that later. Up until a few months ago we had every significant hardware revision Apple has ever released since the dawn of time. We even had a section of the Lab we affectionately called the "Mac Nursery" where we kept all the older Macs going. We even had an old SE/30 and IIci and super expensive Mac II all connected via PhoneNet running Spectre, just for fun. It's always super fun to boot Word 1.0 or Excel 1.0 on these old machines and see how much things have changed. Due to lack of space in the Lab we had to put all of these older machines into storage and recycled the following:
- Macintosh (original)
- Macintosh SE
- Macintosh SE/30
- Macintosh Classic
- Macintosh Centris 610 <-- I had this one in my room during high school
- Macintosh IIci
- Macintosh IIsi
- Power Macintosh 7100/66
- Power Macintosh 7100/80
- Power Macintosh 7500/100
- Quadra 650
- Power Macintosh G3
- Duo Dock with Powerbook Duo 2300c
- Power Computing PowerCenter Pro 210
May they rest in peace.
One of the realities of working with computers is that things fail. More often then you might think. We've used different backup robots, but ever since we moved to ADIC we've never had a robot failure. They just make great stuff. We have 3 robots and use Veritas Backup Exec. It works pretty well, as you can see:
We also do offsite backups just incase "The Big One" hits, but for regular use these tapes work just great, except when they don't. Recently we had a failure and lost 400 GB of data! We restored it from the tapes and then discovered that the Mac version of the Backup Exec agent had a corruption bug causing the restores to be compromised! Veritas folk were super responsive and they should have a new Mac agent out soon. Backup software and file systems are in the class of software that simply must work, all the time. Alas, this is not always the case.
I'm going to skip the "Build Lab" section of our Lab since it's very much in transition. Maybe I'll post about that later. For now, on to our automation system!
Mac Office is one of those "software in the large" projects. There's really no way a team of our size would be able to adequately test all of Office without the use of automated testing. Every day we get a new build of Office from the build machines, we copy it to our Xserve RAID connected to our dual G5 Xserve for access by our 249 automation machines. We then run thousands and thousands of tests on the new build. Typically we get 4 builds of Office each day: English Ship, English Debug, Japanese Ship and Japanese Debug. We run our entire battery of tests against all the builds and then report any failures to testers via email. The testers investigate the failures, log any bugs and then move on to their other duties as testers. This turns out to be very effective, if used properly, and over time it allows testers to focus on things humans do best, while letting computers verify the repetitious and mundane, but necessary, testing. It all started with our Blue and White G3s years ago. At first when testers would upgrade their test machines, instead of recycling the machines, "The Lab" would get them to add them to our automation machine pool. I think we had about 20 machines to begin with.
After some time we started getting Gray G4s...
Then we upgraded to some dual proc machines...
Then Apple give us a special gift. :-) You'd be probably be very surprised at the cost of running all these machines. There's the obvious electricity costs, but also cooling costs and even the physical space costs. Additionally, our system scales, not with CPU horsepower, but with quantity of machines. Most of the tests we run don't run significantly faster on a dual G5 vs. a single G4. So when Apple announced the Mac mini it wasn't minutes before we were considering how to use it for our automation system. The Mac mini has all the perfect qualifications:
- Low power
- Low heat
- Easy to pack together
So we got a few to test things out...
And then we made the big purchase:
These work extraordinarily well. You might wonder how we control all these Macs. We use two methods: KVM switch box and Apple Remote Desktop. Thanks to our Lab Manager's great relationship with the IOGear folk we have a very reliable solution these days. It seemed like it took for ever to find a USB KVM switch box that didn't leave the machines "headless" after random reboots. The 8 port USB KVM from IOGear has been rock solid. So what does it look like to sit in front of 64 Mac minis? Like this:
This works very well when you must access the machines physically. Even so, just scanning each Mac for 1 second gets very old, very fast and Apple Remote Desktop comes to the rescue! When we need to see all the machines at once we just select them and BOOM! they're there. It also gives us what I believe is the one true reason Apple invented the 30 inch Display. ;-)
ARD displays 50 machines at a time and when you have a capable machine, it uses the "cube rotation" effect to move from one group of 50 to the next. I got a picture of the effect mid rotation below:
So how does it all work? Like this: On each machine we have two volumes: ChangeOS and Mac OS X. The Mac OS X volume is where we install the different versions of the OS. We boot to the ChangeOS volume to free up the Mac OS X volume for modification. When we trigger an automation run we specify the OS version and language. Each machine then reboots to the ChangeOS partition, caches the OS .dmg locally and uses the asr command line tool to restore the image. The tool that does this work is one I wrote (in AppleScript Studio no less!) called Lab Assistant. We have images of the Mac OS from 8.1 all the way up to 10.4.6 in all the languages our products support. It's a lot of data which brings me to the backbone of our automation system:
Right now we've just been testing out the XSAN stuff to see how we want to use it. That's why you see all the Xserves. Just one side of the top Xserve RAID is 1 TB of data. For a fun comparison this whole rack which is about 7 ft tall is full of old RAID arrays is also 1 TB of data storage. We call it the Big Mac Daddy.
Other groups at Microsoft have hardware retention policies that force hardware upgrades every so often, but instead of just "recycling" these server machines, our Lab Manager intercepts them on the way out, and we use them for various things, storage, SQL server etc. We actually have some of the old MSN servers in our Mac Lab!
When you have so many machines to maintain, being able to get behind the machines is very important.
We like to pack in those Mac minis and the cords get pretty dense when we do. The hanging Mac mini box moves if the HVAC is working. If it's not working, we've got to turn off the machines until it's fixed.
Our main automation Xserve has a habit of failing in some serious way once a year, always around Christmas time. :/ For the last 2 years I've been in charge of fixing it and getting it back to operational. Most of our server racks are generic white enclosures, but we do have 1 black Dell rack. As punishment for bad behavior, we put the Xserve in the Dell rack. That'll teach it. ;-) This is what it looks like from the inside of the rack looking out on the world. Poor caged servers...
Of course our iWork/iLife balance is just fine as you can see by the following:
One of our team members bought this awesome Tornado foosball table which we use along with and XBox and XBox 360 to relax after a hard days work.
A while back the Seattle PI actually did a front page story on the Mac Business Unit and you can see from the picture in the Lab it was when we had only the G3s.
As you enter the kitchen we have our MacBU mission statement to remind us what it's all about. :)
Just like everywhere at Microsoft, we get all-you-can-drink beverages.
Part of our team mantra is that we work hard, and play hard. So we do lots of fun morale events. We just take time off work and do stuff. We're good friends and enjoy "just hanging out" together.
This is just a pretty picture to represent what is really a much bigger collection of 3rd Party software we use to test with Office. Most if it is stored on file servers, but this gives you an idea. (There's some old WWDC DVDs if you can find them!)
A big part of Office functionality is printing, and we do loads of print testing. We work really closely with the printer vendors and make sure the printed page looks great. WYSIWYG is fundamental to Mac ethos. All these printers are connected via USB hubs and Ethernet to a Mac OS X Server 10.4 which is the printer server.
I hope that gives you a better idea about what the Mac Lab is like and what it's like to work in the Macintosh Business Unit at Microsoft.
This technote describes a number of 'secret' debugging facilities in Mac OS X, including environment variables, preferences, routines callable from GDB, special files, and so on. If you're developing for Mac OS X, you should look through this list to see if you're missing out on something that will make your life easier.Thanks to Jesse for sending me this gem.
Designing a product requires many skills, and it is the rare individual who has them all. Design is, therefore, an exercise in teamwork, where each team member brings in a different mix of skills, attitudes, and values. Alas, quite often, members think their own set of attributes is the most important. Thinking that one's own discipline is the most important of all gets in the way of teamwork. It creates bigotry, which in our field of usability and interaction design means usability or interaction bigots. I have long worried that in preaching the merits of usability and interaction design, we have lost sight of the goal. For customers, it is to provide pleasure and accomplishment, letting them fulfill their needs both effortlessly and pleasantly. For purchasers, the product must be affordable and return value commensurate with the cost. For the company, the product must be profitable: people must buy it in sufficient numbers without requiring expensive service calls. And there must be a continuing revenue stream so that employees can remain paid and future profitability assured. Whose profession is this? Nobody's, at least in the sense that it belongs to any single group. Everybody's, at least in the sense that we all have to come together to ensure that products are successful from everyone's point of view. This means that there may be usability flaws, technology glitches, appearance issues, and marketing fumbles. But so what? We must look at the larger picture and ask, does each flaw really matter? And so everyone must pitch in, everyone must make compromises and tradeoffs, and everyone must put their profession second and the interests of the company and customer first. In October 2004, Rashmi Sinha and Richard Anderson organized a symposium on this topic ("User Experience: Why do so many organizations believe they own it?") for the San Francisco Bay Area Designing for User Experience Society (BayDUX). During the question period, an audience member asked me "Who should be in charge of the product," clearly expecting me to say "the user experience team" (because it's all about UE, isn't it?). My answer was disappointing to her and many others in the audience. I said that everyone was in charge - marketing, sales, manufacturing, engineering, industrial design, communication design, graphics design, and, yes, user experience design. Mainly, the person in charge is the product manager. If the product manager can't resolve the issues, then it becomes a business decision. Business decisions are for managers and executives to decide. (Disclaimer: I was once one of those executives.) When there are conflicts, we need to step back and ask what the total impact is upon the customer and the company: How serious are the problems? Does the product give pleasure. Will people recommend it to others? If, for example, form gets in the way of function, does the enhanced appearance help sales more than the impact on function hurts? The decision needs to be based upon what is best for the company and the customer: hopefully, these two coincide. So, the person in charge is the product manager first, managers and executives second. "Product manager?" was the puzzled response? "But, then," came the follow-up question, "what discipline should the product manager come from?" My answer: "Who cares?" I have seen excellent product managers who were trained in English history, or engineering, or psychology, or Chinese calligraphy. What matters is that this person can pull the team together, can balance the talents of the teams and the needs and requirements of the customers, the company, the sales and marketing teams, the manufacturing and packaging teams, and all those big, powerful egos (as well as the brilliant, but quiet and insecure). Talented product managers make this all work in harmony. I am always awed by their skills. They are - and must be -- generalists, people who can see the big picture. What discipline should they represent? None of them; All of them. Whose profession is this, anyway? Nobody's and everybody's. We are all in it together, we all need one another.As I've observed it, this PM position is so often the unsung hero of a product that delights the customer. Our users really don't know all the constraints that go into making a piece of software, but I guess that's the beauty of it. They don't need to know the complexities. Good design is the essence of building Mac software and I love it.
Q: What do rocket scientists say when they want to describe a portion of their work as easy? A: "This bit isn't exactly brain surgery." I think that pretty much everyone would agree that rocket science and brain surgery are both intellectually demanding pursuits. But it seems to me that there's a fundamental qualitative difference between them. Rockets are devices constructed by humans for specific purposes. Though there may be considerable systemic interactions between all the parts of a rocket which must work harmoniously together, those interactions are the result of careful design. The rocket as a whole can be decomposed into its original parts, and those parts can be independently tested to see if they meet their design criteria. Furthermore, the aim of rocket science is usually to deliver a specific payload to a specific place at a specific time. Though it is no mean feat to get to the moon, we can least for all practical purposes calculate the future position of the moon at any time you care to name to any degree of precision. And yes, it's hard, and yes, mistakes are made, sometimes with tragic consequences. But fundamentally, rocket science is about shaping raw matter into precise forms to achieve precise tasks. Brain surgery isn't nearly so much like that. We're presented with a lump of thinking, dreaming meat that we barely understand how it works and is presently being used to solve all manner of problems that evolution did not design it for. Our understanding of brains is strongest at the very low level - the neurochemical level - and at the very high level - the gross divisions of the brain into the centers that control particular muscles, responses, etc. But we have practically no understanding whatsoever of any level between those - we are no where close to understanding what algorithms the brain uses to recognize faces or write music. And a working, living brain has trillions of interacting parts which were not designed with orthogonality or functional decomposition in mind. I've been thinking about the difference between brain surgery and rocket science lately in the context of my coworker Peter Hallam's essay on the difference between writing new code and modifying old code. Naively, one would think that adding new features to code would be like rocket science. You understand all the parts, you figure out how to redesign the parts to admit the new desired behaviour, you implement it, test it, and you're done. It's an intellectual challenge, but it's fundamentally amenable to analysis because every part has a clear, well-designed function that can be tested independently. But as Peter points out, anyone who has actually tried to add major new functionality to existing code knows that most of the time its more like brain surgery. We know what the code does on a gross level. (Say, tokenize source, parse, create symbol tables, bind type annotations, check reachability, generate code.) We know what any part of it does on the microscopic level. (Say, move the contents of this register to that memory location.) The hard part is understanding what the algorithms are that make up the large-scale behaviour, and understanding how to tweak them to admit the new desired behaviour without breaking anything. (How does tweaking early nullable realization during binding affect rewriting lambdas into expression trees? Who knows? Not me, that's for sure!) Understanding all that stuff is the incredibly hard part; actually putting the code under the knife is comparatively trivial. A big part of good implementation of software-in-the-large is making sure that the program is more like a rocket than a brain. Coming up with tools to enable that admirable goal is the hard part.I'm a bit of a 3rd party observer of all the work our developers are doing to get Office through this transition from my perspective I am very impressed. There's a lot of brain surgery and rocket science going on. It's amazing to watch.