The other day I noticed one of my friends login to his laptop. I do this all the time, so that he logged in wasn't unusual, but how he did it, that caught my attention. He slid his finger on a small sensing area and after some delay he was logged in! No username and no password! This piqued my curiosity, put me on a research track. Here is what I discovered.

What are Biometrics?

When you login to your computer, security experts would say you authenticate. That is, you prove to the computer that you are indeed who you say you are. There are three ways you can normally prove who you are [2]:

1. What you know
2. What you have
3. Who you are

When you login with a password, you are using the first form, what you know. If the password is kept secret and sufficiently hard to guess, when you enter your password the computer assumes it must really be you, because no one else would know the secret password. By this logic you are authenticated. There are other ways you can prove that you are who you say you are. When you purchase something with your credit card, or unlock your car door, you are normally authenticating with something you have. When someone checks your passport picture against what you look like, this is using two forms of proof: What you are, your face and what you have, namely the passport.

So what are Biometrics? From the word you might be able to deduce some of the meaning. The "bio" part means of or pertaining to biology. The "metrics" part relates to measurement. Put together biometrics it is the measurement of biological data. For authentication purposes a biometric is a physical characteristic that can be measured and then used to identify a specific individual.

When you use a "what you are" proof, it helps that you choose an attribute that is unique. If I used my height to prove that I am who I say I am, most of you would laugh. Why? Because there are lots of other people 6 feet 4 inches tall. For a strong identification, uniqueness matters.

How do fingerprint biometrics work?

Each fingerprint contains ridges in the skin which are detectable. These ridges make patterns which you can see with the naked eye. The ridge patterns can make loops, arches or whorls. In each fingerprint there are areas where one ridge changes, these points of change are called minutia. When a ridge ends, splits into two ridge, joins another ridge or creates a island; all of these features in a fingerprint are minutia. [10]

A fingerprint scanner detects the larger ridge patterns and records the following information about the minutia:

1. Orientation - the direction the minutia is facing
2. Spacial Frequency - how far apart specific features are located
3. Curvature - the rate of orientation change
4. Position - the (x, y) location relative to some fixed point

All of these can yield up to ideally 60 or 70 minutia for a single fingerprint. [10]

When a person first configures a scanner to use their fingerprint, this is called enrollment. This is a key time where strict process must be followed so as to avoid someone enrolling another's fingerprints as his or her own. In this process several fingerprints are recorded by the system and then an averaged template of the fingerprint is stored in the system. [10]

After enrollment, when a new fingerprint is supplied to the system, a statistical match is sought. If no match is discovered, the person is denied access. If a match is found, the person is granted access. Since biometric measurement can change slightly with each scan, determining a match is a probabilistic process. There is always a possibility that a valid fingerprint is rejected or an invalid fingerprint is determined to be good. [10]

What are the advantages and disadvantages?

To compare the relative merits of fingerprint as a biometric, we can considering the following properties of a good biometric [11, 6, 3]:

1. Universality - each person has the characteristic
2. Uniqueness - the characteristic is unique per person
3. Permanence - characteristic remains the same over time
4. Collectability - how easy is it to measure the characteristic
5. Performance - accuracy, speed, and resource requirements
6. Acceptability - culturally accepted by the population
7. Circumvention - robust against fraudulent attacks

Universality

Fingerprints are largely universal. About 2% of the population can not use fingerprints due to skin damage or genetic factors.

Uniqueness

An important aspect of any biometric is that the characteristic be unique among all participants. Empirical evidence of fingerprints gathered since at least 1892, have found no two pairs of fingerprints that are identical, even between twins. [10] While this is reassuring, more recent research [13] indicates that the minutiae-based uniqueness as measured by fingerprint scanners may not be sufficient to determine individuality in every case.

Permanence

Fingerprints can be damaged and change due to manual labor or injuries.

Collectability

A key aspect of any biometric is the reliability of the measure coupled to the convenience of making the measurement. [10] Fingerprints are easy to collect and the systems today very inexpensive, around $20 per scanner if purchased in bulk.

Performance

Fingerprint recognition systems for large-scale deployments require a large amount of computational resources. For smaller populations the computational resources are smaller making this trade off allows for the current use in personal computers. Still compared with face recognition, for example, fingerprint scanners are more accurate, faster and require less computation and storage. Face recognition may seem very intuitive for humans, but the current mature computer systems are still far away from reliable face recognition in practice. [9]

Acceptability

Fingerprints are perhaps the most accepted biometric short of facial recognition. Still, the acceptance of biometric authentication technology depends on context, perceived benefit for the user and perceived privacy risks. [1]

Circumvention

Fingerprints are not impervious to attack, but using multiple finger scans can make the system harder to attack. ( see Myth Busters http://www.youtube.com/watch?v=MAfAVGES-Yc ) One method for improving the accuracy of biometric authentication systems is to enforce "two-factor authentication", that is one must authenticate by some biometric form as well as use a password or posses a token. [10] This additional factor can significantly harden the system.

Problematic Properties

Among the currently available biometrics, fingerprints have a large collection of positive characteristics. Some of the advantages of a biometric are that you don't have to remember passwords, don't have to worry about forgetting a password or smart card and it cannot be easily changed. [4] Because of these advantages systems have been developed for so called "One Touch Logon" [12] but in these systems one still makes significant trade offs between a password-based system and a fingerprint biometric based system. There are large trade offs between security and privacy [11] when using fingerprint biometrics.

Fingerprints have other problematic properties. They can be changed or damaged and some simply don't have fingerprints. Often because of these limitations fingerprint authentication systems have an alternate password based "back door" which can become the weakest link in the authentication system.

I think the biggest problem is that fingerprints are not very private, we leave fingerprints everywhere! Storing large amounts of fingerprint data in a safe way becomes a real challenge. If that data were exposed, criminals might devise a way to create fingerprints from the template data that fool fingerprint scanners. Biometrics in general are or can be unique identifiers, but they are not secrets. [7] Once a fingerprint is stolen, it's stolen for life! You can never get back to a secure situation! With a password or token system you can re-issue the token and invalidate the old token, or change the password. Not so with fingerprints! They are useful, but they are not keys. Keys need to be secret, random and easily updatable. Fingerprints are none of these things.

If fingerprint were to become the common authentication across different applications used for everything from logging onto your PC to accessing your bank account to opening your garage door, then value of stealing this information goes up significantly. At least with passwords you can spread the risk across different passwords for different services creating multiple barriers that must be overcome for full disclosure. With our fingerprints we have only ten of them.

References

[1] Heckle, R. R., Patrick, A. S., and Ozok, A. 2007. Perception and acceptance of fingerprint biometric technology. In Proceedings of the 3rd Symposium on Usable Privacy and Security (Pittsburgh, Pennsylvania, July 18 - 20, 2007). SOUPS '07, vol. 229. ACM, New York, NY, 153-154. DOI=http://doi.acm.org/10.1145/1280680.1280704 

[2] Guinier, D. 1990. Identification by biometrics. SIGSAC Rev. 8, 2 (May. 1990), 1-11. DOI=http://doi.acm.org/10.1145/101126.101127 

[3] Jain, A. K. and Ross, A. 2004. Multibiometric systems. Commun. ACM 47, 1 (Jan. 2004), 34-40. DOI=http://doi.acm.org/10.1145/962081.962102 

[4] Boatwright, M. and Luo, X. 2007. What do we know about biometrics authentication?. In Proceedings of the 4th Annual Conference on information Security Curriculum Development (Kennesaw, Georgia, September 28 - 28, 2007). InfoSecCD '07. ACM, New York, NY, 1-5. DOI=http://doi.acm.org/10.1145/1409908.1409942 

[5] Markowitz, J. A. 2000. Voice biometrics. Commun. ACM 43, 9 (Sep. 2000), 66-73. DOI=http://doi.acm.org/10.1145/348941.348995 

[6] Jain, A., Hong, L., and Pankanti, S. 2000. Biometric identification. Commun. ACM 43, 2 (Feb. 2000), 90-98. DOI=http://doi.acm.org/10.1145/328236.328110 

[7] Schneier, B. 1999. Inside risks: the uses and abuses of biometrics. Commun. ACM 42, 8 (Aug. 1999), 136. DOI=http://doi.acm.org/10.1145/310930.310988 

[8] Bergadano, F., Gunetti, D., and Picardi, C. 2002. User authentication through keystroke dynamics. ACM Trans. Inf. Syst. Secur. 5, 4 (Nov. 2002), 367-397. DOI=http://doi.acm.org/10.1145/581271.581272

[9] Zhao, W., Chellappa, R., Phillips, P. J., and Rosenfeld, A. 2003. Face recognition: A literature survey. ACM Comput. Surv. 35, 4 (Dec. 2003), 399-458. DOI=http://doi.acm.org/10.1145/954339.954342 

[10] Alfred C. Weaver, "Biometric Authentication," Computer, vol. 39, no. 2, pp. 96-97, Feb. 2006, doi:10.1109/MC.2006.47 http://doi.ieeecomputersociety.org/10.1109/MC.2006.47 

[11] Salil Prabhakar, Sharath Pankanti, Anil K. Jain, "Biometric Recognition: Security and Privacy Concerns," IEEE Security and Privacy, vol. 1, no. 2, pp. 33-42, Mar. 2003, doi:10.1109/MSECP.2003.1193209 http://doi.ieeecomputersociety.org/10.1109/MSECP.2003.1193209 

[12] Beomsoo Park, Sungjin Hong, Jaewook Oh, Heejo Lee, Yoojae Won, "One Touch Logon: Replacing Multiple Passwords with Single Fingerprint Recognition," cit,pp.163, Sixth IEEE International Conference on Computer and Information Technology (CIT'06), 2006 http://doi.ieeecomputersociety.org/10.1109/CIT.2006.128 

[13] Pankanti Sharath, Prabhakar Salil, Jain Anil K., "On the Individuality of Fingerprints (2002)," IEEE Transactions on Pattern Analysis and Machine Intelligence, http://biometrics.cse.msu.edu/cvpr2001_indiv.ps

Since the beginning of the Internet people have shared URLs with each other. Often the vehicle for sharing URLs was an email or a personal web page with favorite links. Today there are bookmarking websites where users share URLs, comment on them and rate them. [1] As the popularity of Twitter increases, more and more people are using Twitter to share URLs with each other. [2] The 140 character limit on communication with Twitter combined with the reality that many URLs are easily longer than 140 characters has given rise to more and more URL shortening services. While these services do a simple mapping, exchanging one short URL for a longer one, there are risks involved with trusting a third party to redirect you to a web page.

The basic idea for a URL shortening service is to exchange one URL that is short to another that is long. Typically the long URL is the desired destination. A person might send the short URL to a friend. When the short URL is clicked, the website looks up the longer URL and redirects the user to the longer URL. For example, suppose I just got an Amazon Kindle 2 and I wanted to share with my friends more information about it. Amazon typically has very long URLs. The URL for the Amazon Kindle 2 is as follows:

http://www.amazon.com/Kindle-Amazons-Wireless-Reading-Generation/dp/B00154JDAI/ref=amb_link_83624371_1?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=center-1&pf_rd_r=0YPC2AH8155PQV3FWRPN&pf_rd_t=101&pf_rd_p=469942651&pf_rd_i=507846

That's 215 characters! I'll use this URL as the original URL with the following services to give you an ideas how they work:

bit.ly - http://bit.ly/

http://bit.ly/Z6eYE  19 characters

budURL - http://budurl.com/

http://budurl.com/bsfs  22 characters

eweri - http://eweri.com/

http://eweri.com/8rC  20 characters

hex.io - http://hex.io/

http://hex.io/ajz  17 characters

idek.net - http://idek.net/

http://idek.net/3kH  19 characters

is.gd - http://is.gd/

http://is.gd/lg7L  17 characters

lin.cr - http://lin.cr/

http://lin.cr/fvc  17 characters

POPrl - http://poprl.com/

http://poprl.com/Lm3  20 characters

snipurl - http://snipurl.com/

http://snipurl.com/cucc2  24 characters

tinyurl - http://tinyurl.com/

http://tinyurl.com/bngrky  25 characters

twurl - http://tweetburner.com/

http://twurl.nl/no316s  22 characters

urlBorg - http://urlborg.com/a/

http://ub0.cc/60/3G  19 characters

zi.ma - http://zi.ma/

http://zi.ma/65226b  19 characters

As you can see the original URL was 215 characters long, while the longest of the shortened URLs was only 25 characters long. I could post this shortened URL on Twitter and still have an expansive 115 characters left to comment on this URL. Perfect.

There are over 90 URL shortening services available online. A more complete list of URL shortening services is located at http://mashable.com/2008/01/08/url-shortening-services/.

Trusted or Untrusted

The most obvious risk associated with URL shortening is that it's difficult to know where the URL will take you, until you click it. The true destination of the URL is opaque. Often when I receive a dubious link via email, I hover my mouse over the URL, or view the HTML source to discover the real URL destination address and evaluate if I trust it enough to click. With a shortened URL, it's hard to know where it will take me, until I click it. Email Phishing scams are using URL shortening service for this very reason. [7]

Another problem with URL shortening is how it interacts with filters. A spam filter could use the URL in the past as one more hint that the email could be nefarious, but with a URL shortening service as the broker of URLs, the filter can't make any judgment about the URL. Many URL shortening services take spam complaints and will disable URLs if they are discovered to point to spam websites. [3] Some also proactively search their URLs for blacklisted websites and remove or disable these shortened URLs. [4]

Not just spam filters can be bypassed. Both Firefox and Google Chrome web browsers use Google Safe Browsing [5] a feature with warns users of malware or phishing sites. In the past using a shortened URL, instead of getting a warning message, users are sent directly to the dangerous web page. [6]

Less serious, but still problematic is using URL shortening services to hide the motive for an online review or recommendation. A seemingly objective review is tainted when readers discover that the author gets a monetary kick back for sending people to the reviewed product's site. Since shortened URLs hide the real URL they can be used to hide affiliate URLs and surreptitiously link to online stores. Most affiliate URLs are easy to spot, but when wrapped in a shortened URL, detection is more difficult. [8]

Another more remote, but still plausible problem with URL shortening is that should a URL shortening service become compromised, hacking one site would allow for redirecting popular shortened URLs to phishing or malware sites.

Getting More Transparent

Many URL shortening services have added some level of "see before you click" functionality. For example, any tinyurl can be prepended with the text "preview" in the URL and it will not redirect, but show the destination URL for inspection at tinyurl.com. Take the tinyurl above

http://tinyurl.com/bngrky

and modify it as follows:

http://preview.tinyurl.com/bngrky

While this adds characters to the URL, it allows the user to evaluate the URL before redirecting to the site. BudURL has an even more compact preview function. Just adding a '?' to the end of the URL will turn it into a preview URL.

http://budurl.com/bsfs  will auto redirect to the original URL

http://budurl.com/bsfs?  will preview the link first

Some of the services provide a little popup window that displays a picture of the webpage when you hover over the URL link.

Conclusion

A hacker or spammer is empowered by using a "benign" URL shortening service that everyone uses and everyone trusts. Once the click is made, a homographic attack may follow and will make it very difficult for a normal user to detect that they are being redirected to a phishing site. The real danger is that people have become habituated to trusting unknown links from their friends. This is dangerous because if their friend's account is compromised, it might not be their friend sending a link and the shortened URL will be clicked without concern.

An example of this propensity to click occurred 12 Feb 2009. One of my friends tweeted, "Don't Click: (link)". I was curious, but I didn't click the link. Next another posted the same thing, than another! It seemed fishy to me, and I later found out that the link presented a web page with another button that said, "Don't Click!" Naturally curious people, and trusting in their friend's recommendation, clicked the button and all of the sudden they noticed that they had in fact tweeted the same link though they never consented to doing so! It was the first socially engineered twitter virus. [9] While this virus was started as a joke, it spread extremely fast. [10] Luckily this social virus was harmless, but it reinforces how effective a socially engineered virus can be.

There are always trade off decisions to be made. In this case, the trade off is between the convenience of a short URL and the need for disclosure of a URL's destination.

References

[1] Tony Hammond, Timo Hannay, Ben Lund and Joanna Scott. - Social Bookmarking Tools (I): A General Review In: D-Lib Magazine 11, Nr. 4, 2005 http://www.dlib.org/dlib/april05/hammond/04hammond.html

[2] State of the Twittersphere - Q4 2008 Report - http://blog.hubspot.com/blog/tabid/6307/bid/4439/State-of-the-Twittersphere-Q4-2008-Report.aspx

[3] is.gd - Technical Information - http://is.gd/tech.php

[4] SURBL http://www.surbl.org/

[5] Google Safe Browsing for Firefox BETA http://www.google.com/tools/firefox/safebrowsing/  

[6] Finjan's Malicious Code Research Center, Evasive URL techniques, 25 Jan 2009. http://www.finjan.com/MCRCblog.aspx?EntryId=2153

[7] McGrath, D. Kevin, Gupta, Minaxi. Behind Phishing: An Examination of Phisher Modi Operandi. https://www.usenix.org/events/leet08/tech/full_papers/mcgrath/mcgrath_html/mcgrath_gupta.html

[8] Parker, Ryan J. Shortening (Affiliate) Links For Prettier Linking. 20 Feb 2007. http://www.ryanjparker.net/shortening-affiliate-links-for-prettier-linking/

[9] Korben. Petit cours de Twitt Jacking :-). 30 Jan 2009. http://www.korben.info/petit-cours-de-twitt-jacking.html

[10] Johnson, Clay. What is this Don't Click business? 12 Feb 2009. http://sunlightlabs.com/blog/2009/02/12/what-dont-click-business/

"Performance is the perceived measure of how fast or efficient your software is and it is critical to the success of all software. If your software seems slow, users may be less inclined to buy it. Even software that uses the most optimal algorithms may seem slow if it spends more time processing data than responding to the user. ... Remember that the perception of performance is informed by two things: The speed with which an application processes data and performs operations and the speed with which the application responds to the user." [1]

Instantaneous (0.1 to 0.2 seconds)
Immediate (0.5 to 1.0 seconds)
Continuous (2 to 5 seconds)
Captive (7 to 10 seconds)

If you address another human being, you expect some communicative response within x seconds-perhaps two to four seconds. ... In conversation of any kind between humans, silences of more than four seconds become embarrassing because they imply a breaking of the thread of communication. [2]

Especially with a code base that is mature, assumptions correctly made years ago can be terribly difficult to deal with later when the current and new assumptions reign. Writing code that lasts even 5 years and "is robust" is what everyone wants to do for sure, but the recipe for doing just that is not easy to learn and even harder to actually do. For example, you might know what changes need to be made to conform to even the most obvious object oriented design principles, but the business requirements and time to market needs dictate leaving once again old crusty code alone and racking up yet another round of technical debt. Over time, this technical debt will demand payment and the effects on your ability to hire, employee morale, design changes possible, speed of delivery, testing burden, marketing message etc. become very real and very painful. I wonder, can these concepts can be fully learned without actually experiencing pain? Could you even attempt to learn these concepts experientially in college? My experience so far makes me think that one may know something intellectually, without really knowing it. It seems like for so many, one may talk about design patterns or abstraction or low coupling, but until you actually try to build something the other way, the painful way, you just don't appreciate what you are avoiding. What's worse, junior developers who, for no fault of their own lack experience with "the hard way" have a difficult time understanding why one must "go the long way around" to do what seems like such a direct solution. Passing on the stories of the past and their consequences and lessons seems to be an unending challenge.

From my economics book comes this fictional story which I think illustrates the point:

"One Pepsi plant is managed by an economics major with an MBA and has a labor force with an average of 10 years of experience. This plant produces a larger output than does an otherwise identical plant that is managed by someone with no business training or experience and that has a young labor force that is new to bottling."

I definitely fall into the "young labor force that is new to bottling" camp. In the technology industry there is a tendency to glorify the young and bright rather than those who have the experience and have paid the price to learn the lessons that matter in the long run. Whenever I talk to some developer and we talk about where they learned some valuable lesson about software development, rarely do they refer to a book. Almost invariably, they say, "I worked with Joe on this project and he showed me x, y and z. I learned so much working with him." Smart developers, experienced developers who know how to teach and share important lessons to junior programmers seem like a key to the experience problem. Sadly, junior developers who have the presence of mind to ask, listen and apply what they hear are hard to find, and senior developers who are both willing and able to teach effectively are even more rare.

Perhaps the answer to all this is simple. Perhaps it's just he or she who writes the most code, wins. What I mean by this is simply when one writes a lot of code that increases the probability that he or she will make more of the key mistakes needed to learn how to write software with longevity. Just getting more exposure to "how bad things can get" helps bring a sober reality to each line of code written thereafter.

What's remarkable about all this, is that failures in teaching and learning are what keep us "discovering" new ideas that are 40 years old. Truly, "there is no new thing under the sun."

As with most forms of art, there are those pieces of music or sculpture or painting that you'll dislike. Perhaps what they portray or teach don't match with your ideals of right and wrong. You'll disagree on a moral level. Perhaps they bring forth memories of the past. Maybe they just look or sound chaotic and simply don't make any sense to you. In all honesty you may not know why you don't like the art, but it might just be grating to you and make you want to turn away. Still, there was and often is a real, living, feeling, breathing human being full of senses, sympathies, misgivings, prejudice and paradoxes behind that creation. Behind all art, prose and poetry are the feelings approximated in the expressions of their craft. The ability to see and feel through the art into the heart of another person, this is the challenge and amazing quality of art. Everyone has a song and they are always singing it. They want to be heard, really listened to, and find out they are not alone.

Appreciating art because of the human behind it, has for me become a simile to working with people with whose opinions I disagree. My Dad would say, "Son, behavior has its reasons." I first remember him saying this when I worked as a Scout leader and got a front seat to the varied expressions of teenage boys. Invariably as I got to know each boy, I found many reasons for strongly held opinions, and behaviors. Often, the life experiences behind these behaviors even for young boys are deep and poignant.

I see similar issues when working in a team of people. The disagreements had in office discussions of the "obviously objective" technology problems often have their roots in other aspects of life much deeper and more powerful and often hidden. This is why teams that have learned to interact with each other "off the clock" as friends and treat each other as respected individuals, for who they are, today, are more successful at solving problems.

I like a good debate and enjoy the challenge of a difficult problem and opposing viewpoints. My experience at Microsoft was of a very "challenging" kind of culture. There you can't squeak out an idea without several counters and objections. This can be good, honest disagreement, but also can turn into ugly contention that yields no redeeming fruit. (It can also drown out innovation since ideas don't have time to germinate and grow and most importantly interact.) Everyone has different tolerance levels for this kind of discord and those who are naturally shy, not quick witted in a debate, or easily persuaded will often find themselves quieted and discontent. This is especially sad when that individual is full of really good ideas, ideas that need to be listened to and acted upon.

Certainly you can't change others, but how can you avoid the destructive discord? How do you know when a good debate has turned into a bad debate? I have noticed in myself the following warning signs:

Respect. If I don't have a deep respect, on a personal level, for the people with whom I'm working, a discussion can degrade incredibly fast. When I'm working with people like this, I have to keep on a higher state of alert.

Civility vs. Hostility. This includes the obvious things like pointing and repeating "you" a lot. In all things keep the discussion civil. Take the time to reinforce with sincerity that you think there's something you don't understand. Let them know you are pushing forward because you think there's something important worth understanding.

Desire to understand. When I sense my desire to understand dissipate and my desire to prove myself right increase, this is a sure indication that the discussion is heading down a destructive path. I can feel it. There's something not right in the air. I get tense, not relaxed. These are signs for me that I need to regroup, reevaluate and possibly try again later.

Judgement. Another indicator is if I am in my mind passing judgement on the individual. I can sense this when I find that I'm thinking about what I'm going to say next while they are still talking, or when I interrupt their thoughts and don't let them finish. I think I already know what they are going to say, so why wait it out? This kind of impetuous behavior indicates that I'm placing myself on a higher moral ground, and this lack of humility doesn't allow understanding, and without understanding, there's little possibility of unity or resolution.

I'm sure there are other even better ways to avoid the pointless and destructive arguments, but this is what I've found so far. In reality, everyone has a song. Listen to it. Find the art in it. Discover what it is saying and if possible the reasons behind the melody. Don't fight the music. There's a person in there waiting to be found.

Microsoft's recent introduction of Live Mesh is a perfect example of a core difference between Apple and Microsoft. Apple is, at heart, a product company. Microsoft is, at heart, a platform company. Both produce products and platforms, but the way they approach the problem and communicate with developers is decidedly different.

With Live Mesh Microsoft says, "Come, build on our platform and do amazing things!" What is the oft cited example for using Live Mesh? Multi-device synchronization of data. Now, to be sure, this is a big and very hard problem to solve. In fact, I really wish Apple's Sync Services were much, much better, but do you wake up in the morning thinking, "Man, I really have a Multi-device synchronization problem?" Most people don't. What they do think is, "Man it's great that when I put stuff on the web, I can get it wherever I am. All I need is a web browser." You see, syncing folders or sharing data across devices isn't top of mind in the way a developer thinks about it. Microsoft's challenge is mapping their platform to real problems in a persuasive way.

Contrast this with Apple's normal product focused pattern. First they release a product that solves a real, tangible problem people have. Say, "I hate it that I have to carry around my iPod and my Cell phone everywhere!" or "Man I wish I could buy that cool song I just heard, right now." or "I have so many digital photos, I wish there was an easy way to do something cool with them." or "Man I hate it when I loose files on my computer, I wish I could just go back in time." To solve these problems Apple, like Microsoft, has to build a platform, but this platform is built for a product first, which they use and improve. Then when they talk to developers they can say, "Did you see this cool thing we just did? You can do the same thing or even something better! Here's how we did it..." It's the difference between saying, "Here are the tools, let me show you how to use them." and "I used these tools to do this great thing. Let me show you how and maybe you can do the same thing." Ultimately, it's leadership by example.

Another great example is Apple's use of the Cocoa APIs in their own applications. Apple builds amazing products and then is able to say to developers, "We used the same APIs available to you today!" Contrast this again to Microsoft. Windows Vista was released with some remarkable new C# APIs. Many of them very cool and very interesting, but what is Microsoft Office written in? C and C++. What APIs does Office use? A multitude of Office only APIs and libraries shared among the applications. Does this hurt C# and the new "WinFX" platform "street cred"? I think so.

Ultimately, Apple and Microsoft are trying to solve many of the same problems, but the path you choose while logical to you, may not be so logical to those you most need to persuade and inspire. No one can argue with the results. For me Apple's "solution first, platform second" approach makes for easy understanding of new ideas as well as providing the activation energy needed to try something new.

A few tweets ago, (I always feel weird referring to Twitter in the past tense) I posted: Why are the unintelligent or uninformed so arrogantly confident while the intelligent and well informed so often unsure and apprehensive? There is something very human to thinking you know more than you really do about a subject or issue. While, this problem can be seen in many areas generally it's particularly acute in software development. For example...

AppleScript in Office X

Back when Mac OS X was just about to go 10.0.0 we were busy working on getting Mac Office working on the new platform. There was a lot of pain involved with the transition. Many APIs had been removed and alternatives needed to be provisioned (and tested), the new Aqua interface guidelines had to be applied to the whole of Office and a host of other issues needed to be addressed. I was at the time on the team that wrote the tools for automated testing and I was pushing hard to get Office wide AppleScript support on the list of features we'd commit to doing for Office X. I wanted this not only for our customers, but to augment our testing efforts. With an API to drive the applications we could automate many "smoke tests" on a daily basis as well as set the stage for long term AppleScript based test suites. Automation testing benefits are often not fully realized, especially static automation, until the version after you setup the automation, so I was especially anxious that we get our API in Office X, so we could reap the return on investment for Office 2004.

I remember talking with Jim Murphy about getting AppleScript support like it was yesterday. We were in building 44 and his window office was near the 2nd story walkway that connected our building to the cafeteria. In my naivete I asked why we couldn't just build a some kind of layer to call into the engine code of Office and presto, AppleScript dictionaries for Office would be complete. I pointed to some improvements Apple had made to make AppleScript a better inter-application communication protocol and ask, "How hard could it be?" Jim turned to me and said, "Hard? There's nothing easy about it. It's all pain. All pain." Then as was typical, turned back to his work, leaving me to think.

In this case, I was the unintelligent and uninformed, but very enthusiastic novice. Jim turned out to be right. For Office X, we did try to do the AppleScript work, but it turned out to be much more difficult a problem to solve. After months of work and many dead ends, we pulled the feature from the Office X feature list. For Office 2004, we tried again. In this case, Jim himself, one of our best developers, ended up spending the better part of a year getting AppleScript to work with Office, which is probably worth another post in and of itself. I thought I knew more than I did, I thought the problem was simpler than it was. My confidence and thinking was miscalibrated. The root cause was my lack of understanding and experience.

Wicked Problems

Sometimes, even the best developers underestimate the scope, breadth and depth of a problem. In a drive for simplicity, I observed in myself and others another kind of design time problem where thinking was miscalibrated. The cycle I have observed looks like this:

1. Developer looks at a problem A, and thinks he or she fully understands the problem.

2. Developer designs a simple solution to problem A.

3. Developer codes the simple solution.

4. Developer or tester or marketing or customer use simple solution for problem A.

5. Bugs trickle in and solution A is modified, bit by bit, bug by bug, until it is patched in a thousand ways and no longer looks or acts like the simple solution. The solution is complex.

Some will say, "Hey! Why didn't that lame developer take the time to really understand the problem? Then he could have taken the time to fully comprehend the complexities of the problem, and then design a simple, elegant solution!" The problem with this is that many subtleties to the problem do not manifest themselves until very late in the development process which make re-architecting the solution for such a small issue un-reasonable. But this is a "death by a thousand paper cuts" issue. What's worse, many architectural issues can only be comprehended after years in a problem space, and as promotions or attrition happen, the so called simple solutions, with their complex instantiations proliferate in code.

Adam Richardson spoke to the challenges of wicked problems when he said:

Wicked problems are very difficult to understand by staring straight into them and looking for clear detail, however. They need to be approached from the edges, sort of like doing a jigsaw puzzle where you find the edge pieces first. Having peripheral vision that is trained to be sensitive to the edges is a key capability (this applies both to product teams and to business units - wherever wicked problems occur).

Normally this causes the developer to tack on simple solution to solutions with patch upon patch applied with the Hippocratic Oath echoing in their ears to "do no harm", but deceptively the bandage is not big enough for he wound, and no one knows.

The Desire to Learn

Another reason why some can feel "informed" when in fact they are not is that they have lost the hunger to learn. They've lost the desire to learn and grow. René Descartes said it this way:

Good sense is the most equitably distributed of all things because no matter how much or little a person has, everyone feels so abundantly provided with good sense that he feels no desire for more than he already possesses.

There is a reason for diversity of opinion, it provides the natural tension that keeps us from thinking we've got the problem solved. When I left Microsoft to go back to school and finish my degree, many asked, "Why are you doing that? What are they going to be able to teach you?" I'll admit that this partially resonated with me. I wanted to be the sage of wisdom, but as I've spent time with teachers and students, many of whom do have less "industry experience", I have learned an enormous amount! To put it bluntly, I have been humbled. I've learned things technically, intellectually, physically and even spiritually. I feel so blessed. I'm beginning to think that what any given situation can teach you depends largely on the person experiencing the situation and very little on the experience itself. Put another way, there's a great difference between 50 years of experience and 1 years worth of experience repeated 50 times.

Personal Pride - the Anti-change Agent

Closely related to the lack of desire to learn is the desire to avoid being wrong. So much in school is focused on being right, knowing the right answer to a test or the correct proof or solution. In sports, no one likes to lose. Everyone likes a winner! But this desire can work into our minds in a limiting way. Leo Tolstoy wrote:

I know that most men, including those at ease with problems of the highest complexity, can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.

At work I would say, "We don't fail half as much as we need to." I still think this way, but now I've found a new reason: Failing keeps our mental muscles and joints from stiffing with pride and forming into the arthritis of the mind.

The Well Intended Deception

Another reason for metacognitive miscalibration I think stems from the very basis of good object oriented abstraction and encapsulation. When I write some simple app with Cocoa's AppKit and Foundation libraries, I am, as they say, standing on the shoulders of giants. I've wondered how many actual lines of code are called when I double click a simple app like TextEdit. Who wrote these lines of code? When did they actually get written? What were the discussions behind their design and implementation? Even Apple couldn't fully answer these questions. It is all this code that even the simplest of Cocoa applications depends upon. But still we continue to have demos around how "few lines of code" were needed to accomplish something. This is a well intended deception, because really, more code is being executed and written, but only now how this code works is opaque. It's well intended because who wants to write, maintain and debug more code? Why not offload this to Apple? I expect these demos to continue, because less code for a developer to write is the canonical example of efficiency, but there's something wrong that comes from this. It's the feeling that you actually really understand what is going on to make your application tick. These assumptions and abstractions can be benign, but they can also make one feel and think that they know more than they do and can do more than easily possible.

In the process of detail management that is software development, making things simpler and more tractable is an excellent goal, but there's a limit to how "simple and easy" things can get and still be valuable. Recently I overheard two students talking about building an iPhone application. The one said to the other, "Dude, they showed this awesome demo of Spore, where in only 2 weeks, 2 weeks! they got this awesome iPhone game running! It's so easy. We can totally build an awesome iPhone app!" First, I love the enthusiasm and I hope they do in fact build an awesome iPhone application. Apple was trying to demo how easy it was to build iPhone applications and I think we've never seen a mobile platform that's better for developers, however:

Taking a senior developer who as spent the last several years of his life building games and developing Spore, bringing him into Apple with full and uninterrupted access to Apple's staff of iPhone engineers that have been developing the iPhone SDK and iPhone applications hardly equates to a college student being successful at all the complexity involved with boot-strapping a company and building a successful iPhone application from scratch. Building great software is still hard, hard work. It's tedious, often un-glamorous and takes someone who doesn't mind getting down and dirty with the details to make something great. Most will simply give up, and even those with the passion and stamina to continue are hardly assured of success. I suppose I'm arguing simply that a healthy dose of Steve Job's famed "Reality Distortion Field" do not a great developer, company or software make.

There are probably other reasons we tend to think we know more than we actually do. But the lesson for me is this: I need to take some time to ponder and reflect regularly. Am I in the "unintelligent or uninformed and arrogantly confident" camp? If so why and how can I get humble? Am I part of the "intelligent and well informed but unsure and apprehensive" group? If so why and what can I do increase my tolerance for risk and decrease my fear of being wrong? A little meta I know, but the title should have warned you. :-)

You don't have to understand calculus to appreciate this entertaining story:

A teacher, trying to explain what a theory is, asked this question: "If you take a letter half the distance to a mailbox and stop, then start over going half the remaining distance and stop, then repeat the process over and over, theoretically will you ever really get to the mailbox?" One bright student said, "No, but you'll get close enough to mail the letter."

There are many definitions for what a heuristic is, but the one I like best is illustrated in this story. A heuristic is "close enough." I am beginning to believe that more and more what matters in software isn't so much building the perfect algorithms, that match and mimic real life in every way and hold up in every edge case. What matters most is getting a model that comes close enough. What matters are heuristics.

However, although laws summarize observed behavior, they do not tell us why nature behaves in the observed fashion. This is the central question for scientists. To try to answer this question, we construct theories (build models). The models in chemistry consist of speculation about what the individual atoms or molecules (microscopic particles) might be doing to cause the observed behavior of the macroscopic systems (collections of very large numbers of atoms and molecules).

A model is considered successful if it explains the observed behavior in question and predicts correctly the results of future experiments. It is important to understand that a model can never be proved absolutely true. In fact, any model is an approximation by its very nature and is bound to fail at some point. Models range from the simple to the extraordinarily complex. We use simple models to predict approximate behavior and more complicated models to account very precisely for observed quantitative behavior. In this text we will stress simple models that provide and approximate picture of what might be happening and that fit the most important experimental results.

The textbook then goes on to explain postulates for this model of thinking, many of which by themselves are absolutely false, but taken together and used properly, they create a system of thinking that works for a wide range of situations. This collection of half-truths produce a half-truth baked solution, absolutely, but one that is indeed close enough.

For example, some "half-truths" or "simplifications" if you will, involve assuming things like: each molecule of gas is perfectly spherical in shape and any collision is perfectly elastic in result. Or worse, the volume of all these individual molecules is assumed to be zero! Individually, each of these 3 statements is categorically false, but they were the right bits to "design away" as they defined the problem space so that the model could be simplified, made useful and the problem of dealing with billions of particles be made into something tractable.

To me, this is more than simple object oriented encapsulation and abstraction. This is thinking about the whole problem differently. It's about looking at individual behaviors from a very small sample and knowing, by some spark of genius, which attributes are the important ones to the ultimate outcome of the system as a whole, and which are not. This is writing software that is able to predict things, for example, test software that is able to predict when "something good" has happened and also when "something bad" has occurred. This is about designing software by building software models that categorically do not reflect the real complexities of the system, but that taken together, get "close enough" to do real work in the real world. Ultimately your models will fail when pushed to the limits, but even just understanding those limits will help you better understand the problem you are tasked with solving! It even begs the question: What kind of programming language best allows for the definition and use of heuristic models?

I don't know where I first heard this, but someone once said something like, "Where Microsoft codes if statements, Google codes in Bayesian probabilities." There are more data and variation in that data than there ever was before. If you are going to write or use programs (very likely) that deal with large amounts of data (also very likely) it might be a good idea to get used to thinking about things in heuristic terms. It may not be exactly perfect in every case, but it will be close enough.

When I was a little boy my Mom had me memorize this little poem:

Stick to your task 'til it sticks to you;
Beginners are many, but enders are few.
Honor, power, place and praise
Will always come to the one who stays.

Stick to your task 'til it sticks to you;
Bend at it, sweat at it, smile at it, too;
For out of the bend and the sweat and the smile
Will come life's victories after a while.
--Author Unknown

I think my Mom had me memorize this poem because she knew I would need it. She understood better than I the old adage that "Life does not reward us for effort expended." Finishing is required.

For me, it is exciting to find a problem and imagine a way to solve it. The creative exhilaration in coming up with a solution that will work within all the constraints involved is almost intoxicating. I have a remarkable tolerance for ambiguity and when the major "problems" as I see them, have been solved, filling in all the details seems so much less important. The hard design work has been done. There's perhaps little glory in all the simple, small and detailed work needed to connect the dots and make the grand vision a reality. However, software is ultimately just simple 1s and 0s and if you don't fill in all the details, then all you are left with is a dream. You've got to have both the vision and the finishing of all those tiny details.

"This is all your app is: a collection of tiny details." - Wil Shipley

Missing a few details can drastically reduce the value of the whole idea. I guess that's why I love Mac software so much: There's the constant demand from both the users and my peers for my concerted effort across the entire spectrum of "pie in the sky" ideal to actual, practical details in implementation. To make it work in software, you need to consistently execute well across the whole spectrum of work. Let me underscore the words consistently execute again, they are very important.

Matt Ball recently wrote a nice post about some up and coming Mac developers that worked so hard on their first release, but since then have produced relatively little. They haven't created new apps, updated their 1.0 apps, even posted to their blogs. Some still have ideas in picture form posted in all their high fidelity glory, but with no application to show or sell to the customers that have been waiting to see the finished product. These developers seem to be struggling with consistently executing against their plans.

Matt goes on to explain that he thinks this is related to their young age. Most of these developers are young (19 years old) and he thinks suffer from some kind of "shiny ball syndrome" where they are easily distracted from one project to the next. I don't know the developers, and they could be easily distracted or they could have absolutely justifiable reasons for their delays, but the result is the same: Doubt builds as to their ability to consistently finish their ideas. Their credibility and reputation weakens.

Many of life's failures are people who did not realize how close they were to success when they gave up. - Thomas Edison

I don't think it's age. I think that's far too simple an answer. I know developers in their 50s who struggle from this exact same problem. It's not size or lack of resources either. Look at Microsoft. Here's a company where a large part of their problems revolve around consistently producing, not a lack of money or great people or innovative ideas.

The idea is not the thing!

One part of the problem comes from patent law. There is a remarkable and universally held assumption that ideas are worth a great deal. I will not say ideas are worthless, but they are worth far less than most of us realize. Even the most simple idea takes remarkable effort, and follow through to get designed, built, packaged, and ultimately used by others. This is why I felt My Dream App was destined for difficulty. They had enthroned ideas as the product, when in reality it's all the grunt work after the idea that make the product. It's all those pesky little details and the consistent effort required to follow up and deal with each of them that matters.

Finishing is the thing!

When the iPhone was released, there was a collective groan world wide from designers who had years before envisioned the ideas that Apple had now so beautifully produced in a finished product. Kim Lenox, Senior Interaction Designer of Adaptive Path explains:

With the launch of the iPhone, I've been hearing many grumblings from interaction designers who've worked for various, well known consumer electronics companies. We can all see in the iPhone aspects of our concepts from years past that were brushed aside or died prematurely. Our concepts are suffocating under the pile of NDA verbiage, never to see the light of day. What sets our mere concepts apart from this final product however, is a company with leadership who has the fortitude to take the risk, find the budget, and push the technology for the single cause of designing compelling user experiences. Apple got it right.

Amazing isn't it? Once again, it's all about execution and finishing, not just the ideas. Leadership is important for sure, but finishing the job in a company is so much more than Steve Jobs simply saying, "We're going to build an iPhone and it will have a compelling user experience." It's thousands of decisions made by hundreds of employees at Apple and elsewhere. It's dealing with setback after setback and still pushing forward. It's taking the right calculated risk (EDGE and AT&T) and saying no to other things (10.5 on time, a Dev SDK) in order to finish. That is the task of finishing and at Apple, it seems to be part of their DNA.

Tranquil and Steady Dedication of a Lifetime

One of the most important attributes of a software company I would like to work for, comes from an idea the late Adlai Stevenson a U.S. Democratic politician explained when referring to patriotism:

What do we mean by patriotism in the context of our times? I venture to suggest that what we mean is a sense of national responsibility ... a patriotism which is not short, frenzied outbursts of emotion, but the tranquil and steady dedication of a lifetime.

In a great software company, there wouldn't be "short, frenzied outbursts of emotion" but a consistent focus on finishing in a steady and sustainable manner. My guess is that those prone to "putting on a big, glitzy show" and those that don't effectively resist the "constant bombardment of new and exciting things to try out" will have set themselves up as an unsustainable business, and ultimately end up disappointed.

Saying No: a feeling of strength in reserve.

One of the biggest challenges is just saying no to things. What's hard about this is often you need to judge between what is "good", what is "better" and what is "best". In order to do that which is "best", you will, you must say no to many, many things that are "good" and "better". This is heart wrenching work, but choosing what you do now to remain focused and finishing, this is your competitive advantage. When asked what work he was most proud of from among his work at Apple, Steve Jobs famously said, "All the products we didn't ship." Many people and businesses talk about focus and priorities, but very, very few actually finish the idea and follow through with the well executed decision making and focus required.

"You must always work not just within but below your means. If you can handle three elements, handle only two. If you can handle ten, then handle five. In that way the ones you do handle, you handle with more ease, more mastery and you create a feeling of strength in reserve." - Pablo Picasso

Keep Moving Forward!

There is a scene in Disney's animated movie Meet the Robinsons that I love. The story is of a young boy inventor who is learning. During this particular scene he is trying hard to fix a peanut butter and jelly gun, used to automate sandwich building. Everyone is watching him and it looks like he's going to succeed, finally the time comes to try his fix. The whole thing explodes sending peanut butter and jelly everywhere and onto everyone in the room. He is devastated, but immediately he hears cheers and people start to comment on what a great failure that was! "You Failed!" "And it was awesome!" "Exceptional!" "Outstanding!" "Uh, I've seen better." "From failing you learn, from success, not so much." They congratulate him like he succeeded. They ultimately propose a toast to his brilliant failure. He is stunned. The motto of this family is: Keep moving forward!

This is a good motto for anyone working with software. The challenges are so great and the problems so complicated and frequent, you simply must have the determination to keep moving forward, to and through the finish. Start small and build momentum and keep finishing small things, just to keep in the habit of it.

This is not always easy. Sometimes I would come home from work so frustrated with how slow things were going and how little progress was being made, I'd tell my wife I just needed some time alone to cool down. I'd go into my room, open my laptop and write a blog post. I'd post it and point to it while saying to my wife, "There, I did it. I produced something today! It may not be much, but at least I produced something tangible!" You've got to keep in the habit of producing or finishing. You can't let those muscles atrophy.

One of the best development techniques I've seen over the years is test driven development. The pattern is to build a small test that represents an improvement you want to make to your program. Once the test is built, run it and watch it fail. Then write just enough code to make the failing test pass, then run the code and watch the test pass. Repeat. This tends to lead to low coupling and good cohesion and a reasonable test bed. The real hidden value is regular focus on tangible completion in a consistent way, over time. Just keep moving forward step by step and then after some time you'll be impressed as you look back on the mountain of work you've accomplished by such simple means with constant effort.

There are bugs to be fixed, old code to re-examine and refactor, performance problems to analyze and improve, build automation, test automation and website improvements, help docs to write, blogs to read, posts to write, ideas to explore, customers to contact, emails to read and write. Your job is to choose which of all of these you will do now, and then keep moving forward. Those who master the art of consistently and sustainably producing value, are setup for success. Be one of them. Don't quit. Don't stop. Focus on consistently finishing something of value, no matter how small. That's what the world will pay you for, and since so few seem to stick to it, there's plenty money in play for those who pay the price to consistently finish the job.