Bob and Beyond: A Microsoft Insider Remembers

Behind the scenes with Bob and its offspring--Clippy and the Microsoft Agent.

By  |  Monday, March 29, 2010 at 1:00 am

Building a Better Bob

During this time, I set out to try to fix some of Bob’s implementation issues. First, I would create an interface that was not proprietary such as Bob and Office 97 had done, but available for any developer to incorporate in a Windows application or Web page. As a part of that I would include tools that enable developers to create their own characters.

I also set out to make the character able to more closely integrate with an application (or Web page) by displaying each frame of animation in a window shaped to the character’s frame. In essence this enabled the character to be manipulated like “sprites” in video games, able to move around the interface without the underlying application having to repaint the screen when the character’s animation window moved. This now allowed the character to point to virtually any element of an application’s interface.

I then added an option for speech input and synthetic speech output (text-to-speech). Microsoft’s own speech recognition could be used as well as those from IBM, AT&T, and others. I also licensed a number of text-to-speech engines that developers could distribute as the voices of the characters. Finally, I had an animator create four characters including Robby (a robot), Peedy (a cracker-eating parrot that had originally been created by Microsoft researchers), Merlin (a short bearded magician), and Genie (a blue genie). I dubbed the product Microsoft Agent, and posted it to the Web for free download. My goal was to create a technology that would enable any developer to incorporate it.

The character just became a sugar coating for the Help interface.

The Office team picked up Microsoft Agent for their next release, but opted not to use the characters I had created as they preferred their own unique ones. To avoid the past user-reported annoyances, they gave users more control over when the character would appear, but did little to reform its behavior when it was present. So, you still had the same cognitive disconnect between the character’s reaction to your actions in the application’s primary interface. The character just became a sugar coating for the Help interface, which, if it failed to come up with useful results, left the user unimpressed and thinking that the character was not very useful.

Similarly, the Windows team picked up my Agent code for the search interface and the user assistance in the registration user interface. It was really only in this latter usage that the technology started to approach what I had intended. Here the character only appeared when summoned and was actually implemented to offer help contextually. So if the user had a question about a particular field or form or had not completed some required information, the character would move and point to that location and say something helpful (e.g. “you need to enter your zip code”). In addition, the interface had progressive assistance, so if you repeated an error or asked for help on the same thing, it would provide more expansive assistance.

I had also tried to add more substantial “intelligence” to the interface that would better interpret intent from a query a user would type in using techniques like natural language processing techniques such as semantic and syntactical analysis. But alas, it was too late at that point. The reputation of Bob and ineffective reforms of the Office Assistant had tainted the waters.

There actually was data that many users liked this kind of interface and were just just annoyed about certain aspects of it, but Microsoft finally decided to abandon any further attempts to do anything with virtual personality-based software agents (except in Microsoft Research where the concept continues to be explored). Clippy was relegated as a joke, in a series of online animations, voiced by comedian Gilbert Gottfried (a very fitting assignment). As a result, I also went on to work on other things.

Bob in Retrospect

Bob is a bittersweet story to me. I still believe that Karen’s motivation and Nass and Reeves’ research point to an important opportunity to make interfaces easier and more efficient. In the area I focus on now, personal robotics, I see the human-robot interaction definition as one of the biggest challenges facing the nascent robots industry. Research from several universities indicates that even Roomba users treat this technology as a social personality, giving their floor cleaning robots names or other social characteristics.

I still believe that Karen’s motivation and Nass and Reeves’ research point to an important opportunity to make interfaces easier and more efficient.

Even soldiers using the iRobot Packbot in Iraq often reference their reconnaissance and IED defusing robot as if were a team member. When sending robots back for repair, teams asked for their specific bot to returned. And as interfaces evolve to be more natural, the social aspects of human interaction become more important. For example, there is ample evidence that speech interfaces are socially modulated – how something is said and who speaks it are very important to how it is perceived.

Some may wonder why Microsoft spent so much time on interactive social agents in the interface. A part of the answer reflects that people at Microsoft had a sincere desire to improve the user interface. In addition, the culture of Microsoft is that the people that work there do not give up easily—they play to win. Finally, the impact of social aspects of design was compelling, but is harder than it seems at the surface to design well.

I know that not everyone will see the potential value of appropriate applying social principles to application interfaces. But then it took some time before the cognitive aspects of good user-centered design were explored and well applied. And to this day there are some users that still prefer command-line interfaces to down-down menus and other graphical components. However, it is clear that people respond to such social attributes of technology, and learning how to apply them appropriately and effectively remains an area we need to understand better.

More on Microsoft Bob and the Office Assistants at Technologizer:

The Bob Chronicles

A Guided Tour of Microsoft Bob

The Secret Origins of Clippy



Read more: , , ,

7 Comments For This Post

  1. tpzahm Says:

    Thanks for a wonderfully balanced account.

    I’d like to add an example of how the implementation of the ideas behind Bob was deficient: If you entered three wrong passwords in a row, your assistant would offer to change your password for you! The zeal to make things friendly destroyed the underlying function (security, in this case).

  2. Gordon Says:

    For quite a while I used an office assistant that wasn’t from Microsoft. I can’t remember it’s name right now, but it was a rat.
    I liked it, because it had attitude, which it exhibited by some rude behaviors. I wished it had been more complex instead of such a rudimentary, simple device.
    That is another way that the MS Office assistants are distasteful to people like myself– they are OBSEQUIOUS and flat, in a word– BORING.
    If someone will ever succeed with this idea, above all else they must make the characters have some attitude, some spunk, verve.
    If you make it exciting and fun, I will buy it!

  3. dudette Says:

    Actually, if someone ever succeeds with this (Office Assistant) idea, they have to make it easy for the user to permanently turn it OFF! With Clippy, you could change to other “cute” animations — what you couldn’t do was set it to “never show this crap to me again!”

    Microsoft’s continuous FAIL on UI comes from a fundamental misunderstanding of how (and why) human beings interact: when people are trying to accomplish a task, they are NOT in the mode to be entertained.

  4. Peter Laman Says:

    I believe there is definitely a place for an ‘office assistant’, but only for one that avoids the pitfalls Bob and Clippy suffered from:

    1 – Only on demand. Automatically popping up and telling the user what to do will annoy the user. After a while the user is familiar with basic functionality and then the need for an assistant disappears. At that point it starts to be irritating. Just provide kind of a ‘guide me through this’ button thus giving control back to the user.

    2 – Don’t try to be funny. By nature, computer programs perform repetitive tasks and repetition of funny gadgets is not funny at all. Who wants to hear the same joke over and over again? So now dancing paper clip, or other nonsense, but just a sober pointing device with a balloon, or spoken text to assist the user.

    3 – Avoid all childishness. Adults don’t want to be helped by a puppet on the screen. Just give the information, that’s all. A talking dog would be appreciated by my 5 year old son, but he doesn’t use office applications very often!

    Maybe it would be a good idea, if applications had a kind of ‘tutorial mode’, that offers somewhat more assistance to the beginning user. Once the user masters the app, he can switch it off.

  5. Jeff Says:

    The most annoying thing I recall about Clippy and Friends was the sudden disk grinding and utter loss of control over the system.

    *type type type* I type my salutation line
    *grind grind grind* uh-oh
    *dink dink* a window border has appeared. Crap.
    Some time later: “hur hur! Looks like you’re writing a letter!”
    I continue to furiously click on the close box to no avail. The system is too busy loading DLLs and rendering ridiculously advanced (for the day) animations.

    Maybe the experience was better for business users with the latest and greatest hardware, but I doubt there were many of those. I was just trying to write design documents for my college project on second-rate CS lab boxes…

  6. Descartes Says:

    This is a very fair article, and I agree that the group that has developed Bob and the Office Assistants have good intentions. However, I agree with the commenter regarding the ‘childishness’ of the agents. It is not that the developers intended to insult users, but the appearance of cute, comical on-screen animated characters in a serious business application creates an unintended consequence: it would seem to the legions of hardworking office users that the developers believe their customers to be simple-minded– like small children– and need extra-special help to understand computers. This appears to be an unfortunately recurring theme in the philosophy of MS UIs– that ease-of-use equate to stupidity, producing endless products that incorporate cartoon characters, endless dialog boxes, and pedantic wizards. The success of Google’s search portal in its stark bareness and the iPhone OS UI’s quiet simplicity demonstrate that what users truly desire is a minimum of options and interface that gets out of one’s way.

  7. Karen Fries Says:

    Well, I’ve read a lot of analysis of the story of Bob over the years, the reasons it failed, etc.

    The thoughts by Tandy are entirely accurate and the story itself gets the history right for the most part.

    Well done.

    I’m still licking my wounds.

    I just got a new puppy and didn’t even name her Rover.

1 Trackbacks For This Post

  1. Обзор свежих материалов, апрель 2010 « UI Warehouse Says:

    […] Bob and Beyond — A Microsoft Insider Remembers […]