Hello.
Welcome to the LCS Distinguished Lecture Series.
For those of you who are not familiar with this series,
I am not Michael Dertouzos.
There are several distinct differences
which I won't go into.
Michael asked me to substitute for him.
I've decided not to give the traditional introduction
and instead to leave some time for our distinguished lecturer
to speak himself.
Our lecturer today is Andries van Dam
from Brown University where he's the T.J.
Watson, Jr., University Professor of Technology
and Education and Professor of Computer Science.
It's certainly an impressive title.
And in fact, he's even an impressive fellow
to go along with the title.
He was an undergraduate at Swarthmore College,
got his doctorate from University of Pennsylvania.
And he tells me it was the second doctorate in the United
States granted in computer science.
So he must be older than he looks.
He left Penn and went to Brown University
as an assistant professor and has been there ever since.
When you started at Brown, you were in the Applied Math
Division, I guess it was called.
Yeah.
Within that, he then started a Computer Science program,
was its first head for three years,
and then co-founded the Computer Science
Department where he was its first chairman for six years.
So he seems to have an incredible stomach
for that sort of thing, though he has escaped it
and is now doing more interesting things.
He's most well known probably for his books
on computer graphics, first Foley and van Dam,
and then Foley, van Dam, Feiner, and Hughes.
I knew Andy before he was a famous person,
even before he was--
when I was an undergraduate at Brown,
I was early in his tenure there.
In many ways, you have Andy to blame for the fact
that I'm standing today, standing here today.
I'd probably be standing somewhere anyway.
He took under his wing a number of people
who came to Brown thinking they were going
to do something other than computing,
and convinced us that it would be a foolish way
to waste our lives.
And in fact, the only thing that was a lot of fun to do
was to play with computers.
And he certainly persuaded me and several other miscreants,
some of whom have been speakers in this Distinguished Lecture
Series.
So he has a lot to answer for.
He's founder of ACM SIGGRAPH from which he
received the Coons Award in 1991,
and in 1993, won ACM's Karl Karlstrom Teaching Award.
And for those of you who don't know it,
it's probably the most distinguished award
you can get for teaching in the field of computer science.
Let's see.
What else is there to say here about him?
Well, let me read a little story.
This will capture a lot about Andy.
This has to do with--
in addition to graphics, he had the misfortune
to work on hypertext long before it was a good idea.
In 1969, when I was at Brown, we would use the entire Brown
Computer Center, the thing that did all the data
processing, all the salaries and everything else
from midnight to 8:00 AM to edit a single hypertext
document, usually a rather short hypertext document,
on a 2250 display.
This was text only, but it was in 1969 WYSIWYG editor
several decades ahead of technology,
or at least 10 years ahead of it.
And Andy had a lot of faith in his students,
his undergraduates.
And he would have us do his correspondence for him
from midnight to 8:00 AM on this hypertext system,
just to show it could be used.
So one night, several students are there editing a letter
to the provost.
And I should say Andy's taught me everything I know about how
to deal with the provost.
We produced two versions of that letter, the one Andy gave us
and one in which several [? nots ?] were
inserted at judicious places and removed from other places,
thereby changing the whole sense of the letter.
About 6:00 AM, we finished, left it on Andy's desk--
the fake later on Andy's desk with the [? nots ?] in it
and out of it, and left.
Thought it was a very good joke.
Came back later in the day.
Andy didn't say a word.
Day went by.
Andy didn't say a word.
Finally, a little bit trepidatiously, people
came up and asked Andy what was going on.
What about the letter?
And he said, oh, I didn't bother to check it.
I just mailed it.
The perpetrators of the would-be hoax immediately whipped out
of Andy's office, spent the entire day
trying to retrieve that letter from various sources
throughout the university mail system.
At 6:00 PM, having missed a whole day of classes,
returned to Andy's office and confessed
the sin, at which point he smiled
and pulled the letter out of his middle drawer of his desk.
So I think that tells a lot about our speaker today.
So as Mike asked me to, I asked Andy
if he had any aphorisms to pass on to the audience.
And he said we should be grateful for the privilege
of being paid to do things that are fun, and if what we do ever
stops being fun we should find another profession.
Andy.
[APPLAUSE]
Thanks, John.
I think that's an Ed Lazowska story that might actually even
be true.
It's a privilege to be back here.
Last time I spoke here, I couldn't speak.
And I sort of hacked and coughed my way through my lecture
and barely made it back to my seat.
I'm in fine voice today, I hope.
And I'd like to say that I got started
in the field about which I'm going
to be speaking today by someone you should all know about.
Some of you are too young to have met him in person.
Some of you knew him, and that's J. C. R. Licklider
whom I believe to be the first person
to have emphasized the importance of user interface.
A great man, a great human being.
And in a sense, I'm giving a lecture in his honor
here today.
So let me tell you what my main thesis is.
You've already read it in the abstract.
It will not surprise you.
I got to learn not to point at the screen.
That's the first trick.
Escaping Flatland.
Our user interfaces are flat today.
And what do we intend to do about that?
So I'm going to motivate why there is something
beyond what we're used to today, talk a little bit about what
it feels like to design 3D user interfaces,
then switch hats a little bit and talk about some
of the implementation software issues,
and then draw the inevitable conclusions.
So let's talk about user interface
and what we would like.
And what we would like, I believe,
is no user interface at all.
In fact, we wouldn't like any of those stinking computers.
We'd like to be able to interact as we
do with our fellow human beings and talk, and emote,
and move about the world and not ever be aware of the fact
that there are computers and that we must control them
or steer them via user interfaces.
The closest thing to a vision about this
was given to us in Stanley Kubrick's immortal film, 2001.
And I just want to refresh your memory with a little, tiny clip
from that.
[VIDEO PLAYBACK]
- [INAUDIBLE] this video provides fully interactive
direct [INAUDIBLE] replacing and modifying--
Not that one.
Forget that, a miscue here.
- [INAUDIBLE]
[END PLAYBACK]
Somehow, it seems to have rewound itself.
I hate to do this.
[INAUDIBLE]
Yeah.
I left it paused, and it seems to have come unglued.
OK, let me rewind.
Sorry about that.
[VIDEO PLAYBACK]
- [INAUDIBLE] along its length.
We have positioned [INAUDIBLE].
[END PLAYBACK]
Play, rewind.
Actually, I'll stop first and then rewind.
Ah, the wonders of technology.
[INAUDIBLE]
[LAUGHTER]
So you know, I can't even tell whether it's doing anything
because nothing much is happening.
[INAUDIBLE]
Now I'm hitting the rewind button.
Good.
It did something.
All right, let's try again.
I would like the room to be as dark as possible
when the tape is on.
And you certainly don't need me for video purposes.
All right, here we go.
Oh, nuts.
[LAUGHTER]
See, this interface is modal, and that is a really bad idea.
OK, so we rewind it, we hit stop to get out of the mode.
OK, and then we hit play.
Now that should work, right?
One more time.
[VIDEO PLAYBACK]
- PERSONA is a part--
No.
Not yet.
- [INAUDIBLE] user interface group at Microsoft Research.
We explored--
[END PLAYBACK]
Rewind again.
It clicked to the end.
I hit stop, I hit play.
That ought to do it.
[INAUDIBLE]
It can't possibly be worth this, right?
That's what you're all thinking.
You're right, it can't be.
It would have been OK as a 30-second clip,
but I'm going to push fast forward because [LAUGHTER] it
ain't doing anything.
Would somebody like to control this who is more mechanically
apt than I am?
I'm hitting play again.
[INAUDIBLE]
No, that's not the problem.
[INAUDIBLE]
Yeah, all right.
You want to see if you have any better
luck with this, [? Carl? ?]
There's somebody in the control center behind that [INAUDIBLE]..
I've hit play, and it does--
here we go.
There we go.
All right, long leader maybe.
Boy, am I building up the suspense.
[VIDEO PLAYBACK]
- Knight takes bishop.
Mate.
- Ah.
Yeah, looks like you're right.
I resign.
- Thank you for a very enjoyable game.
- Yeah, thank you.
[END PLAYBACK]
All right, stop.
[? Carl, ?] let's see how well you do at this.
You can't do any worse than I did.
[INAUDIBLE] plug here someplace.
[LAUGHTER]
No, see.
[VIDEO PLAYBACK]
- Several of the projects that we're working on--
[END PLAYBACK]
[LAUGHTER]
The power works.
No, no, no, no, no.
That won't do.
[LAUGHTER]
OK, maybe I have to be closer.
OK, we'll leave this for now.
We'll switch channels, get the PowerBook back.
That ought to work.
It worked before.
Oh, [INAUDIBLE] disaster.
PowerBook.
Something, all right.
So HAL 9000, what was going on there?
We were talking to the computer, or at least Dave
was, and wasn't aware of a particular interface, conversed
as if dealing with a human being.
Well, that these days is called the idea of an agent,
or some people call it a social interface.
There are folks here, particularly
Pattie Maes in the Media Lab, who
are dealing with the technology and the sociology of having
these kinds of pseudo beings do intelligent things
on your behalf.
It's a very important area.
And I'm not going to talk about it today.
Neither am I, obviously, about the next bullet,
which is what people sometimes refer to as DMA.
I think fortunately, this won't happen in my lifetime.
I can't begin to imagine all the mischief that can be put to.
So I'm going to talk about interfaces that give us
a quality that interface designers like to talk about,
which is transparency.
So what do we mean by that?
Again, it's this notion of not being
cognitively aware of the fact that you
are driving an interface.
When you drive a car, after you've mastered the art,
you are not aware of the fact that you're driving.
Your mind can wander.
You cannot even remember where you've been over the last 10,
20 minutes, because you're on automatic.
And some people get that way even in Emacs.
So that's what we mean by a transparent interface.
If we look at the history of user interface,
we can sort of see four distinct plateaus,
punctuated equilibrium in Gould sense.
Namely things are stable for a long time.
And there is some kind of profound paradigm shift,
and then you do something else.
In the beginning, there was no user interface.
We didn't even talk about the term.
And then we got to time sharing, invented largely here.
And we used glass teletypes with first real teletypes,
of course.
Those of you not old enough to remember them,
they are these archaic mechanical devices
fed by paper tape.
And you could sort of type like this on them.
And you drove your application program
with a command line interface, a shell you would call it today.
And then the big revolution was at Xerox PARC
where bitmap graphics workstations were invented.
And basing a lot on Engelbart's work and even some of the ideas
that first were shown here by Sutherland in Sketchpad,
we got the idea of carving up the screen
into individual foci of attention
doing various tasks in each of these things
called windows, and by the magic of bitmap graphics
came up with icons, and menus, and all those things that today
we call the WIMP interface, the WIMP look and feel,
the messy desktop metaphor.
You're all familiar with it.
I needn't belabor it.
What's happened since the early '70s?
Answer-- essentially, more of the same polishing
third order effects, adding drop shadows,
making things look pretty.
As far as the interface is concerned, nothing much.
From an implementation point of view, a lot has happened.
We have tons and tons of GUI building support software.
And I'll have more to say about that in a moment.
What I'm going to say today is forget that stuff.
It's going to be useful for a long time.
Keep your eye on the future, and the future
lies in post-WIMP interfaces.
So that's what the lecture will be about.
Where are we today before we launch into that?
I claim that finally, user interface technology
has become a discipline.
And we finally recognize that usability
is much more important than functionality,
above a certain threshold of functionality,
as a key to productivity.
It is nothing short of miraculous
that you have preschoolers who can neither read nor write,
yet productively use computers.
And if you find that hard to believe,
think about managers doing the same thing.
[LAUGHTER]
And if you find that miraculous, think about senior faculty
who use computers productively.
That is a revolution.
Why is it possible to do that today?
Because of all of these great interface techniques
and because we are finally paying attention
to the needs of the user.
And that is called user-centered design.
We do usability testing on software products
before we release them on an unsuspecting public.
And the corporations now spend big bucks
building usability labs and testing things and finding out
whether they're going to work before they release them.
We have a professional society, SIGCHI, devoted
to computer-human interaction.
We have textbooks and courses.
And finally, the ultimate proof, we have lawsuits.
We have Apple versus Microsoft, Borland versus Lotus,
and so on.
The field must be real.
So why have these WIMP GUIs stuck around for so long?
Why are they standard?
Well, because it is said that they provide us
with ease of usability, learnability, transferability,
and all those abilities.
X for almost any value of x for your user.
And also because we now have layers of support software
so that we can take advantage of the increasing amounts of RAM
that the manufacturers are providing
us to soak all that up and have something useful to do with 100
MIP processors, which otherwise, well,
what would you do with them?
Nowadays you service the user interface.
So we get ease of implementation and maintainability
because we have a standard way of implementing GUIs.
We have simple tool kits like the X Toolkit.
We have interface builders on top of that and even user
interface management systems.
And that layer cake provides us with lots of support.
And even so, it's still too hard to build user interfaces,
and it takes too much time.
Why do we have limitations in such WIMP GUIs?
Well, the support software doesn't do everything for you.
And in particular, if you look at your stack of motif manuals
about this high literally, more than a foot worth,
you're forced to learn two environments,
not just C and C++ but all the motif conventions
and libraries.
And you're constantly aware of the fact
that you have two environments that you
have to learn to use and to integrate
through the ugly callback mechanism.
Furthermore, the support of higher levels of functionality
is primarily at the level of layout.
Very little support for behavior,
and that's bloody hard.
We shouldn't be surprised.
So it does the part of the job that is easy,
and it does it better than the way
we used to do it 20 years ago.
We've made progress, but not enough.
More importantly, for the post-WIMP
interfaces that I'm going to be talking about,
there is a certain mold that the WIMP GUI casts us
in which is very inimical in my point of view.
And I'll say more about that.
It imposes a sequential dialogue model.
I sit in the neutral state.
I wait for the user to do a single thing.
I react to it, and I go to the next state.
Typical finite state automaton model.
It's deterministic.
You know whether a key has been pressed or not.
It's very difficult to have simultaneity of input.
Imagine how hard it is to have two data
gloves through a standard user interface builder.
It's not set up for that.
It's not set up for two-handed mousing even, two mice.
And in particular, it's not an all natural for virtual reality
of the immersive variety, real virtual reality as it were.
That is, I'm immersed in an environment.
I look around.
It tracks my head, if not other body parts.
And I have the feeling that I'm inside
the scene instead of looking out through a tiny, little glass
window to a tiny world that lies behind there.
So now I'm going to try to run the magic of video again.
Good luck to all of us.
I forgot to roll one video because I got discombobulated.
Let me back up for a moment to the issue of agents and persona
and show you that little bit of Microsoft tape.
[VIDEO PLAYBACK]
- --conversational assistant which interacts with the user
in a natural spoken dialogue.
Our eventual goal is to allow users to express themselves
within a limited domain in whatever way they
find most comfortable.
And our system will be able to respond appropriately.
In addition, our system will be given a rich visual presence
using reactive three-dimensional graphics, expressive audio--
- What?
--and cinematic techniques.
[INAUDIBLE]?
Madonna?
[CLATTER]
[CRASH]
[WHOOSHING]
[POP]
[MUSICAL NOTES]
- I [INAUDIBLE] 1990.
Would you like to hear something from that?
- Play a couple of songs from that.
- How about "Like a Prayer" and "Justify My Love"?
- Sounds good.
- OK.
[END PLAYBACK]
[LAUGHTER]
Now, it's a research project.
It's not yet ready for prime time.
But don't think of social agents like Bob as the answer.
We're much more interested in this kind of technology.
Anyhow, agents, another story for another day.
And I'm going to talk for a little bit
about virtual reality just so we're calibrated
on what I mean by that term.
So here's my first example of that.
May we douse the lights again, please?
This is the NASA virtual wind tunnel project.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
- Before the first flight of the space shuttle,
models were tested using wind tunnels and computers.
The virtual wind tunnel project combines advantages of both
by placing a scientist directly inside a computer-generated
flow field.
CRTs present computer-generated wide-field stereo images
from the user's point of view.
[END PLAYBACK]
OK, so you're inside, and you're wearing kinky black clothing.
And you're gesturing hypnotically.
That's sort of a non-scientific definition of VR.
It doesn't have to be that way, of course.
And see channel two.
What we're looking for is stuff that might even
work pretty comfortably in the privacy of your office,
and so desktop virtual reality or fish tank
virtual reality where you're wearing stereo glasses.
It's being head tracked, and you may have some gear on your hand
with which you can grab and manipulate objects.
That's another kind of probably more accessible, cheaper,
and safer kinds of VR.
Now, why am I concerned about VR and using it
as a driving application, in Fred Brooks's
terminology, for talking about post-WIMP interfaces?
Can we pull the lights down just a little?
You really don't need to see me.
And I'd like to have the screen show up better.
What we have here is a little trip
in virtual reality by our heroine.
And the reason I thought this was amusing
was that she still has a Macintosh style pull-down
menu hanging from the menu bar.
And I want you to think about whether you
want to be walking around in some kind of virtual reality,
but still use this 20-year-old WIMP pull-down menu paradigm.
I don't think that fits very well.
So what is the characterisation of post-WIMP interfaces?
Multiple channels going full bore simultaneously
and probably multiple participants.
It's very important for us to raise our sights
and not think about computing as what I call a solitary vice.
That is something that we do by ourselves.
Computing, more and more in the future,
is going to be a group activity, things where
we work collaboratively in teams together, possibly not located
in space and time, but through the medium of the computer,
working together nonetheless-- multiple participants
in shared virtual worlds, and not just for tank battle
simulations.
When we are walking around in the simulated world,
gesturing with our hands, moving our heads, talking and so
on all at the same time, we don't
have the deterministic key press which
makes WIMP interface decoding so much easier.
It's probabilistic.
We have to be able to backtrack when we make a wrong guess.
We have to be able to disambiguate.
We have to deal with objects that
have their own independent behavior, that
don't sit still as they do in PowerPoint on the screen.
So Negropontes put that their demo from the '70s
is exactly a good paradigm for what we're talking about.
So I'm walking around in the simulated world.
There are objects flying around and doing
their own autonomous behavior.
And I say, put that, as I reach out and gesture and speak,
there.
Put that over there.
What is that?
What is there?
What is noun?
What is verb?
All of that has to be disambiguated.
And that's a very, very hard job.
And WIMP interfaces give us no handle on that.
Why am I so interested in WIMP interfaces?
Not just they're sexy, they're neat,
but they're going to happen.
They're going to come upon us for the usual push
and pull reasons.
Pushing us is the technology, personal digital assistants,
the immense 3D graphics power that we're
going to have for almost free in our settop and game boxes,
more than we get in most high-end workstations
for that matter.
And the fact is that in '95, we'll
see commodity 3D accelerators for several hundred dollars
that all of a sudden will open up
the technology in the same way that bitmap graphics
accelerators got us out of the ASCII world.
Virtual reality, particularly for games,
is going to play a big part.
What kind of applications are people putting together?
Games especially.
They are going to be driving our industry.
I hate the games personally, but they are
what is driving our industry.
And 3D games are far more compelling than 2D ones.
Interactive television, if that can ever be done economically.
Online shopping, not on a 2D representation
but in a virtual mall where you'll
be able to walk in a store, and open up
the cabinets and the showcases, and look at the contents,
and spent lots of money, the developers hope.
Virtual teleconferencing where we're
doing group work together, and we're gesturing about objects
and how they ought to be modified.
Scientific visualization where we're inside our vortices
as in this wind tunnel example.
And all kinds of trainers, starting with flight simulators
and so on.
Those are among the ones.
Well, why is 3D so hot?
What are the potential benefits?
Emphasis on the word "potential,"
haven't been realized yet.
And also realize that it's extremely task dependent.
Some tasks will not work well.
I don't want to do my bloody word processing in 3D.
I'll do it on my flat screen, thank you very much.
But other things that require physical movement, for example,
or estimations of size are much better done not just in 3D
but in VR, is the claim.
So I want to be able to leverage the natural abilities that I've
developed all my life walking around, reaching out
for objects, moving things and so on.
In particular, I want to be able to reduce the physical as well
as the cognitive distance between me and the task
that I'm trying to execute.
And I'm going to explain that via a bit more of videotape.
Lights down, please.
[VIDEO PLAYBACK]
- The deformation operators are powerful,
high-level tools for manipulating the geometry
of objects.
However, users struggle to control
as many as 14 different parameters
in this 2D array of sliders.
These one-dimensional sliders have a number of disadvantages.
First, they consume a great deal of screen real estate.
Second, they artificially separate related parameters
from each other, such as the three components of a vector
or point.
Lastly, they remove all controls for parameters
from the 3D view of the object, increasing
the distance between user intention
and their actual effects.
We can see how difficult it is here
to relate the positions of the sliders
to the actual deformations performed on the object.
Our original rack combines the twist, taper,
and bend operations in a single widget.
With three handles, we control 10 degrees of freedom
with one and two-way constraints.
The angles and positions of the bars
control and indicate the parameter values
of the deformations.
The handles are modeled to look like actual handles
so that they invite twisting or moving,
but they do not indicate what their functions are until one
actually pulls on them.
By introducing a suggestive visual representation
for each of the handles, the rack widget
becomes more self disclosing.
For instance, the twisted shape of the twist
handle indicates its function.
By pulling on the twisted handle,
the object twists in direct proportion
to the angle through which the handle is rotated.
When we push down on the taper handle,
the object is made to taper down to the height of the bar.
When we pull on the bent handle, the object
bends in the same direction.
All of the operations occur in the range between the two twist
and taper handles.
The rack itself can be manipulated
to change the axis along which the deformation is applied.
[END PLAYBACK]
OK, so the idea is simply that you
act as if you're in the virtual world with a set of tools
at your disposal, a vice, a set of pliers, whatever.
And those tools are part of the three-dimensional scene,
as opposed to what I think of as the television model where
I have some sliders mounted elsewhere on the screen.
And I'm pulling on the slider while I'm
peering into the world.
That's indirect and has greater cognitive distance.
So what we want to do also as much as possible
is to shift from cognition, where I'm explicitly
aware of reasoning and thinking my way through a problem,
to having the perceptual system that I'm comfortable using
in the 3D world take over.
So it's very important that we computer scientists,
as we think about these matters, learn from our colleagues
in perceptual psychology about what cues are important
and how human beings perform tasks in the real world.
That wasn't necessary for 2D GUIs.
We won't be able to do it properly in the 3D world
if we don't learn from these folks.
I'll have more to say about that in a moment, too.
What I'm going to show you next is to say,
well, I talk about tools in the 3D world,
and that was pretty easy to understand
in the case of trying to mold this piece of clay
with appropriate tools.
But can we extend this and say, what about information
visualization where the information intrinsically
doesn't have to be geometrical?
So I'm now going to roll you a little clip from Xerox Palo
Alto Research Center, Stu Card, George Robertson, et al.
And these folks have spun off some of the work
that you're about to see to their XSoft company.
And they're making use not just of three dimensions but also
of time, animation to say we have greater information
density.
Actually, before I do that, let me show you
an alternative which some people may say they actually prefer.
This is Ben Shneiderman's treemap.
And he can show an amazing amount of information
in a very small amount of real estate.
This is a file system.
And size is proportional to the number of bytes.
And any time you see a rectangular container,
you see its contained entities, and that of course nests.
So this is a nice static overview,
and you can definitely learn a lot from it.
But the folks at PARC bet that by making things three
dimensional and by using the time dimension,
they can give you more in the same amount of time
and give you a higher level of understanding.
So let me show you that.
[VIDEO PLAYBACK]
- This visualization, called the cone tree,
shows part of a Unix directory hierarchy.
Each node is a directory.
When a directory is selected, the tree
rotates through the shortest path
to bring the selected path to the front.
The selected path is highlighted with the directory name.
3D is used to make more effective use
of available screen space.
A 2D layout of the same tree would not fit on the screen.
The user would have to scroll to 2D the layout
and wouldn't have a view of the whole tree.
Interactive animation is used to shift
some of the user's cognitive load
to the human perceptual system.
You can see this by looking at a selection done
without animation.
It takes several seconds to re-assimilate the relationships
between the parts.
If we walk into the next room, we
see an alternative layout which allows text
to be displayed for all nodes.
This visualization is called the cam tree.
As before, selection rotates the tree
to move the selected path up to the front.
The perspective wall is used to lay out 2D information
onto a 3D segmented wall.
The front segment is used for viewing details.
And the side segments provide context.
In this case, each node represents a file
with its horizontal position determined
by its date of last modification.
Vertical position is used to indicate the class of the file.
When a node is selected, the wall
moves to bring the selected node to the center
of the front section.
This is the 3D rooms overview.
Like a rooms overview, it shows all the rooms
that are available.
However, it is also active.
Everything that you can do in a room
you can do from the overview.
[END PLAYBACK]
So we could look at that last image
as essentially a 3D desktop which is tiled.
And of course, you can zoom in and blow up
the window in the same way that you do that on the 2D version.
OK, so switch back here.
So another benefit is that we have many more things
that we can play with, many more special effects.
We have an incredibly larger number of things,
including texture mapping, lighting, and so on.
But that complexity is both a feature and a bug.
It takes a lot more design knowledge and implementation
ability to be able to implement 3D user interfaces.
So let me say now, whatever I said about 3D
is true to the max when we consider
virtual reality and its potential benefits
as well as its current problems.
It's especially good, the claim is, for problems where
I have to make size or precision judgments,
which I'm very comfortable doing in the real world
because my whole system has been trained since birth to do that.
So in particular, if I want to do a walkthrough
through a new building, I can better
simulate that by walking around on a treadmill
or just standing in place and looking around
than looking at a little scale model,
because things will appear more like they will
when the building is finished in that virtual reality.
This is a claim.
No usability studies have been done
to prove that that is true for a mass of people.
But I certainly buy into it.
The Boeing 77 was extensively studied
from a reachability point of view in a VR rig
where you want to know standing in the hall, in the aisle,
what can you reach in the way of a cupboard upstairs?
Or for a maintenance person, how does he or she
have to contort him or herself to get at something?
These are things that you can simulate in a VR trainer
that a desktop simulation is utterly useless for.
It's also good when we want to involve multiple channels,
and that's a hallmark of VR, and do
things that are pretty natural like making small size
adjustments.
And one of the most beautiful examples
of this that I've seen that's actually used in production
is a joint project between University of North Carolina
and one of the California universities,
called the nanoManipulator, where
they are steering a scanning tunneling microscope
and are literally moving atoms around, and feeling the forces,
and seeing things as the scan is being done in real time,
and get to use their proprioceptic system,
their eyes, their muscles, their skeleton, everything
to feel their way around in a trial and error sort of way.
You can imagine that there are a lot of folks
interested in surgical training for much the same reasons.
So I think this is terrific.
Going to show you a raft of examples and then,
having finished motivation, get into some more substance.
So here's an overview of some of the things
I'll show you very quickly.
So this is 3D faked to the max.
This is a beautiful look, but it's standard functionality.
And I think this is as far as we should go.
No more obsessing about 3D interfaces
which aren't really 3D.
Microsoft Bob, enough said.
[LAUGHTER]
Some people actually like it.
I think it would drive me stark raving bonkers
within five minutes.
But it will get better over time.
And it's a very interesting point of departure.
I want to show you Arc Workspace for a moment
because this is a fine example of faked 3D interface.
This is 3D wannabee.
The picture looks 3D, but it isn't 3D.
It's just static 2D.
So switch sources and play.
By the way, feel free to interrupt,
hector, badger, ask questions any time.
It's warm in here, and I'm worried you're all
going to fall asleep on me.
So here is this fake 3D room.
I can get an explanation of what I'm looking at.
I can pick up these tools, the calculator, the drawing
program, the paint brush, and so on.
And, of course, when I need any real information,
you pop up a standard 2D dialog box.
[LAUGHTER]
And you obscure the room, which means
you lose all your spatial memory to a first approximation.
So it's pretty, and it's more appealing, I admit,
for at least the first 15 minutes.
I wouldn't want to use it because it
takes too many pixels, and pixels are scarce.
And I'd rather put more icons on the desk.
But if this moved, and if you could take advantage
of the three, dimensionality that
would be a horse of a different color.
So a little on this I think goes a long way.
I have less than a minute of it.
So here's a project planner that I pulled off the wall.
I can get information in detail.
Here is a file.
I don't know what it is.
Oops.
Still got those problems.
3D interface doesn't save you from that.
And here I can map which icons I want to associate with which
applications.
So a simple, little 3D fake interface.
What makes it fake and not real?
Because I can't move around.
None of the objects are really manipulatable.
They don't have any autonomous behavior.
It's just a different way of portraying 2D icons.
Is it qualitatively different from the cone tree,
for example?
Yes, the cone tree is animated.
It's animated in ways that are responsive to what I'm doing.
Here all you do is you can open things,
and you could put them back on the desktop
and do a minimum amount of rearrangement.
You can change your point of view
arbitrarily in the 3D world in which the cone
tree is embedded.
So it is real 3D.
There's a 3D model, whereas this was just a bitmap.
Another example that I think is finding some significant amount
of commercial use at this point is the Magic Cap interface,
which is not very pretty in the sense
that the previous one was pretty.
It uses a minimum number of pixels.
And I think it's OK as a mnemonic device--
the desk, the library, controls, standard icons.
Here is the same idea of the desktop
with all the usual artifacts.
One of the things that's cute is you can actually
type on this keyboard.
And it's small, but it works.
And you can dial the telephone with an infrared link.
And I think it's OK.
And here are some standard apps that you
can fire up that you can launch by walking into the store.
And I think if this helps people get familiar
with the technology, fine.
Another thing that probably many of you are familiar with
is the Myst phenomenon, gorgeously pre-rendered images
with a very minimum amount of movement
that you can take through there.
It's really kind of a hypercard application.
Or something that has more possibilities, which
is Apple QuickTime VR, which of course has
zero to do with virtual reality, except that they
like the title.
Virtual reality, from my point of view,
is this immersive experience with objects
that are not static, that have autonomous
behaviors, whereas this is just a very clever way
to duplicate the work that the Negroponte group did, again,
in the '70s with the Aspen walkthrough
where they shot a whole bunch of pictures of Aspen
and then let you essentially put them together
quickly one next to the other to simulate
driving through the town.
Here what they do is they take panoramic views
from a number of nodes and then have a very clever image
transformation program that lets you map one frame into the next
and give you a feeling of being in the room.
So let me roll that for you.
I think this is quite a neat technique.
It's a good hack, and I think it will catch on.
Who needs to go?
Feel free to leave any time you need to.
So here we are in the Starship Enterprise and the main control
room.
And the nodes on the right-hand side in that bird's eye view
tell you where all the camera positions were.
And you can sort of teleport yourself
from one node to another.
And you can see you can pan, and zoom, and change
your point of view, zoom in on something,
get a pretty crude bitmap at first.
And then over time, you'll get a better bitmap.
So it has levels of detail.
And it's rendering, which is quite a common technique
these days.
So for static scenes, I think this is a pretty decent way
of doing things.
You can either pre-render or actually take photographs.
It's the same technique inside the program.
Andy.
Yes.
What you can't do here is you can't walk around [INAUDIBLE]
and take a look.
Absolutely not.
There is no occlusion relation.
These are fixed in time.
So this is not 3D, and it's certainly not VR.
But it's a very neat hack that I think has applicability.
So we'll stop that, switch channels,
and talk about actually another development which is
perhaps only a month or so old.
We're not seeing it yet, huh?
Thank you.
Again, uses virtual reality for advertising purposes.
Has nothing to do as yet with virtual reality.
It's essentially being able to share
display files over the network.
Display files are graphics commands
that paint polygons and lines and so on.
And this particular combination, how can it miss?
Virtual reality, World Wide Web, HTML.
It's a winner, right?
Never mind whether it's any good or not.
I think it is just good enough to be above threshold,
and a lot of people are going to have fun with this.
I think it will even find commercial applicability.
Right now, it's not exclusively based on
but takes good advantage of SGI's Open Inventor class
library, which is kind of an object-oriented display list
package, if you will forgive me that shortcut.
So let's go look at a tiny clip of that
so you can see what that looks like.
How many of you have seen it already, just out of curiosity?
Not so many.
Good.
I have something new to show you.
A and play.
So here is a standard look.
And now you see we have this little golf cart
controller at the very bottom here with which I'm rotating,
or I can change my angle of view.
And I hope you notice the texture just popped in.
It took a while to transmit over the net,
but it finally made it.
So levels of detail again, very important to get
real-time response.
Once we're here, we use whatever power of machine
we have to work of this inventor display file.
And this is now running on an SGI in our lab.
And in this radiosity room, we get pretty decent response
time.
So the little doohickey on the right-hand side
of that telescope-like thing changes my pitch.
And then I can change my angle.
And I can also lift myself on the floor.
So it has appropriate viewing controls.
They don't take very long to master.
So this lets me do a share of 3D models across the network.
And that's a good thing.
It's primitive now.
It will get better over time, is the hope.
Back to the slides.
Now I want to talk about some real 3D widgets which
imitate things that I see in the real world,
like my control knobs on radios.
Yes.
What method would [INAUDIBLE]?
Would you do it [INAUDIBLE]?
As I understand it, how would you share this 3D space?
They're still working on sort of unifying the MU
stream of collaboration with this VRML
so that you'll be able to have shared experiences
and be able to have multiple players each controlling
their version of the scene and seeing avatars that represent
other players.
OK, so the world of real widgets,
how do we imitate those?
First of all, what do I mean by 3D user interface?
It's a collection of 3D widget.
What is a widget?
In the X/Motif sense, it has geometry, and it has behavior.
A 3D user interface, in my book, therefore
is not a user interface for 3D applications.
We have many CAD/CAM applications,
for example, that have panels and panels of motif controls.
That's the television model where you separate the user
interface from the world that you're trying to manipulate.
My controls are in the scene itself.
And they are made, as we will see in a moment,
by the same system that makes the application objects.
And I'm going to run through these examples in a moment.
Now, being inside the 3D scene for some people
is achieved already when they're playing a video game like Doom
or Marathon.
They sit in front of the screen.
They are close to it.
They feel surrounded.
And the story sucks them in to the 3D space,
and they feel that they're in it.
They say that they are immersed, even though they really
aren't-- for example, they can't turn their heads and see
what's behind--
but that they feel as if they are there.
And so an important point that I want to make
is that we talk about technological differentiation
between what is real VR and what is not.
But the fact is that, first and foremost,
it's the experience that the user reports.
And Doom comes a lot closer to what we really
want in the future than our traditional computer
applications.
And one of the main reasons that it does
is that it is incredibly responsive.
It generates a hell of a lot of frames per second.
And it does so in a guaranteed way
because they've really optimized for that.
And our normal applications don't do that.
I'll have more to say about that.
So technology and the sensory experience.
And for real VR, as I showed you in the previous clip,
the visual channel wants to have head tracking.
So I really have the feeling that the system is being
responsive to me and that the world is
changing just as the real world does when I turn my head.
It has wide field of view so that I really
feel that I'm inside.
And stereoscopic view for depth perception helps a lot.
Since I walk around in a world that
has auditory sensations brought to me, I like to hear sound.
If balls bounce, I want to hear them.
If things somehow collide, I want to hear that.
So spatially generated sound is important.
And increasingly, haptic or tactile feedback
is becoming important.
The nanoManipulator, I mentioned.
Surgical training obviously demand it.
A lot is going on at MIT and multiple labs.
I saw this afternoon Ken Salisbury's Phantom device.
His thesis is that it's better to have
a limited but very high-resolution
experience, low lag, high resolution
than to have something that, for example, involves lots
of fingers and hands but reports back very crudely
with lots of delay.
And I encourage you to go see this nifty device.
It's quite evocative.
Well, we don't just have virtual reality where you're immersed,
and the computer generates all your sensorium.
I think what will be especially important in the future is
a variation on the theme, which is augmented reality, where
we superimpose the real world and the computer
generated world.
And an example of that is this thought experiment
from University of North Carolina, where
you see the airplane.
You see parts of it exploded.
You see documentation in the hypermedia sort of way.
And the real and the virtual world are combined.
Here's an early experiment in Columbia University
by Steve Feiner, KARMA, knowledge-based system
in which you can see the insides of the copier
as you get directions on how to repair it,
kind of X-ray vision.
And you see that through monocular vision
superimposed on the real world.
We are in this whole field in this stage of the game.
Anybody recognize what this is?
Early '40s Link Trainer.
It looks like a toy, sort of like a soapbox derby
kind of thing.
And it was a toy, but it saved lives
because it was the best kind of training that World War II
pilots could get at the time.
And it sufficed unto the purpose.
Not fancy.
I think we're not even at that stage yet, but we will be,
and you got to keep the faith.
So that was the motivation.
That was supposed to take 20 minutes.
Oh, well.
I'm going to be cutting some.
And any time you have to leave, feel free to walk out.
Having tried to persuade you with some examples
that 3D is coming and that these kinds of immersive interfaces
are on the way, what I want to talk about now
for about 15 minutes is, from an interface designers
perspective, how does this new world feel?
What are the kinds of things we worry about?
And then we'll talk about implementation.
First, the devices suck.
They are completely inadequate unto the task.
They are uncomfortable.
They weigh too much.
They're unhygienic.
They are infinite latency.
They are miserably low resolution.
And we're supposed to perform a multitude of tasks, which
in the real world we do with specialized tool,
with one general purpose tool.
One size does not fit all.
The mouse even for 2D doesn't do it all.
And in 3D, the problem is much worse.
And in 3D VR, it's much more.
I'm not going to talk about this anymore.
Instead, I'm going to talk about the next two points
in some more detail.
How do we do 3D user interface design?
Much the way we do 2D user interface design.
So I'm going to gloss over this very quickly
in the interest of time and let you read it,
and simply keep saying, whatever it was in 2D,
it's more so in 3D.
So we start with task analysis.
What are we trying to do?
Then we have to map from the problem domain
to a particular 3D scenario.
How are we going to go about doing this task?
What are the metaphors we're going to choose?
How much 3D do we in fact want?
You could say, well, put somebody
inside the head-mounted display.
No one has ever worn a head-mounted display
for more than about 10 minutes.
And if you tried to do it for longer than that,
you would get the most amazing muscle and eye and everything
else strains that you could imagine.
Well, these little glasses are a lot better.
But even still, they are not all that comfortable.
So you better have good reason for wanting to be immersed.
And it's easy to say, for intrinsically 3D problems,
we'll go 3D.
For information visualization problems,
looking at large amounts of data,
the case may be a lot harder.
Then we got to find suitable devices.
We got to talk about whether we want
a highly-schematic representation
or something that looks as realistic
as possible for which we have to pay a lot of compute cycles.
Do we want to directly manipulate, reach in there
with our data gloves where we have
to gesture very, well, evocatively
and suggestively but imprecisely?
Or do we want some specialized tools
with limited degrees of freedom that we can control more?
That's a choice you make.
You can design a special purpose widget like the rack widget.
And you, of course, want to involve other channels
as much as possible and learn from what people have learned
from in the past, namely the 2D world and the real 3D world.
You have to learn a lot about perceptive psychology,
as I mentioned.
And then there are all these other motherhood principles.
And I'm not going to take time to elaborate on them,
except to say all these issues are compounded in 3D.
Mapping from devices to tasks to interaction techniques.
And be aware of the lowest common denominator phenomena.
The mouse is in a sense a lowest common denominator.
And as great a device as it is, it
has restricted our ability to do things, even in 2D,
because it isn't perfect for every task.
I'll skip over that point.
Let me show you some more examples
because it's much easier on you than looking at these text
slides.
So what I'm going to show now is some examples
of 3D widgets built with 3D construction tools.
So not the television model.
I'm going to start by showing you
some by now fairly classic widgets that the folks at SGI
did.
We started with those, and then other people looked at them
and said, yeah, those make sense.
And they're by now kind of standard in the industry,
although there are many variations on the theme.
So those arrows indicate that there are textural annotations.
I want to zoom in and put various kinds of tools
on that stop sign.
That was rotation.
This is another kind of rotation.
Here is a translation and a scale box.
I pull at the handles in a moment.
And then I will be able to change
its size all in the scene, no menus, no dialog
boxes, no dials.
[INAUDIBLE] with the mouse or 3D?
This is all done with the mouse, yeah.
They're still very mouse-centric.
And here we're going to type a little textual annotation.
I don't particularly care for those handles.
I think they're kind of intrusive and ugly,
but it's a way of solving a hard problem.
I don't know what a better way of doing it would be.
The T there stands for this is annotated with text.
So translation, rotation, scale, and textual annotation,
everybody has got those.
Next I'm going to show you some examples of our widgets.
We have all of those kinds of things.
But now we have a new idea, which is
application-specific widgets.
So these widgets can be used for essentially any task.
And now what we're going to do is say special purpose tools
for special purpose jobs.
And these widgets again are in the scene.
[VIDEO PLAYBACK]
- We have been developing 3D user interfaces
for exploratory scientific visualization
of computational fluid dynamics data sets.
Built on top of the [? OTIS ?] system,
the 3D widgets presented in this video
provide fully interactive direct manipulation techniques
for placing and modifying parameters of commonly
used visualization techniques.
To demonstrate the 3D widgets we have developed,
we will be using a data set computer
at NASA Ames of airflow past the body of the space
shuttle in which the velocity is computed
at each point on a curvilinear grid.
We have implemented a set of probes
which generate tufts, streamlines, cutting planes,
and isosurfaces at their sample points
and provide interfaces for modifying
some relevant parameters.
First, we show a vector probe which samples the vector field
at a point.
One visualization technique which
can be attached to a probe is a streamline.
The path is colored by velocity from blue to red
with increasing speed.
We can transform the point probe into a line, otherwise known
as a rake, which generates a set of streamlines
along its length.
We can position the rake, rotate it, scale it,
or change the number of streamlines
on the rake using simple 3D widgets.
This could be extended to multidimensional
isosurface probes as well, though the usefulness
of this technique is uncertain.
We have implemented a number of different interfaces
for specifying the position of visualization tools.
As we have seen, screen-aligned translation
is simple but doesn't allow one to translate objects out
of the screen plane.
The shadow on the floor plane is a useful depth cue
and a widget in its own right.
Translating an object's interactive shadow widget
also translates the object itself in the plane parallel
to the shadow plane.
Using this technique, we can move the streamline quickly
to the tip of the wing.
The shadow widgets have some disadvantages, however,
when there are large amounts of geometry in the scene.
Here the shadow body sometimes obscures the streamlined shadow
on the right wall.
Another technique called object handles provides
an alternative direct manipulation interface
to constrain translation using significantly less geometry
than the interactive shadows.
Using this technique, we do not have
to worry as much about important widgets becoming obscured
by other geometry in the scene.
If a selected object is visible, the handles
will also be visible.
Here we see a slight variation of the object handles widget.
Instead of becoming aligned with the Cartesian coordinate
system, these handles display the basis of the curvilinear
computation grid at the point closest to the selected object.
By translating the selected object,
we can see the local structure of the grid.
Because the grid is non-linear, the handles
are not straight lines.
Using this technique, it is straightforward to translate
the streamline along any of the lines in the grid,
including the line which corresponds
to the [INAUDIBLE]--
[END PLAYBACK]
OK, next thing I'm going to show you
is an experiment to try to make use of a 6 degree
of freedom flying mouse from Logitech.
[VIDEO PLAYBACK]
--introduce a new user interface metaphor
for immersive VR, the virtual tricorder.
The virtual tricorder offers a visual and tactile interface
that is programmable engineer form and that
takes advantage of the user's familiarity
with real-world tools.
It visually duplicates the six degrees of freedom input device
such is the Logitech flying mouse
and met all translations and rotations of the input device
one to one to the visual model.
Desktop setup is for illustration
only since the virtual tricorder applies strictly
to immersive VR.
Effects-based boom services are a head-mounted display.
We demonstrate four interaction techniques for the tricorder.
First, anchored menus or 2D menus
anchored to the front of the tricorder.
To see the menu better, the user pulls the tricorder closer
to his face.
By pressing the labeled buttons, he chooses and selects items.
Second, zero-gravity grappling lets the user
grab arbitrary points in space and push or pull himself
away or towards them.
So this is like the hand in 2D
- To move more than an arm's length,
he uses the ratcheting motion or throws himself
in the design direction.
He then uses a built-in jet thruster
to correct the resulting flight path.
[LAUGHTER]
Thus, he smoothly integrated the scene
at hand applying [INAUDIBLE] aviation techniques.
Third, we demonstrate selecting and positioning objects.
An object selection with a semi-transparent cone
projects from the front of the tricorder.
To each selection, the user adjusts its width
interactively.
The object it highlights here, a built-in spotlight
at the cone's apex is selected when the user
presses the Select button.
Once an object is selected, the user
transmits it by pressing the right-most button
and transmitting the tricorder.
Or he rotates the object by pressing the middle button.
The tricorder's orientation is transmitted
to modify the object's orientation.
Fourth and finally, we use the tricorder
as a visualization tool.
A wireframe lens renders the scene in wireframe mode.
By controlling eye to lens distance,
the user also controls how much of the scene
is displayed through the lens and thus rendered
in wireframe mode.
The magnifying glass magnifies the scene.
Once more, the user controls how much of the scene
is magnified by varying the eye to lens distance.
The user can also interactively change the glass magnification
factor.
And he can even change the tool into a magnifying lens.
[END PLAYBACK]
OK, I think that's a child's garden of widgets.
Let me summarize the design issues before briefly moving on
to software.
So I said 3D design space is hugely larger than 2D
and much, much less well explored.
And there's a much higher learning curve
for getting into the game.
Essentially, we're faced with still creating the design
discipline.
When PARC started in the early '70S,
there was no design discipline for bitmap graphics-based GUIs.
And they invented it.
It was polished by Apple and then polished some more
by other folks.
That's what we need to do in 3D.
So now I'm going to talk very briefly about software issues.
I know that 5 o'clock is essentially the drop dead time.
And I'll work to that.
This is the list of things I'm going to cover ever so quickly.
So let me get to it.
First of all, the development methodology, I claim,
will be quite different for 3D than in 2D.
In 2D, we still use a kind of software engineering waterfall
model where we can go pretty much top down
through all the standard phases.
In 3D, we don't know enough to be able to do that yet.
And so it's much easier to have somebody quickly prototype
something and say, "Here's what I like.
Here's what I don't like," and play with it
than to go through a specification
and see whether you're going to like it.
So this says, it's essential to have a rapid prototyping
environment which has an interpreted or visual language
or both.
So we're going to sit in this design loop refining it.
And some of those rack widgets had 20 or 30 variations
before we were happy with them.
And some of them took as little as half an hour to do.
Why is that?
Because we are building the widgets
in exactly the same environment as which
we build the application objects.
Uses the same system.
We don't have this problem of having
the motif hierarchy over here and the application
hierarchy over here and trying to link them
through callback registration.
So I want to reset the clock.
I want to blur the distinction between what is user interface
and application and take advantage
of the semantic feedback that we get only from the application.
For the programmer, this means things are easier to learn
and easier to use because it's the same machinery.
And we can do faster prototyping.
We also can get higher bandwidth communication
between the parts that are user interface and the parts that
are application object, because, in fact, there
is no distinction.
You don't have this barrier of the mass of spaghetti
that the callbacks generate.
Now, that is not to say that user interface design is
the same as application design.
We still need that design discipline,
and modularity is still useful.
We have barely begun to understand, particularly
we graphics bigots, how to integrate the other media
into our graphics presentation.
Most of the time when you see things,
you're only seeing graphics channels.
But when you have bouncing balls or Peedy the talking parrot,
you have to worry deeply about sound synchronization,
beak syncing as they called it at Microsoft.
And that's difficult, and there are remarkably few tools
for doing that.
This brings me to a topic that we're
studying in my research group.
I have a PhD dissertation in progress
on this, something that we've been calling
time-critical computing.
In normal computing, you design algorithms.
And the outcome is you have a correct algorithm.
It produces the results that you want.
In virtual reality, in particular,
a result on time given according to certain time guarantees
is worth more than a better answer, but too late.
Why?
Well, for one thing, you're going to get motion sick.
If latency is too high and if you don't have adequate refresh
rate, the illusion is going to break.
Very unpleasant psychophysical effects
if you don't deliver in time.
So timeliness in this domain is more important than quality.
This involves research at two different levels,
operating system research.
I was talking to Professor [? Castle ?] earlier today
about his risk style operating system in which it should
be a lot easier to design a soft real time
library for the kinds of effects that we need here.
But we also need to turn algorithm research on its head
and say, let's have classes of what the AI folks sometimes
call any time or interruptible algorithms.
Well, you tell the algorithm how much time it has,
and it computes to that time.
Or conversely, it's always got partial results
that are meaningful.
And the more time you give it, the better the result.
In rendering, 3D rendering, that's
called progressive refinement.
We had another PhD dissertation for collision detection
where it used a three or actually
four-dimensional computational geometry problem
to put bounds around objects that
were going to collide, and look for potential intersections
and very rapidly give a approximate answer.
And then given more time, we'd give you a more precise answer.
Plenty good enough for government work,
and much better than taking non-real time amounts of time
to get the right answer.
So we can do these kinds of interruptible algorithms
only if we are informed about what makes sense
for a particular task in terms of the perceptual cues
that we need to perform the task.
Computer scientists don't talk about tasks.
They talk about algorithms.
In this game, you've got to talk about tasks,
you've got to talk about users, you've
got to talk about perceptual cues.
Are shadows important or not?
Is beautifully photo realistically rendered
essential or not?
Or could you make do with wireframe or faceted shading?
How accurate does the simulation have to be?
If you want to see a pool table simulation,
must the balls move smoothly, or must you
be guaranteed that at certain time, collisions happen?
And in between, you could stand some amount of jerkiness?
It's all a question of the perceptions and the tasks.
And we have to become sensitive to that.
Another major problem is that we're trying to build these
systems in antiquated 1970 styles programming language,
C or C++.
Devising algorithms in this new world is hard enough.
And we don't really have the tools
at our disposal to think of the new algorithms in the new ways.
And I claim it's kind of a Whorfian hypothesis.
We haven't been exposed to these better languages.
The actor model is about the only thing
that I saw many years ago that kind of broke the paradigm
and got us into a new way of thinking.
What do we need in this new world?
Well, first of all, again, not a solitary activity,
multiple participants, what programming language
knows for multiple participants, an active world with autonomous
objects whose behavior is time-varying,
lots of channels coming in simultaneously
and continuously.
And no programming language knows anything
about any of these concepts.
In particular, the object models are broken.
C++ style static class instance kind of model does not work
because our objects have to be able to change their behavior
over time, to acquire new traits.
So we are much more impressed with a prototype delegation
kind of model which allows for dynamic behavior.
We need multi-methods.
It's not reasonable to say all of the behavior of an object
is the property of that object.
It's the way objects interact that
determines the behavior of the individual interacting objects.
Again, most programming languages, even
object languages, don't support that.
And multi-methods is one way of getting the message
to be in a sense decoded by the multiple objects involved
in sending and receiving that message.
Distributed processing.
MIT is one of the hotbeds of this.
And they've certainly invented languages,
but no language in common use knows
anything about distribution.
That's left to libraries.
Concurrency, synchronization, and so on.
So we have lots of needs.
But if we throw them all together
into a single object-oriented language,
we'll wind up once again with something like PL/I.
So we can't have a kitchen sink approach.
And I have another PhD student working on a very simple object
models that can be grown to have the properties that you need,
which might be application-dependent.
The system we currently work in is implemented in C and C++.
It supports a LISP-ish style interpretive language.
And it also has an interaction tool kit
with which we can do some amount of visual programming
in a kind of dataflow-ish way.
And I want to finish up with that.
Essentially what we noticed is that there
are a bunch of things that get recycled over and over again--
points, lines, planes, polygons, cylinders,
and so on, and constrained behaviors.
Things slide relative to each other.
Things rotate relative to other things.
So affine transformations come up a lot.
So we said, let's encapsulate those simple behaviors
and the geometry and try to program this stuff visually.
So I'm going to show you how we set up
one-way dependencies into multi-way constraint networks.
And then our system continuously does constraint maintenance
in some sense.
And you will see that it isn't all that peppy.
The system isn't good enough, and I
would say it was a decent first effort and no more.
So let me show that to you now.
Actually, I need to switch tapes.
Let's see.
I think I have to use the thing to eject.
Now I don't know where I am on this tape
because this is the one that was rewound.
So we're going to go through the same nonsense, I'm afraid,
that we did earlier.
[VIDEO PLAYBACK]
- [INAUDIBLE] widgets and interactive illustrations.
Oh, perfect.
- [INAUDIBLE] provides [INAUDIBLE] with the direct
manipulation of 3D primitives--
This is the first [INAUDIBLE]--
--which can be linked through a visual interface
to create objects with complex constrained behavior.
All demonstrations in this video were recorded in real time.
The basic tool kit primitives are
based on a coordinate system metaphor of points, vectors,
planes, and 3D volumes.
Each primitive has a set of characteristic traits called
slots that define the primitive's
interactive behavior.
The point primitive represented by the small sphere
has one slot, its position slot.
The vector primitive represented by the arrow
has three slots for its position, direction,
and length.
The plane primitive represented by flat sheet
has slots for its normal orientation, center, and scale.
3D volume primitives, like this cube,
have slots similar to the plane primitive,
but also contains slots for scaling in the third dimension
and for non-geometric attributes such as color, transparency,
and visibility.
To build up objects with complex interactive behavior,
we specified persistent relationships
between a slot of one primitive and one or more slots
of another primitive.
We call this process linking.
The constraints imposed by the linking process
are established and maintained by the toolkit.
For example, to link the vector's position slot
to a point, we first select the vector
and then select the point.
The pending length is then illustrated by the large arrow.
The motif window shows us the available slots
of the vector and the possible linking behaviors.
The tool kit selects a slot and a behavior
by default. In this case, the default slot
is the position slot.
And the default behavior positions the base
of the vector at the point.
We confirm the pending link by clicking on the glue bottle.
The constraint is established, and the vector's position slot
is linked to the point.
This link is bidirectional.
We can translate the point, and the vector follows,
or we translate the vector, and the point moves.
The other slots of the vector remain
free to be manipulated or constrained.
We can continue this construction
to build up the behavior of a line segment
by first linking the vector's directional slot
to a second point.
Since the position slot is already constrained,
the tool kit selects the direction slot
by default. Notice the vector always points from the first
to the second point.
Next, we link the vector's length slot
to the same second point.
Now the vector resizes in order to span between the two points.
To constrain a point to lie along this line,
we link a point's position slot to the vector.
Notice there are several options presented in the menu
for the linking operation.
The default linking behavior constrains the point simply
to lie along the line.
If we decide we're unhappy with this behavior,
we can undo it and apply a new constraint.
By clicking on the point while holding the Shift key,
we remove the [INAUDIBLE]--
I'll just fast forward and go to an application.
This shows encapsulation of the--
- We are modeling the portion of a bicycle
where the head tube is connected to the rest of the bicycle
frame.
We want to make the joints between these two smooth.
[INAUDIBLE]--
This is an application.
-- to investigate this particular problem came from
a bicycle designer.
We want to design a blend region that joins the tubes smoothly
and attractively.
The specification of the blend range in between two
of the tubes is somewhat complex.
The boundaries of the blend region
must be specified as curves on the tube services to be joined.
We see two such curves, known as rail curves,
shown here in white.
We examine the two blend regions together
and see that the result does not have a nice shape.
Instead of the flat area on the head
tube between these regions, we would like a smooth shape
that unifies both blends.
To accomplish this, we will alter the head tube
by warping it.
This will create a smooth saddle between the blends.
We use a previously constructed encapsulated warp tool
to accomplish this operation.
The warp tool consists of a center of warp, direction
and magnitude of warp, and a sphere of influence.
Because of the required symmetry of the construction,
we need to be able to constrain the warp interaction.
To do this, we constrain the center of the warp
to a previously constructed access line
through the center of the head tube.
Then we constrain the direction of the warp tool
to lie in a plane that contains the center lines of each
of the bicycle tubes.
Once the warp tool is in approximately
the right position, we use the toolkit to link the head tube
to the warp tool.
We can interactively warp the head tube
by adjusting the direction, center, and radius of the warp
with the parameters encapsulated in the warp tool.
Once the desired warp is achieved,
we freeze the warp by unlinking the surface from the warp tool.
[END PLAYBACK]
So that was an exercise that used
Utah Alpha 1 geometric modeler and our widgets.
And that was all real time that you saw.
It's still a toy exercise.
It's not ready for use by a production designer.
But it was the first time that the Utah folks
had more than an Emacs interface to their parameters.
It was the first time that they could
see the effect of various kinds of parameters
on the actual surface that they were warping.
So let me wind up here now.
My last slide.
3D is happening.
3D is going to be here sooner than we think.
It's going to drown what we've seen in the last 20 years
in terms of the slow, steady rise of the WIMP interface
because it's going to be driven not by research labs the way
the first revolution in user interface design
was, but by commodity folks such as the game
people, the interactive television people.
And we research lab folks still have plenty of work to do.
But we're going to be trying hard to follow the wave.
We won't be leading it.
The current technology is wholly insufficient unto the task.
It's not fast enough.
We don't have control over it.
It isn't time-critical.
The hardware doesn't work.
There are lots of things in a university environment
for people to pay attention to, not to mention commoditizing it
after we find out how to do it.
3D is still very much a solution looking for a problem,
except in the games area.
We have this implicit belief that it's
going to be wonderful.
Some of us, who have been in the 3D interactive graphics
game all of our lives, have been saying, it's coming.
It's coming any time now.
And '95 is the time when it's really coming.
But what we can't answer, except for intrinsically 3D problems,
what is it good for?
Is it cost effective and so on?
But it's going to happen because it looks good,
it makes people feel good in the same way
that you couldn't justify color.
And all the people trying to predict how fast color would
come up were wrong.
Black and white is sufficient unto the purpose,
but people want color in the same way
people will want 3D because of the glitz factor,
if nothing else.
And it's only because we will design the right kinds of 3D
user interfaces that they will actually say,
and it's useful for solving problems.
It's terra incognito, and I invite
you to join us on this quest.
Thank you.
[APPLAUSE]
As John remembers, I'm unfortunately
capable of staying forever.
So you leave, because I'm not going to be leaving.
I'm happy to stick around and answer questions.
Yeah.
On your conclusion slide, you said that the hardware
isn't fast enough.
What's your prognosis [INAUDIBLE]??
Let's wait for people to clear.
Mike.
I want to fix the screen.
Yeah.
Good animation, right?
No.
It works in my PowerBook.
It's the connection to the video.
What's your schedule?
I'm going to dinner with Michael and stuff.
Finally I made it here.
Come along.
[INTERPOSING VOICES]
It was so sunny outside that, when I came in,
I was totally blind.
I couldn't [INAUDIBLE].
I was stumbling upstairs.
OK, [? Miguel, ?] you get the first question.
OK.
On your last slide, you mentioned that the hardware's
not fast enough.
So my question is, what's your prognosis on that?
Is it ever going to be fast enough?
OK, I'll repeat the question.
Will the hardware ever be good enough?
Well, the standard academic's answer is of course not.
But you get certain threshold at which
a certain amount of adequacy is clearly recognized.
For example, I think for ordinary 2D
desktop applications, hardware to a first approximation
is good enough.
The mouse works well enough, the screen
is good enough, to CPU power, the amount of memory.
You always want more, but you can get useful work
done with what you have now.
That is not yet the case, except for the half a million
to a million dollar acquisition at the high end of 3D graphics,
let alone for virtual reality.
And for virtual reality, you need all of that processing
power, you need all of the rendering power,
and you need far better device technology than we have today.
There are people working on it, and things will get better.
But don't expect any miracles in a year or so.
Henry Fuchs, my colleague at University of North Carolina,
dreams of flip down sunglasses, that light weight
and with high resolution, with which you then see either
VR or augmented reality.
And he has his hands on some very good LCDs
that they use in view finder cameras.
They're very small.
They can be viewfinders of video cameras.
They're very small.
And he believes that with that form factor,
within a few years, we'll have these kinds
of flip down sunglasses.
And that will help a lot.
Doesn't help with haptic feedback, which
is actively being worked on both at UNC and in this University.
Sound recognition is still a bad problem,
although again this place is a world
center in that particular research area.
In time, that will come as will spatially-located sound.
We have early examples.
And for small, selected demo applications, it's good enough.
So I think by the turn of the century,
things will look a lot different.
It will move a lot faster than 2D graphics moved.
That took a long time to take hold.
It wasn't until Macintosh, which was more than 10 years
after Xerox demonstrated the concept amply and had
commercial products in the marketplace.
Paul, do you want to tell us why it's not
the fault of the LSI designers?
Oh, you'll get all the hardware support you need.
Just wait a couple years.
My question is you've drawn a distinction between 2D
interfaces--
and you call them 3D, but I call them
greater than or equal to 3D.
And in particular, some of these involve
time in an essential way, and some did not.
Is there a useful distinction that
can be made between truly 3D and 4D?
We've often called our system a 4D modeler.
Did everybody hear the question?
3D plus time, because in our descriptions,
time is a first-class citizen.
So any attribute can be time-varying.
That includes geometry.
That includes behaviors.
That includes constrained dependencies, any of these.
So I would say our system is a 4D system.
Now, even if you only have a 3D description, if you go in there
and you do the editing by hand, you
can get it to change over time.
And in any 3D system, you at least
have the time dependency that you
get from navigating through the scene
in a three-dimensional way and shifting your point of view.
The camera is always traveling over time.
So--
I think you've weaseled out of the question.
Possibly.
I'm an academic.
And with all [INAUDIBLE],, a 2D interface, which
is able to change under your direction,
and you mouse over it, you would almost call a 3D one.
And fine.
If yours really supports time as a first-class object,
isn't there an intermediate stage,
which doesn't, which is actually three-dimensional,
but which is therefore much simpler
and might be further in, further closer to realization
and might be useful?
Yeah, sure.
If you take a standard 3D graphics library,
and it does not have support for time,
but it does support viewing controls
and does support your editing of that description
so that you can by hand animate things, that's a way station.
That's 3D plus by hand support for time variance.
And that's what we've used all these years
to do our 3D graphics.
So it's a well-known way of doing things.
And you don't see 2D time-varying interfaces.
About the only thing you do is, when a window expands,
you see a little, tiny animation.
And when it contracts, it does.
Very little use has been made of animation for 2D interfaces.
People haven't been particularly creative there.
But for 3D, since you're constantly
moving your point of view, it's more natural.
And you tend to have more autonomous objects
in these kinds of applications.
Your 2D desktop applications, the entities
basically sit there on the screen and are static.
In 3D, much less so.
All right, take one more question.
Do I have someone who wants to ask the final question?
[INAUDIBLE]
Last week, Silicon Graphics had the demonstration of WebFORCE.
And they mentioned quickly that they either have available
or will put out shortly, in a week
or two, 3D on the internet.
And I'm not sure what that they meant.
Do you have an idea?
VRML, I'm quite sure.
So essentially being able to send inventor files
over the internet.
They also accept other kinds of file formats.
And that's a perfectly reasonable way station.
Nobody argues that it's the best,
but it's a de facto standard.
Many--
Do you see it having a good commercial application?
Yeah, I think people will be building applications
on top of VRML because it combines various features
that they're used to in the 2D world
with 3D geometry description.
So, yeah, I think we'll see some use of that.
The fact is that the WWW community now
has the power to set standards.
When they say, we subscribe to XYZ--
I won't say X because that's already overloaded--
when they say, XYZ, people stop questioning
whether it's any good or not.
They don't argue.
They say, fine, we'll write to that de facto standard.
So that's why I think VRML is going to take off.
It has some amount of papal blessing bestowed on it.
[APPLAUSE]