Figure 1. Alexa with electrodes, ready to use EagleEyes.
Figure 2. EagleEyes system diagram.
Introduction
EagleEyes [1] is a new technology that allows a person to control the computer simply by moving his or her eyes or head. The technology is based on the measurement of the EOG, or electro-oculographic potential, through electrodes placed on the head. For the past year we have been working with people with severe disabilities to try out the system and to teach them how to use the system to control the computer. The current system runs on a Macintosh and allows us to run both custom-developed software and most commercial software. Through EagleEyes, people can run educational and entertainment software, spell out messages, and navigate through the internet just by moving their eyes.
Table of Contents
There are several approaches to sensing eye movements. [2]
Most approaches involve the use of a camera or imaging system to visually track some feature of the eye and then a computer or some electronics to do the reverse geometry to determine where the user is looking. Several systems use of infrared illumination and an infrared-sensitive video camera. The ISCAN system [3] uses a lightweight head-mounted infrared-sensitive camera to track eye movements. The Erica system [4, 5] uses an infrared-sensitive camera mounted below the computer monitor. Canon has developed a miniaturized system that allows some of its cameras (for example, the EOS A2E and the newer EOS ELAN IIE) to sense where the user is looking so the camera can autofocus along that sight line [6]. There also is work on systems that use visual light, rather than infrared. Pomerleau and Baluja [7] reported using a neural network to process ambient light video camera images of the eye to determine where the user is looking.
The approach we are taking is to attach electrodes next to the eyes and use them to sense the EOG or electro-oculographic potential. The EOG is also known as the ENG or electronystagmographic potential. The EOG / ENG has been investigated for over 70 years [8,9]. Currently the major use of the EOG / ENG is in diagnosing vestibular and balance problems [10]. A similar approach to ours is taken in the EyeCon / Biomuse system [11].
With the EagleEyes system, five electrodes are placed on the head of the user. An electrode is placed 1 cm. above the right eyebrow and another 2 cm. below the right eye. Electrodes are placed 2 cm. to the right and left of the outer canthi. A fifth electrode is placed on the user's forehead or ear to serve as a ground. (See Figure 1) The leads from these electrodes are connected to two Grass Instruments 7P122 differential amplifiers, which amplify the signals by a factor of 5,000. The amplifier outputs are connected to an analog-to-digital converter on a National Instruments signal acquisition board in a Macintosh computer. A system diagram is given in Figure 2.
|
Figure 1. Alexa with electrodes, ready to use EagleEyes. |
|
Figure 2. EagleEyes system diagram. |
Custom software interprets the two signals and translates them into cursor coordinates on the computer screen. The electrodes placed around the eyes measure the electro-oculographic potential, the variation in the standing potential between the retina and the cornea. The EOG is a function of the position of the eye relative to the head. Basically, the difference between the voltages of the electrodes above and below the eye indicates the vertical position of the eye relative to the head. The difference between the voltages of the electrodes to the left and right of the eyes indicates the horizontal position of the eye relative to the head. Move your eyes up and the vertical signal increases. Move your eyes to the left and the horizontal signal increases. The EOG signal changes approximately 20 microvolts for each degree of eye movement. In our system the signals are sampled 60 times per second.
The EagleEyes system software allows us to run EagleEyes with most existing commercial software. Our system software runs in the background. Every 1/60th of a second it springs to life, senses the two values on the A/D converter, translates the values into screen coordinates, and saves them as the official mouse coordinates in the system. An option in the software allows a mouse click to be generated whenever the cursor remains within a settable small radius on the screen for a certain period of time. That is, the user can generate a click by staring at a spot on the screen for a certain fraction of a second. The software can run in the background with any well-behaved application. Thus, Macintosh software, whether commercial or custom-developed, can be run by eye control instead of mouse control.
We currently have three EagleEyes systems, two in our laboratories and one in the Campus School. The Campus School is a day-time educational facility for students, aged three to twenty-two, who have multiple impairments. It is part of the School of Education at Boston College and is located on the main campus. The EagleEyes facility at the Campus School is reserved for the Campus School students in the morning and for visitors and mainstreamed students from the greater Boston area in the afternoon.
The EagleEyes system tracks the EOG, which is proportional to the angle of the eye in the head. Learning to use the EagleEyes system is an acquired skill.
A person without disabilities usually requires about 15 minutes to learn to use the system and to become proficient enough to spell out her name using a board like the one in Figure 3.
|
Figure 3. Spelling out a message. |
For a new user we usually explain that the system is measuring the angle of the eye in the head and that the cursor can be moved either by holding the head constant and moving her eyes or by fixing her eyes on a spot in the screen and by moving her head or by some combination of the two. New users practice moving a cursor on a blank screen and then play a simple video game we have developed for training. When a user is proficient enough to shoot down 9 out of 10 aliens in the video game, then the user is proficient enough to be able to spell and use the other software.
Dozens of people with disabilities, mostly quite severe, have tried the system. They can require anywhere from 15 minutes to months to gain proficiency with the system. We currently have about a dozen profoundly disabled youngsters using the EagleEyes system on a regular basis. [12, 13].
A major challenge has been the design of the human-computer interface. That is, given the capabilities of people to control their eyes and head and given the physical characteristics of the EagleEyes system, the amplifiers and so forth, how should the software be designed so that it is easy for people to use ? Jacob [14, 15] points out important potential benefits and problems of using eye movements to control computers. For example, he discusses the "Midas Touch problem":
| "At first, it is empowering to be able simply to look at what you want and have it happen, rather than having to look at it (as you would anyway) and then point and click it with the mouse or otherwise issue a command. Before long, though, it becomes like the Midas Touch. Everywhere you look, another command is activated; you cannot look anywhere without issuing a command. The challenge in building a useful eye tracker interface is to avoid the Midas Touch problem. [14, p. 156] |
Starker and Bolt [16] describe an information display system that makes real-time inferences, using an eye-tracking system, about a user's interests and modifies the display, for example by zooming in on an item inferred to be of interest. Nielsen [17] discusses the trend towards "noncommand user interfaces" and the place of eye-tracking in these systems.
Generally the software used with EagleEyes must be controllable by large buttons or clickable areas. The basic issue is accuracy and control. With EagleEyes, the user can move the cursor with fair accuracy and can issue a single mouse click. We have not devised an effective way for the user to simulate holding down the mouse button or issuing a double click. So, for example, the EagleEyes user cannot yet control pull-down menus.
We have developed and used several types of software with the system.
EagleEyes users are run on a variety of different softwares which can be broken down into the following categories:
Most of the people we are working with are unable to speak. We would like to provide them with the facility to use EagleEyes to communicate with people. We have used a number of different approaches. Some of the people know the alphabet and how to spell. Some do not.
We have developed two "spell and speak" programs. The first simply puts a picture of a keyboard on the screen, as in Figure 3. The person spells by keeping her gaze on the desired letter for a fraction of a second. That letter is then selected and placed in the message area at the bottom of the screen. The back arrow serves as a delete key. When the user selects "end" the message is spoken by the computer. People who have mastered EagleEyes can spell using this program at the rate of about one character per second. For people who do not have the control required for the keyboard we have developed a two level spell and speak program. Here, the user selects from a group of five (or six) letters as in Figure 4. The message selected so far appears in the middle of the screen. If the user selects "ABCDE" then the screen in figure 5 appears and the user selects the desired letter from the group. The program then changes back to the first screen and the process is repeated. The user can select the "speak" button at any time and have the computer speak the message. Each button of letters is a different color on the first screen. That color is repeated in the corresponding letter buttons in the second screen.
Figure 4. The first screen of the two-level speller.
Figure 5. Pres the "ABCDE" button on this screen to select the letter.
Note that the size of the buttons or target squares are about the same in the two spell and speak systems. The problem with the keyboard in Figure 3 is precisely the Midas Touch Problem. Wherever the user looks, a key is selected, so the user has to be careful not to hold her gaze at any location unless she wants to select that key. The second program provides plenty of dead space, plenty of area where nothing happens if the cursor stays at the location and a click is issued.
We also have developed some specialized communications software. For example in the program illustrated in Figure 6 the user selects a category on the first screen and then a specific message on the following screen. The user looks at one of the squares for a fraction of a second and the square is selected. The result might be another screen of five choices within the category selected.
Figure 6. A specialized communication program.
These systems can be customized for individual users. Looking at a picture of Mom in one of the squares might result in a voice saying "Mom, could you come here?" Looking at the picture of a favorite athlete or movie star might result in a digitized video clip of the person being played on the computer. The selection squares are made large so they are easier to keep the cursor in.
EagleEyes works well with commercial software, such as "Speaking Dynamically" and "Boardmaker", for augmented communication.
We have developed a general text reading program and a general multiple choice program for use with EagleEyes. The text reading program allows text to be entered ahead of time. The user then has buttons to control the reading of the text. The general multiple choice program allows a series of questions and four possible answers to the questions to be entered. Then the user can go through the questions and select the answers. These programs are being used by a thirteen year old young man with quadriplegic cerebral palsy who is in the sixth grade in his public school. He comes to the facility two or three times per week to use the EagleEyes system. His mother and teachers enter lessons and homework problems for the young man to do during his EagleEyes sessions. We are hoping to be able to provide him with his own personal EagleEyes system to use at school and at home.
We also have used several commercially available educational software packages with EagleEyes. The Living Books series ("Just Grandma and Me", "Arthur's Teacher Trouble", etc.) are very popular with the students using the EagleEyes system and work very well with the system. These CDs fall somewhere between educational and entertainment software. Similarly, "The Lion King" CD is very popular.
Eye Love Movies
Figure 7. Opening screen of "Eye Like Movies."
Figure 8. "Eye Like Movies" scenes from Beauty and the Beast.
For a feature on the Disney Channel on our work we implemented a system called "Eye Like Movies" that allows for the selection of Disney movie clips by eye control. The initial screen, shown in Figure 7, allows the user to select Aladdin by looking at the left part of the screen and Beauty and the Beast by looking at the right part of the screen.
Selecting Beauty and the Beast leads to the screen shown in Figure 8. Looking at any of these squares for a fraction of a second causes the associated film clip to play. (In all of these systems, the fraction of a second required for selection is adjustable.) This system, as with the others described, is implemented for the mouse and can be run under mouse control. However, running the software in conjunction with EagleEyes allows the user to control the system by eye movement rather than by using the mouse.
EyeVenture
Figure 9. Opening scene from "EyeVenture."
Figure 10. Look at the window in "EyeVenture..."
Figure 11. and see a storm.
"EyeVenture" is an experimental game designed for the general population that also was implemented for the Disney Channel feature. The game combines graphics, sound, and digitized video clips from Disney movies. The opening screen of EyeVenture is shown in Figure 9.
The user is placed in a simulated environment, as shown in Figure 10. Looking to the left causes the user to move to the left in the environment. Looking to the right causes the user to move to the right.
Looking at an object in the environment can cause an event to occur, for example a digitized video clip to be played or a voice to give instructions or a clue. For example looking at the window in Figure 10 causes a video to be played, as shown in Figure 11. By exploring, looking in the right places, listening, and solving the puzzles the user can move through the adventure.
Eye Painting
Figure 12. Finger painting with your eyes.
"Eye Painting" is a program we have developed that is very popular with younger users and their parents. It is the equivalent of finger painting with the eyes on the computer screen. Wherever the user looks, colored lines are drawn. At the end of a session, we print out the EyePaintings on a color printer and give them to their parents to hang up on the refrigerator or to put in a frame on the wall. A sample EyePainting is shown in Figure 12. A youngster creating an eye painting is shown in Figure 13.
Figure 13. Eye painting with EagleEyes.
EagleEyes works well with many shareware and commercial games.
EagleEyes also works with web browsers like Netscape. Thus, a user can roam the internet using just her eyes to click connections. For a 55 year old former CEO who is completely paralyzed from a stroke, we have set up a home page with connections to sites of interest, such as Sports Illustrated and CNN.
We are in the process of developing a miniaturized version of EagleEyes. We expect this version will work with notebook computers and be a totally portable system. We hope to be able to provide our students with their own EagleEyes system to use at home, in school and wherever else they travel.
References
Back to Papers
