The Evolution of User Interfaces • Ken Pfeuffer • GOTO 2025

Ken Pfeuffer discusses the evolution of user interfaces, from traditional hand-centric inputs to the emerging paradigm of eye-hand symbiosis. He traces the history of gaze interaction research, explains his decade-long work on "gaze and pinch," and shares how these concepts are now becoming reality with devices like Apple Vision Pro, outlining future directions for intuitive spatial computing.

June 24, 2026 7 min read

Ken Pfeuffer, a researcher in Human-Computer Interaction (HCI), delves into the profound evolution of user interfaces, tracing their history from rudimentary tools to the sophisticated spatial computing of today. His presentation focuses on the concept of Eye-Hand Symbiosis, a decade-long research quest exploring the natural fusion of eyes and hands for digital interaction.

The Evolution of User Interfaces

Pfeuffer begins by illustrating the fundamental human ability for eye-hand coordination with a personal anecdote: assembling a bed. He observed how his eyes precisely fixated on target points, guiding his hands, which then executed actions often without direct visual monitoring. This instinctive behavior forms the bedrock of his research into intuitive digital interaction.

The discussion then shifts to the current landscape, introducing the Apple Vision Pro. While still a high-end, non-consumer device, it represents a pivotal moment due to its primary reliance on eye-tracking for navigation and pinch gestures for input. In this paradigm, eye movement functions as the “mouse movement,” and a subtle pinch acts as a “mouse click.” This marks a significant shift in user interface design, as it’s the first consumer-oriented device to foreground eye-tracking in its core interaction model.

Historically, human tools were always hand-optimized. The digital era introduced devices like keyboards, mice, and styluses, similarly tailored for hand manipulation. While eyes provide visual attention, they traditionally haven’t been direct manipulators of digital content. The trend, however, is towards more “natural” interfaces, leveraging three-dimensional space, with Extended Reality (XR) aiming to integrate digital content into our physical world.

Pfeuffer highlights key paradigm shifts, notably the transition from the mouse to the touchscreen, exemplified by the famous 2007 debate between Steve Jobs (advocating for multi-touch, rejecting styluses) and Steve Ballmer (insisting on physical keyboards for business productivity). Jobs’ approach, characterized by a radical, risky, and novel reimagining of interaction, ultimately proved successful in the mobile phone market, demonstrating the power of innovative user experience over “legacy thinking” that tries to port old paradigms to new contexts.

In Virtual Reality (VR), the “second wave” initiated by Oculus around 2011 brought significant industry impact, but usability remained a persistent challenge. While game controllers excelled for specific tasks, broader VR adoption was hampered by complex control schemes and a lack of a universal, intuitive interaction model comparable to the mouse or multi-touch. Early VR interaction paradigms for distant objects included headpointing and hand-based ray pointing (e.g., pointing with the hand and pinching to click). However, both suffered from head or hand fatigue, preventing widespread adoption. The emerging gaze-and-pinch concept, now embraced by Apple, Meta, and Google, signifies a crucial industry direction.

The Roots of Eye Input in Research

The notion of using eyes for interaction dates back to 1981 with Bolt’s research at MIT, including his seminal “Put That There” work. Bolt’s vision involved active screens responding to gaze, but importantly, his system also incorporated a small joystick for thumb interaction, recognizing the need for an explicit command to confirm user intent. This early combination of looking and physical input became a recurring theme in HCI research.

However, gaze research often lagged new device introductions. Efforts to combine gaze with the mouse, for instance, aimed to improve cursor control but fell into the trap of “legacy thinking.” They tried to apply the mouse’s two-step interaction model (move cursor, then click) to gaze, which inherently slows down direct manipulation—a core principle of touchscreens where selection is immediate. This approach was suitable for large, precise displays but counterproductive for direct interaction.

By 2013, when Pfeuffer began his PhD, gaze research was considered a “desert” within the HCI community. Skepticism abounded regarding accuracy, unnaturalness, and eye fatigue. Pfeuffer, however, saw this as an opportunity, dedicating his PhD and subsequent post-doctoral work to exploring this under-researched field.

His decade-long research involved numerous prototypes and user studies across various contexts:

Multi-touch with Gaze (2D): By mapping multi-touch finger input to gaze, he demonstrated that tasks like sorting could be executed fluidly and rapidly. Users could look at targets and use specific finger counts (e.g., one finger for one target, three for three) to select, minimizing broad hand movements.
Gaze-Enhanced Pinch-to-Zoom: This innovation allowed users to pinch-zoom from the side of a display while their gaze continuously corrected the zoom’s center. This meant users could focus on incoming details without their hands obscuring the view, improving precision and visibility.
Stylus and Gaze: Recognizing the natural drawing semantics of a pen, Pfeuffer designed a system where gaze contextually altered the pen’s function. While drawing, gaze was irrelevant, but looking at a menu would change the pen’s stroke from direct drawing to an indirect command, enabling a seamless blend of direct and indirect control.
Tablet Browser Interaction: For practical applications, gaze could predict user intent (e.g., looking at a tab to switch) and, combined with subtle thumb movements, offered highly precise interactions like button presses and even text entry.

Around 2016, with the advent of state-of-the-art VR like the HTC Vive, Pfeuffer extended his concepts to 3D. He identified pinching as the most straightforward and trackable 3D hand gesture. His work combined eye and hand tracking in VR, allowing for intuitive interactions with virtual objects, seamlessly switching between direct (physically reaching) and indirect (gaze-selected from a distance) manipulation, even on a virtual canvas or menu.

Despite publishing extensively, Pfeuffer’s research initially went largely unnoticed, entering a period of “hibernation.” However, Apple’s 2023 announcement of the Vision Pro, with its subsequent operating system integration in 2024, provided immense validation. Apple’s own UI guidelines explicitly describe interaction concepts strikingly similar to his direct and indirect gesture dimensions, confirming the real-world impact of his foundational work.

The Road Ahead: Future Directions in Eye-Hand Symbiosis

Pfeuffer asserts that the Apple Vision Pro, while groundbreaking, is merely the initial step. His research group is already exploring the “next generation” beyond current gaze-and-pinch implementations, building upon their foundational work from 2017.

Future research directions aim to refine and expand eye-hand symbiosis:

Beyond Gaze-and-Pinch Limitations: While simple, gaze-and-pinch still requires hand movement for object manipulation. Newer concepts seek to further minimize this.
“Look and Drop”: This concept extends eye control to the “drag” component of drag-and-drop. While eyes manage 2D movement, the hand controls depth. Studies showed that this approach, though not necessarily faster, significantly reduced user fatigue and hand movement. Pfeuffer showcased a 3D “Visual” game where a proficient user performed complex 3D object manipulations with minimal hand motion, relying heavily on eye-hand coordination.
Two-Handed Interaction with Gaze: This advanced model allows each hand to independently switch between direct (physical reach) and indirect (eye-controlled selection) gestures. This dynamic flexibility empowers users to execute complex two-handed tasks with unprecedented fluidity and choice of interaction method.

In conclusion, Pfeuffer envisions a profound paradigm shift: moving beyond eye-tracking purely for accessibility towards its ubiquitous integration with hand interaction in every possible context. The Vision Pro’s introduction serves as a crucial catalyst, opening vast possibilities across diverse devices and catering to varying user abilities, compelling a re-evaluation of all hand-based interactions to explore how eye-tracking can simplify and enhance them.