Voice and gaze solve different parts of the request
Voice is good at expressing intent: open, summarize, draft, move, compare, or prepare. Voice is weaker when the target is spatial and the user naturally says "that" instead of reading a long label. Gaze is useful for indicating attention, but it is a poor consent signal because people look at things without intending to activate them.
Fluent assigns each signal a narrower job. Voice or text carries intent. A recent stable gaze fixation can name a semantic target. A separate action path decides what to do, and consequential behavior should pause at an explicit approval boundary.
What happens to gaze data
Eye control is off by default and can be paused independently. When enabled, WebGazer estimates gaze in the renderer. Stable fixations enter a short in-memory buffer. Fluent resolves a fixation locally against the Windows accessibility tree before a command is submitted.
The planner can receive a structured attention target containing fields such as the accessible name, role, process, bounds, fixation duration, and timestamp. Raw gaze samples are not persisted or included in agent input. WebGazer calibration data may remain in local browser storage.
- Camera and gaze permission are optional.
- Raw samples stay in a short in-memory buffer.
- Target resolution happens locally against a semantic accessibility element.
- Structured target metadata is attached only when a command is submitted.
- Coordinates are not treated as a click instruction.
What happens when the target is stale or unclear
A deictic request such as "click that" needs a recent reliable target. If Fluent cannot resolve one, the intended behavior is to ask the user to look again or clarify the target. The automation agent is instructed to observe the interface and use semantic controls rather than click a raw gaze coordinate.
The published research record is candid that gaze onboarding, quality, multi-display behavior, and recovery states still need product-grade work. That makes gaze appropriate for preview evaluation, not a dependable baseline. Voice and text must continue to work when gaze is unavailable.
Why multimodal input is worth testing
Research has long explored combining language with pointing because the modalities carry complementary information. Richard Bolt demonstrated voice plus pointing in the 1980 Put-That-There system. Later studies examined how multimodal input can reduce recognition ambiguity and support correction.
Those studies motivate the product hypothesis. They do not prove that Fluent is effective. Fluent must run its own accessibility-led task studies with the installed Windows product and publish the protocol, failures, and results.