What Does Facebook Envision for the Future of User Interface for XR Devices?

Facebook Reality Labs demonstrated a wristband wearable concept (resembling a smartwatch with a large screen) that detects hand/finger movements and allows the user to mentally control and interact with AR/VR devices. This User Interface (UI) leverages electromyography (EMG), which relies on sensors to translate electrical motor nerve signals that travel through the wrist to the hand into digital commands that can be used to control the functions of a device (such as clicking, swiping, and typing). Facebook claims that EMG is very accurate and can understand finger motion of just a millimeter. Initially, EMG will be limited to tapping on a button, but long term, the control options will be enriched (such as manipulating virtual objects in space or typing on a virtual keyboard) and support haptic feedback for more immersive experiences. To build this solution, Facebook is leveraging the technology from CTRL-labs, a startup that was acquired in 2019 (US$500 million to US$1 billion, estimated) and designed a wristband capable of transmitting electrical signals from the brain into computer input.

Facebook is not the first company that is exploring brain control interface for XR devices. Neurable has developed a prototype that utilizes the HTC VIVE headset and seven electroencephalography (EEG) sensors that allow users to mentally control VR games.

UI is a critical component for AR smartglasses that not only affects user experience but determines application/use case capabilities and device value/utility. At the current stage of the market, the majority of AR smartglasses employ voice commands and embedded buttons as the primary input methods, while more advanced AR/MR devices leverage gesture control and eye tracking.

Due to the technological advancements in voice engines, voice control is one of the most widely adopted input methods for numerous electronic devices, including AR smartglasses, that allow hands-free interaction, which is essential for some critical enterprise use cases. However, voice control is not always a socially acceptable interaction method and voice engines cannot always accurately and swiftly understand/process commands, especially in noisy/windy backgrounds. Touchpads and buttons on the side of an HMD are considered a user-friendly and accurate interaction method, although they limit smartglasses design/size options due to the space that is required in the smartglasses frame and are not convenient for tasks that require hand movement.

In order to keep a small and stylish design, some AR smartglasses manufacturers such as North (now owned by Google) utilize finger-worn devices that are similar to ordinary rings in their design to enable interaction with the device without being noticeable. Apple has filed a patent about a wearable electronic ring that allows users to control devices, while Amazon has revealed the Echo Loop, which is an Alexa-enabled ring. However, available finger-worn devices require further improvements in materials that are used, more stylish and ergonomic design, wireless connectivity, and battery life.

Facebook’s wristband control could not only address some of the challenges with existing UIs but also unlock opportunities for new use cases and applications due to smoother and faster interaction, such as remote working with the assistance of XR devices or supporting people with disabilities. Facebook’s investment in wrist control highlights that hands-free manipulation and interaction with the device will be a key future as the technology matures and AR smartglasses see more ubiquitous use, especially for consumers. For the mass market, users need AR/MR devices that are more autonomous, removing the need for clicking buttons and handling controllers. However, the usage of a wristband is a long-term concept that requires further technological advancements, more intelligent AI algorithms, lower development/manufacturing cost, and discussion around data management and ethical issues that may come up in order to increase consumer awareness/acceptance about this type of technology.

ABI Research expects that 2022 will be a significant year for the AR consumer smartglasses market, with shipments reaching approximately 1 million. Leading technology companies such as Facebook, Snapchat, Samsung, Google, and Apple are preparing for their journey in spatial computing and immersive, and are continuously investing in XR tech by securing patents and acquiring novel startups. For example, Apple recently published a patent that allows users to control XR devices with micro-gestures, using the thumb against the index finger as a sort of virtual joystick or selector. While not confirmed, Google’s Soli tech, seen in last year’s Pixel 4 smartphone and the refreshed Nest Hub, could be leveraged in a similar way.

Apart from design, price, and computing capabilities, input methods and UI are key differentiation features for AR/MR devices. Input can define the type of applications/use cases available, applicable environments (outdoor versus indoor), and possible usage time. It is expected that the first generation of AR consumer smartglasses will leverage “traditional” input methods such as voice commands or control through a tethered smartphone.

ABI Research has examined the different data input options in XR devices and the role of UI in User Experience (UX). The suitability of each input method and UI design strongly depends on the use case needs and goals along with the device’s form factor (binocular/monocular). There is no doubt that simple and intuitive UI are essential for enhancing user experience, reducing friction and churn, and lessening the need for learning/training, all ultimately fueling AR smartglasses growth. At the same time, UI and input methods play a catalytic role for encouraging application and content development, opening opportunities for entirely novel content as well as enhancing existing experiences.

Designing UI and input methods is challenging for enterprise AR/MR devices, due to the design/size limitations, safety restrictions, and pricing (especially for consumer). It is expected that manufacturers with mass consumer electronic UI/UX experience such as Samsung and Apple will have an advantage over social media players like Snapchat, or new, smaller entrants.

Facebook’s activity in AR and VR is broad, and so it is clear that wrist-worn devices will not be a singular solution for the company, but rather an option among many. Experience from Oculus on the VR side could inspire AR input ideas and opportunity as well. The company’s social network strengths are in fact only a portion of the overall vision for AR and VR, where HMDs, input, platforms, and content coalesce into a more holistic market approach.