Engineers from Facebook Reality Labs and Oculus have developed a fully articulated hand tracking system for Virtual Reality (VR) that is based on completely on monochrome cameras.
The new system uses machine learning to improve hand tracking without the help of any depth sensors or any kind of special equipment. It uses the input of the four cameras from Oculus’s Quest, the cable-free, stand-alone VR headset that is passed to a deep neural network model for real-time precise hand-tracking. The network predicts the position of the hand together with well-defined landmarks such as joints on the hands. These keypoints are then used to construct a 26-degree of freedom pose of the person’s hand and fingers. All the output is combined into a 3D model of the hand together with the surface geometry.
Using deep neural networks for accurate hand-tracking on Oculus Quest
Facebook Reality Labs researchers used deep neural networks to develop a fully articulated hand-tracking system for VR that relies only on monochrome cameras. It will be deployed as a software update for the Oculus Quest headset. https://ai.facebook.com/blog/hand-tracking-deep-neural-networks/
Опубликовано Facebook AI Среда, 25 сентября 2019 г.
Researchers together with engineers developed the system in a way that all processing is done on-device and it also supports some basic gestures for interaction (pointing, pinching, etc.). They also developed APIs which will, in turn, enable developers to use these 3D models for new interaction mechanisms for VR apps.
Precise hand-tracking will bring numerous improvements in the whole VR experience. Users will be able to use gesture control in a seamless and entertaining way within their apps.
