FB developing AI that can see, hear, remember whatever you do

Facebook is developing a new artificial intelligence (AI)-based system that can analyse your lives using first-person videos -- recording what they see, do and hear in order to help you with daily tasks.

Imagine your AR device displaying exactly how to hold the sticks during a drum lesson, guiding you through a recipe, helping you find your lost keys, or recalling memories as holograms that come to life in front of you.

To realise this new AI potential, Facebook AI has announced 'Ego4D' -- a long-term project aimed at solving research challenges in 'egocentric perception' (the perception of direction or position of oneself based on visual information).

"We brought together a consortium of 13 universities and labs across nine countries, who collected more than 2,200 hours of first-person video in the wild, featuring over 700 participants going about their daily lives," the social network said in a statement.


This dramatically increases the scale of egocentric data publicly available to the research community by an order of magnitude, more than 20x greater than any other data set in terms of hours of footage.

"Next-generation AI systems will need to learn from an entirely different kind of data -- videos that show the world from the center of the action, rather than the sidelines," said Kristen Grauman, lead research scientist at Facebook.

In collaboration with the consortium and Facebook Reality Labs Research (FRL Research), Facebook AI has developed five benchmark challenges centred on first-person visual experience that will spur advancements toward real-world applications for future AI assistants.


Ego4D's five benchmarks are episodic memory, forecasting, hand and object manipulation, audio-visual 'diarization' and social interaction.

"These benchmarks will catalyse research on the building blocks necessary to develop smarter AI assistants that can understand and interact not just in the real world but also in the metaverse, where physical reality, AR, and VR all come together in a single space," Facebook elaborated.

The data sets will be publicly available in November this year for researchers who sign Ego4D's data use agreement.


As a supplement to this work, researchers from FRL used Vuzix Blade smart glasses to collect an additional 400 hours of first-person video data in staged environments. This data will be released as well.

While it's easy for people to relate to both first- and third-person perspectives, AI today doesn't share that level of understanding.

"For AI systems to interact with the world the way we do, the AI field needs to evolve to an entirely new paradigm of first-person perception," Grauman said. "That means teaching AI to understand daily life activities through human eyes in the context of real-time motion, interaction, and multi-sensory observations - IANS