Mark Zuckerberg, head honcho of the organisation formerly known as Facebook, has completed his first interview entirely conducted within his company’s eponymous Metaverse. And it’s sort of impressive, sort of unnerving and sort of meh, all rolled into one.
Zuckerberg hooked up with Lex Fridman for the interview, a podcaster by whom he has been interviewed on several previous occasions. Zuckerberg and Fridman were remotely located from one another, Fridman in Austin Texas and Zuckerberg presumably at Meta’s California head office.
For the interview, both were subject to detailed physical scans and also some kind of facial expression and movement training process, perhaps akin to a more complex version of the training routines used to learn biometric data for smartphone security including fingerprints and faces.
Zuckerberg explains that this data is then collapsed into a codec which in turn is combined with sensor data from a VR headset to create a lifelike digital avatar. What’s quite clever is that this codec approach means that the bandwidth required is relatively limited. Once each party has a copy of the other’s digital avatar, the data required to animate them is much less than for, say, transmitting high definition digital video.
In theory, then, you have something much more realistic than video but for less bandwidth. But just how realistic? Up to a point, you can judge for yourself. Fridman has uploaded a 2D representation of the interview on his YouTube channel including both avatars.
Sadly, the video is only 1080p. But even at that relatively low resolution, it’s possible to see that the avatars have a fairly unambiguous CGI vibe. They’re good, for sure, but nobody even moderately familiar with modern CGI would mistake them for live video.
Of course, the character of the protagonists themselves arguably doesn’t help. You wouldn’t put either Fridman or Zuckerberg down as being terribly emotive. Their real selves can be pretty robotic. So the extent to which the synthetic feel of these digital avatars is down to the translation into codecs, or just the actual bearing of these rather unusual protagonists, is something of an open question.
Still, there’s a slightly flat, digitised feel to the avatars that probably isn’t entirely down to physical the bearing of Zuckerberg and Fridman. That said, the experience is no doubt orders of magnitude more impressive and immersive when actually using a VR headset.
Likewise, from the video uploaded, it seems like Meta has pretty much nailed the problem of latency. The two seem to be having a perfectly natural, flowing conversation despite the fairly hefty processing that must take place between any speech or facial expression and those behaviours being sensed, translated into the codec and transmitted to the other user.
As for the hardware being used, at a glance it looks like Meta Quest Pro headsets, plus a set of fairly hefty cans. And that, really, remains the main objection to this kind of experience—the clunky hardware you have to wear.
Anyway, check out the video for yourself while you wait for Meta’s hot new VR headset the Quest 3, which launches on the 10th. Stay tuned for our own take on the hardware very soon.