Meta AI will be available on Quest headsets in the US and Canada as an experimental feature in August, and will include the Vision capability on Quest 3. Meta AI is the company’s conversational AI assistant powered by its Llama series of open-source large language models (LLMs). In the US & Canada, Meta AI is currently available in text form on the web at meta.ai or in the WhatsApp and Messenger phone apps, and in audio form on Ray-Ban Meta smart glasses via saying “Hey Meta”.
I would think the quest 3 could probably run a light LLM if that was all it was doing, no pass through, tracking or rendering of any kind.
Sounds about right. But a multimodal one? Ehh… sticking with Meta, their smallest LLaMa is a 7b, and as such without any multimodal features it’s already going to use most of the Quest’s 8gb and it would be slow enough that people wouldn’t like it. Going smaller is fun, for example I like (in the app I linked) to use a 1.6b model, it’s virtually useless but it sure can summarize text. And to be fair, there are multimodal ones that could run on the Quest (not fast), but going small means lower quality. For example the best one I can run on my phone takes… maybe 20 seconds? To generate this description “ The image shows three high-performance sports cars racing on a track. The first car is a white Lamborghini, the second car is a red Ferrari, and the third car is a white Bugatti. The cars are positioned in a straight line, with the white Lamborghini in the lead, followed by the red Ferrari, and then the white Bugatti. The background is blurred, but it appears to be a race track, indicating that the cars are racing on a track.” and it’s not bad. But
I’m not sure I’d call it trustworthy :D
Yeah, it would not be at all practical to run an LLM on device, it would just be theoretically possible. Realistically, they want you to have a constant internet connection anyways to use the device so Meta would probably want a cloud reliant solution.