AIvaluateXR: An Evaluation Framework for On-Device AI in XR with Benchmarking Results
IEEE Transactions on Visualization and Computer Graphics (IEEE TVCG), 2026.
Authors
Dawar Khan, Xinyu Liu, Omar Mena, Donggang Jia, Alexandre Kouyoumdjian, Ivan ViolaDescription
The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance human–AI interaction. However, selecting the appropriate model and device for specific tasks remains challenging when performing direct on-device inference.
In this work, we present AIvaluateXR, a comprehensive evaluation framework for benchmarking LLMs running on XR devices. To demonstrate the framework, we deploy 17 selected LLMs across four XR platforms—Magic Leap 2, Meta Quest 3, Vivo X100s Pro, and Apple Vision Pro—and conduct an extensive evaluation.
Our experimental setup measures four key metrics: performance consistency, processing speed, memory usage, and battery consumption. For each of the 68 model–device pairs, we assess performance under varying string lengths, batch sizes, and thread counts, analyzing the tradeoffs for real-time XR applications.
We further propose a unified evaluation method based on 3D Pareto Optimality to identify optimal device–model combinations considering both quality and speed objectives. Additionally, we compare the efficiency of on-device LLMs with client–server and cloud-based setups and evaluate their performance on interactive XR tasks.
We believe our findings provide valuable insights for future optimization of on-device AI systems for XR. The proposed evaluation framework serves as foundational groundwork for future research and development in this emerging area.