Driving with Vision Pro: A Look into Apple’s Patent for Vehicle Assistance

Earlier this month, a bizarre incident occurred when Apple launched its Mixed Reality (MR) headset, Vision Pro. A user was spotted operating the Vision Pro while sitting in the driver’s seat of a driving car. The driver is known to be in a state of activating Tesla’s autonomous driving mode. The video of the scene spread rapidly, and voices criticizing it began to grow.

As the controversy disseminated, the U.S. Secretary of Transportation stepped in. Secretary Pete Buttigieg stated on his X (formerly Twitter) account, “All advanced driver assistance systems in use today require the passenger to always focus on driving and control the vehicle.” In other words, he warned against utilizing the Vision Pro while driving. Apple also advised never to use Vision Pro while driving.

Adapting Vision Pro as a Vehicle Assistance Device

In the future, the Vision Pro may used in driving vehicles. According to the U.S. IT media outlet TomsGuide, on February 19 (local time), Apple applied for a patent using a Head-Mounted Display (HMD) like Vision Pro as a vehicle system assistance device. Apple filed the patent to the U.S. Patent and Trademark Office (USPTO) in January last year. The patent was made public on February 15 this year.

The main content of the patent is for a Cruise Control system that automatically maintains the speed for the driver’s convenience while the vehicle is in motion. The system automatically controls the car when the user sets the desired speed. The Adaptive Cruise Control system adds a function to the fundamental function to maintain the distance from the vehicle ahead.

“Managing cruise control settings using electronic devices is cumbersome and inefficient,” Apple stated in a patent document. “Some technologies use complex and time-consuming user interfaces.” Apple demonstrated the method of utilizing wearable devices such as HMD and effectively set up the cruise control system in the patent.

Put differently, HMD may used like a Vision Pro operating a particular car system.

How Will HMDs Be Exploited?

Apple revealed a plan to provide a new interface for electronic devices, enabling faster and more efficient cruise control system settings. The given patent drawing below helps us understand. The interface displays driving information such as vehicle speed and road speed limit. Switching to map mode lays out a map. Various information on vehicle driving is located under the map.

The media claimed, “Apple is trying to introduce spatial computing to its vehicles and provide map mode that provides important data such as visual information and lane guidance indication.” The device displays the information above on HMD devices such as Vision Pro. Apple also mentioned in the patent that cruise control system settings can be changed using body tracking and voice commands. Body tracking is an essential feature supported by the MR headset.

This idea is similar to the plan of AR company Xreal, which aims to introduce Augmented Reality (AR) to vehicles. Previously, Xreal announced its plans at CES 2024, the world’s largest consumer electronics fair. One of them is to implement AR functions that aid driving in collaboration with BMW. For example, the company is considering a function that displays various information, such as navigation and danger alerts, in front of the driver wearing the company’s AR device.

It is similar to the plan of Xreal, an AR company, to introduce augmented reality (AR) in its vehicles. Xreal announced its plans at CES 2024, the world’s largest consumer electronics fair. One of them is to join hands with BMW to implement AR functions that help drivers drive. For example, the company is considering a function that displays various information, such as directions and notifications of dangerous situations in front of a driver wearing AR devices. ​

Actual Implementation May Be Difficult

There is no guarantee for the implementation even though the patent is registered since big tech companies such as Apple often file patents first to preempt technology. In particular, Apple annually applies for numerous patents, but only a few products reflect them. Foldable patents are good examples for explanation. Apple has long secured patents for foldable displays but has never released foldable products.
​
Moreover, an aspect to consider is that Apple values its hardware and software ecosystem. The vehicle that works perfectly with the Vision Pro should be the Apple car. However, there has been no news on the Apple car for years. News of the Apple car has been reported since 2014, but no specific schedule has been confirmed. According to Bloomberg News last month, the launch date of the Apple car has been postponed to 2028.