You’re behind the wheel, cruising along at highway speed. But you’re not looking at the road—at least not directly. No, you’re piloting your car from behind a head-mounted display, and the world that meets your eyes is a curious mixture of the real and the virtual.
It sounds outrageous, but the practice of test driving vehicles while wearing a mixed reality headset has proven supremely practical for Volvo Cars, and the results have been nothing short of groundbreaking. Spearheaded by Casper Wickman, the company’s senior technical leader of human-centric methods and tools, these technological trials have dramatically simplified the Swedish marque’s process of evaluating new safety systems and design concepts.
But what’s it like from the driver’s seat?
“You would think that this would be a crazy experience, like you’re in a video game or something,” Wickman said. “But the thing is, as soon as you put the glasses on and you sit behind the steering wheel and start to drive, then your primary task is to drive the car, basically. Then you really forget that you’re inside a headset. It feels super natural to do it. It becomes your world, basically.”
Of course, you can’t do that with just any headset. To arrive at this destination required a tight-knit development partnership with Finnish XR innovator Varjo—and a whole lot of clever engineering.
Starting Up
Wickman has been helping Volvo improve its processes for two decades. He joined the company in 2000 as a student in its industrial PhD program, which employs a group of 50 to 70 students at its Gothenburg headquarters in tasks like variation simulation and visualization, and the analysis of quantitative data with the goal of eliminating manufacturing defects.
After five years in this capacity, he joined a new department formed to study “perceived quality”—to predict how things will look and feel in the completed vehicle. It’s within this general area that he has remained, working his way up through various leadership tiers to his current senior technical role focused on novel methodologies.
“We want to develop tools with new emerging technologies to be able to put the user in the experience in the very early stages of design, in order to be able to evaluate whether we are doing the correct things before we do anything,” he said. “I’m working a lot within user experience, but really in any area where you need to put the user in the experience.”
Getting in Gear
Wickman began considering the prospect of harnessing virtual and augmented reality in his work several years ago, but the technology wasn’t ready. “When it comes to detailing and understanding the look of things in a car context, you need to be able to see the stitching, see the textures, see how things are lined up against each other,” he said. “You need to see more than the complete design.”
Then, a couple years back, he participated in a project that involved using Microsoft HoloLens AR glasses for manufacturing and packaging activities. “But (in my view) they were using augmented reality as if it were an ordinary desktop application in some way,” Wickman said. “They were not utilizing the full capacity of AR.”
After some brainstorming over a better use for the technology, Wickman arrived at the wild idea of driving with the headset. “They answered ‘No, that wouldn’t be possible for several reasons: you wouldn’t be able to track stuff, you wouldn’t be safe,’” he said.
It was around this time, however, that some other people at Volvo tipped him off to the work that Varjo was doing—specifically, the development of a head-worn display with a pixel density so high the company markets it as “human eye resolution.” Feeding this display with actual footage of the environment from cameras on the headset, Varjo’s XR-1 enables the merging of virtual and physical realities with a clarity never before possible.
In January of 2018, Wickman set up a meeting with Varjo’s founder and chief product officer Urho Konttori and chief marketing officer Jussi Mäkinen to go over his vision. “We presented that in terms of different steps that we’d like to go through, with the first step just being able to drive with a pair of glasses, using the cameras with video pass-through, to evaluate just how you would manage motion sickness and whether it would be a safe thing to do,” he said.
If that worked, he wanted to begin overlaying data onto the real world; then, he wanted to be able to interact with the virtual content. He also proposed the desire to analyze “gaze data” to see exactly where people looked as they completed tasks. Lastly, if everything else was possible, he wanted to add interactive virtual objects outside of the car.
“It was pretty cool,” Wickman said. “[Urho and Jussi] said directly, ‘This is the most challenging use case we’ve ever heard of in any industry. Let’s do this together. We’re in.’”
Picking Up Speed
Over the next six months, as the details of the contract were being hammered out, both companies began preparing on their own: Varjo worked to boost the capabilities of its XR-1 headset, and Volvo went about formulating a way to extract all of the necessary signals from the car. They then established a schedule over the following eight months in which they would conduct their tests. “At each trial, Varjo came over to Volvo and they brought a prototype XR-1 headset, and at each drive they had enhanced a lot of things,” Wickman said.
The setup begins with a computer in the back seat running Unity, the software in which everything operates. A local area network is used to handle all of the signal and data transmission. Mounted in front of the driver is a tracking system called 5C, from German company ART, which follows a set of little spherical sensors on the driver’s headset for spatial positioning. Also running through the LAN to the computer are the signals extracted from the vehicle, which consist of the flex ray and local interconnect network (LIN). These communicate a whole host of information to the system in Unity.
With this setup, Wickman and his team can then overlay a conceptual virtual cockpit over the actual one and interact with it. “You can read out signals like the steering wheel angle and connect that to the virtual model, so the physical steering wheel and virtual one are synchronized while driving,” Wickman said. Or, “if you press a switch in the car that’s usually used to activate the radio, we can use that signal instead to activate something else on the virtual model. So, maybe we’ve decided that in the next generation of the car to use a stalk or a switch for something else, then we can simulate that.”
Navigating Challenges
According to Wickman, the toughest challenge on Volvo’s end was getting the tracking system right. “A car is really a super complex environment,” he said. “You have magnetic fields, you have different light properties—sometimes it’s low sunshine, sometimes it’s bright, sometimes it’s dark—so it needs to be able to track in different light conditions.” On top of that, you have to deal with the vibrations and bumps from the road.
He said that Volvo and Varjo both tried out a number of tracking systems independently before arriving at the 5C system from ART, which is mounted very rigidly to the car. “That works super well,” he said. “Of course, some of the content might move a little bit, but it’s really impossible to realize that because you’re moving with the car yourself. So you will actually experience it like it’s super solid and fixed to the physical world.”
The other major challenge was latency. The earliest versions of the XR-1 headset had a latency of around 30 milliseconds—not great, but not terrible. “It was barely possible to drive a car with it, because it’s super sensitive,” Wickman said. “If you just look away to the side, then back to the road again, you might end up in the middle of the road.” Still, they managed to complete the test as scheduled, and eventually Varjo managed to shrink the response time down to just 12 milliseconds.
Then, of course, there’s the occasional hiccup in the system. “We’ve had a number of blackouts,” he said with a chuckle. “Actually, one time on the test track, the computer stopped working, so the headset became black. But we have a procedure for that: the driver says ‘Can’t see anything, take over the steering wheel,’ and they take off the headset. There’s never been a dangerous situation at all.”
The Road Ahead
As of this writing, Volvo had just received its own production version Varjo XR-1 headset, allowing Wickman and his team to begin formulating workflows around it as a bona fide development tool for the company. “We’re setting up a plan for all the future evaluations we’re going to do throughout this year,” he said. “We have a whole stack of things we want to do.”
Before new Volvo models ever hit the road, the company will know exactly what the visual experience will be like for the driver. This starts with visibility, or “how much you see from the car to be able to maneuver it,” Wickman said. “That’s one big application: to be able to bring the new design proposal into the car and overlay that, and then you can drive with the car outside and evaluate visibility.”
Wickman also plans to use the XR-1 to evaluate active safety technologies. Currently, when Volvo wants to test, for example, an automatic braking feature, they have to drive a car toward a physical object. “And if the system fails, you will actually hit the thing you’re trying to avoid,” he said. “Although we’re using soft targets, it’s still damaging to the sensors on the car.” Instead, they can place a virtual obstacle up the road and experiment with new safety systems in the digital realm first.
The next step from there is the development of technology for automated vehicles, or AVs. Wickman said that Volvo is already working on research projects in this field with the help of funding from Swedish authorities. “We’re trying to understand things like how AV cars should behave in terms of overtaking bicycles, for example,” he said. In order to test this the traditional way, you’d need to actually put someone on a bike ahead of the car; now they can do that in Unity. “We just add a virtual object as a bicycle, and a virtual object as an approaching car,” he said. “And you can optimize the algorithms for how the AV car should move without taking any risks for other people while trying it.”
Ultimately, what makes the integration of the Varjo XR-1 into Wickman’s work so successful is that it so directly fulfils the mission of his department: understanding and improving upon the experience people have with Volvo cars. And there’s much more to come.
“We have the perspective of the human being as the starting point for everything we’re doing,” he said. “We also scout a lot of different new technologies. A lot of start-ups all over the world are doing amazing things that we might want to incorporate in our cars in the future. So, being able in a very early stage to evaluate things without the need of actually implementing them, that is a huge advantage.”
