EIRS

All demos · Industrial challenges

VR · AR · Immersive control

Bidirectional VR/AR Control of Digital Twins

A connected digital twin links physical equipment with its digital representation. In this example, a user in virtual reality (VR) operates the JuiceMixer process testbed. Commands reach the real machine, while its current state returns to the connected VR and augmented reality (AR) interfaces.

Bidirectional VR/AR Control of Digital Twins
A virtual hand interacts with the JuiceMixer digital twin.

Control in both directions

Connect immersive commands to physical machine behaviour, then return the machine’s current state to the VR and AR views.

What the demo shows

  • A Meta Quest Pro VR interface connected to the physical machine.
  • Virtual buttons for machine functions, including actions without a dedicated physical control.
  • A rotary control that changes motor behaviour.
  • Two-way communication that returns machine state to the connected Unity VR and AR applications.

Testbed and equipment

The physical JuiceMixer is a process rig with liquid vessels, pumps and sensors. A Meta Quest Pro provides the VR control interface, and HoloLens 2 provides the AR view. The equipment and both immersive applications share local control and data infrastructure.

What it enables

Issue a command in VR, observe the physical machine’s response and inspect the returned state in the connected views. This creates a route from an immersive interaction to a physical action and back to visible feedback.

The operational challenge

A digital twin can show a system without providing a way to interact with the physical equipment. This demo connects the two. Immersive inputs can drive machine behaviour, while machine state returns to the digital views.

System overview

The system connects virtual controls to machine functions and returns machine-state updates to the immersive applications. The same local infrastructure supports commands going to the equipment and feedback coming back to the digital views.

How the data flows

Button presses and rotary inputs pass through a local control interface built with Python and Flask. MQTT messaging carries commands to the physical testbed and returns updated machine state to the VR and AR views. InfluxDB 2.0 stores time-series readings.

How operators use it

The operator uses virtual buttons to select machine actions, including functions without a dedicated physical control on the testbed. Turning the virtual rotary control changes motor behaviour. It does not mechanically turn or reposition a physical knob. The returned machine state provides feedback in the connected VR and AR views.

Technology behind the testbed

Unity applications run on Meta Quest Pro and HoloLens 2. A Raspberry Pi coordinates the physical testbed. The backend uses Python and Flask, MQTT carries messages over the local network, and InfluxDB 2.0 stores time-series readings. The setup runs locally.

Next development opportunities

Further developments could connect the interaction to machine-specific functions, define control and safety requirements, document the operating protocol, and measure command and state update timing on the target network.

Industrial relevance

This pattern could be adapted to remote operation, commissioning, supervisory control, training, digital-twin testing, or remote expert interaction where physical and digital state must remain synchronized.

Industrial applications

A digital equipment model can become an interface for selected machine functions. Connecting commands with returned machine state creates opportunities to explore controls, understand equipment behaviour and develop new ways of interacting with machinery.

Connected virtual machine controls

Map virtual buttons or rotary inputs to physical machine functions and show the resulting state. A project would define the permitted actions, control interfaces and feedback needed for the target equipment.

Interactive equipment models

Let users explore machine components and selected interactions in a virtual environment. This can provide a foundation for equipment familiarisation and discussion of how controls should work.

Explore equipment before installation

Use a virtual machine representation to discuss operation, access and interaction before equipment is installed. A project could incorporate engineering models and selected behaviours relevant to the planned installation.

Connect the controls that matter to your task

A collaboration could develop and evaluate an immersive interface around a defined set of machine functions and operating requirements.

Explore research and collaboration →

Media

Videos

Images

A virtual hand interacts with the JuiceMixer digital twin.
A virtual hand interacts with the JuiceMixer digital twin.
Inspecting a vessel and its tubing in VR.
Inspecting a vessel and its tubing in VR.
Interacting with a vessel in the virtual testbed.
Interacting with a vessel in the virtual testbed.
VR control panel for pumps 1–6.
VR control panel for pumps 1–6.
Pump controls and tubing in the virtual testbed.
Pump controls and tubing in the virtual testbed.
The JuiceMixer in its virtual environment.
The JuiceMixer in its virtual environment.

Research and reports

Project and programme

This work is part of DDIA, Data Driven Immersive Analytics in Digital Industries.

DDIA is an FFG COMET Module (2022 - 2026) at Know Center Research GmbH.

↑ Top