EIRS

All demos · Industrial challenges

Conversational AI · Augmented reality · Process support

Conversational AI Assistance for Process Operations

During a process or task, an operator can ask an AI assistant for help and receive spoken guidance, instructions and annotations in augmented reality (AR). The JuiceMixer example shows conversational support and provides a starting point for future context aware assistance. Camera views and live process data could help an AI interpret changing conditions and update its support as the task progresses.

Conversational AI Assistance for Process Operations
System overview of the conversational AR assistant.

Guidance in the flow of work

Keep procedural support available where the work happens. Conversational AR lets an operator request guidance while remaining at the equipment and relate the response to the task in front of them.

What the demo shows

  • Voice questions answered using supplied procedure content.
  • Spoken and on-screen responses in HoloLens 2.
  • Spatial annotations aligned with relevant task locations.
  • Support for sequential assembly, troubleshooting across procedure sections and continuation of partly completed work.

Testbed and equipment

The example uses a six-vessel physical JuiceMixer process rig and a HoloLens 2 headset. A QR marker aligns the augmented cues with the equipment. The rig provides a physical setting for assembly, troubleshooting and task-continuation scenarios.

What it enables

  • Keep procedure information within reach while attending to equipment.
  • Relate an instruction to the component or task location it concerns.
  • Request clarification or the next relevant step during a procedure.
  • Explore conversational support for workflows that involve several steps or references.

The operational challenge

Process work can involve switching between equipment, instructions and troubleshooting references. Information may be spread across several sections of a procedure, and an operator returning to partly completed work needs to identify the next relevant action. Assistance needs to address both the question and its place in the workflow.

System overview

A Unity AR application connects a conversational backend with supplied procedure content. Answers are presented as speech and text, with structured task-step information used to select relevant spatial cues. The procedure content provides the reference for the current assistant’s responses.

How the assistance works

The operator starts assistance and speaks a question. Speech recognition converts the request to text. The conversational assistant uses the procedure content to form a response, which speech synthesis and the headset display present to the user. Structured response data can identify a task step and position its AR cues.

How operators use it

An operator can request the next step during assembly, ask about a problem that requires information from another procedure section or seek guidance when returning to unfinished work. Follow-up questions provide a way to clarify the response before continuing with the equipment.

Technology behind the testbed

The AR application runs in Unity with MRTK 3, the Mixed Reality Toolkit, on HoloLens 2. The conversational backend uses the OpenAI Assistants API with GPT-4.1. Speech recognition and synthesis support voice interaction. Procedure content is organised as structured data with page references, and QR marker alignment positions visual cues around the task area.

What the study shows

The IEEE VR study compared the conversational assistant with a PDF manual across a sequential assembly task, troubleshooting that required information from different parts of a procedure, and continuation of a partly completed task.

When time spent waiting for AI replies was removed, active task time was lower with the assistant in the assembly and troubleshooting tasks. Total completion time did not consistently fall because response delays remained part of the workflow. In one safety instruction step, users with the assistant more often completed the requested action.

For operations teams, the practical lesson is that guidance must fit both the task and its pace. Response speed matters alongside the quality and relevance of the information.

Where this could apply

This approach could be explored in maintenance, assembly, setup and changeover, troubleshooting, inspection, and operator onboarding. It is most relevant where people need to keep attention on equipment while consulting procedures with several steps or references to other sections. An application for a specific site could align its information and spatial cues with a particular asset and workflow.

Next development opportunities

Further development could connect camera views and live process signals to approved procedure content. An AI could use what it sees and the current process state to adapt instructions as work progresses. This would create a continuous feedback loop between the equipment, the assistant, and the operator. A practical deployment would need operator confirmation, safety rules, controlled access to site information, suitable response times, and validation with operators in representative conditions.

Industrial applications

Procedural work often requires people to consult information while keeping track of equipment and the task in progress. Conversational assistance can make that knowledge available through questions, responses and guidance connected to the workspace.

AI assistance from technical documentation

Make approved procedures and equipment information accessible through conversation. A project would prepare the reference content and evaluate the accuracy and usefulness of responses.

Conversational AR support at the machine

Combine spoken assistance with instructions and annotations at relevant task locations. Guidance can be developed around the equipment and procedures that operators need to follow.

Conversational troubleshooting

Help users find related information as an investigation develops. Questions can move between procedure sections rather than remaining within a fixed sequence of instructions.

Spatial guidance for maintenance tasks

Connect a defined maintenance procedure with the components and action locations involved. The instructions and cues would be authored and checked against the actual work.

Develop assistance around your operations

A collaboration could begin with a defined procedure and approved knowledge. Further development could investigate camera context and live machine data as inputs to assistance that updates with the task.

Explore research and collaboration →

Media

Videos

Images

AR task assistance with the JuiceMixer testbed.
AR task assistance with the JuiceMixer testbed.
Comparison of conversational guidance and a PDF manual.
Comparison of conversational guidance and a PDF manual.
Conversational guidance paired with a procedure step.
Conversational guidance paired with a procedure step.
Spatial cues mark task locations on the JuiceMixer.
Spatial cues mark task locations on the JuiceMixer.
Troubleshooting guidance connects information across a procedure.
Troubleshooting guidance connects information across a procedure.
HoloLens 2 setup for task assistance on the JuiceMixer.
HoloLens 2 setup for task assistance on the JuiceMixer.
The study compares an AR assistant with a PDF manual.
The study compares an AR assistant with a PDF manual.
Concept: an operator asks for guidance at a machine.
Concept: an operator asks for guidance at a machine.
Concept: AR guidance responds beside the equipment.
Concept: AR guidance responds beside the equipment.
Concept: equipment identification linked to task guidance.
Concept: equipment identification linked to task guidance.
Concept: information flow for a conversational AR assistant.
Concept: information flow for a conversational AR assistant.
Concept: operator requests connected with AR guidance.
Concept: operator requests connected with AR guidance.
Task steps for filling vessels, connecting lines, and operating pumps.
Task steps for filling vessels, connecting lines, and operating pumps.
Design perspectives for novice users, designers, and engineers.
Design perspectives for novice users, designers, and engineers.
Conversational task support shown across AR screens.
Conversational task support shown across AR screens.
Concept: ARIA glasses capture task context and user-related information during work.
Concept: ARIA glasses capture task context and user-related information during work.

Research and reports

Project and programme

This work is part of DDIA, Data Driven Immersive Analytics in Digital Industries.

DDIA is an FFG COMET Module (2022 - 2026) at Know Center Research GmbH.

↑ Top