AI assistance from technical documentation
Make approved procedures and equipment information accessible through conversation. A project would prepare the reference content and evaluate the accuracy and usefulness of responses.
EIRSAll demos · Industrial challenges
During a process or task, an operator can ask an AI assistant for help and receive spoken guidance, instructions and annotations in augmented reality (AR). The JuiceMixer example shows conversational support and provides a starting point for future context aware assistance. Camera views and live process data could help an AI interpret changing conditions and update its support as the task progresses.

Keep procedural support available where the work happens. Conversational AR lets an operator request guidance while remaining at the equipment and relate the response to the task in front of them.
The example uses a six-vessel physical JuiceMixer process rig and a HoloLens 2 headset. A QR marker aligns the augmented cues with the equipment. The rig provides a physical setting for assembly, troubleshooting and task-continuation scenarios.
Process work can involve switching between equipment, instructions and troubleshooting references. Information may be spread across several sections of a procedure, and an operator returning to partly completed work needs to identify the next relevant action. Assistance needs to address both the question and its place in the workflow.
A Unity AR application connects a conversational backend with supplied procedure content. Answers are presented as speech and text, with structured task-step information used to select relevant spatial cues. The procedure content provides the reference for the current assistant’s responses.
The operator starts assistance and speaks a question. Speech recognition converts the request to text. The conversational assistant uses the procedure content to form a response, which speech synthesis and the headset display present to the user. Structured response data can identify a task step and position its AR cues.
An operator can request the next step during assembly, ask about a problem that requires information from another procedure section or seek guidance when returning to unfinished work. Follow-up questions provide a way to clarify the response before continuing with the equipment.
The AR application runs in Unity with MRTK 3, the Mixed Reality Toolkit, on HoloLens 2. The conversational backend uses the OpenAI Assistants API with GPT-4.1. Speech recognition and synthesis support voice interaction. Procedure content is organised as structured data with page references, and QR marker alignment positions visual cues around the task area.
The IEEE VR study compared the conversational assistant with a PDF manual across a sequential assembly task, troubleshooting that required information from different parts of a procedure, and continuation of a partly completed task.
When time spent waiting for AI replies was removed, active task time was lower with the assistant in the assembly and troubleshooting tasks. Total completion time did not consistently fall because response delays remained part of the workflow. In one safety instruction step, users with the assistant more often completed the requested action.
For operations teams, the practical lesson is that guidance must fit both the task and its pace. Response speed matters alongside the quality and relevance of the information.
This approach could be explored in maintenance, assembly, setup and changeover, troubleshooting, inspection, and operator onboarding. It is most relevant where people need to keep attention on equipment while consulting procedures with several steps or references to other sections. An application for a specific site could align its information and spatial cues with a particular asset and workflow.
Further development could connect camera views and live process signals to approved procedure content. An AI could use what it sees and the current process state to adapt instructions as work progresses. This would create a continuous feedback loop between the equipment, the assistant, and the operator. A practical deployment would need operator confirmation, safety rules, controlled access to site information, suitable response times, and validation with operators in representative conditions.
Procedural work often requires people to consult information while keeping track of equipment and the task in progress. Conversational assistance can make that knowledge available through questions, responses and guidance connected to the workspace.
Make approved procedures and equipment information accessible through conversation. A project would prepare the reference content and evaluate the accuracy and usefulness of responses.
Combine spoken assistance with instructions and annotations at relevant task locations. Guidance can be developed around the equipment and procedures that operators need to follow.
Help users find related information as an investigation develops. Questions can move between procedure sections rather than remaining within a fixed sequence of instructions.
Connect a defined maintenance procedure with the components and action locations involved. The instructions and cues would be authored and checked against the actual work.
A collaboration could begin with a defined procedure and approved knowledge. Further development could investigate camera context and live machine data as inputs to assistance that updates with the task.
Explore research and collaboration →Conversational AR assistance during JuiceMixer task scenarios.
















This work is part of DDIA, Data Driven Immersive Analytics in Digital Industries.
DDIA is an FFG COMET Module (2022 - 2026) at Know Center Research GmbH.