EIRS
Human–robot collaboration and Physical AISolution portfolio
Human–robot collaboration and Physical AI

Closed-loop multimodal assistance

Investigate how camera views, measurements, language and human input could support assistance that updates as work progresses. The aim is a feedback loop between sensing, guidance, operator action and system response.

Potential benefits

  • Investigate guidance that stays relevant as a task progresses.
  • Reduce disconnected information across sensing, assistance and system feedback.
  • Evaluate whether a combined feedback loop supports the work better than separate tools.

Relevant challenges

Operations

  • Operational information is fragmented across multiple systems

Process Improvement and Decision Support

  • People’s practical experience and machine measurements are rarely considered together.

How the approach works

Several information sources would contribute to an understanding of the current task. Guidance or actions would then produce feedback that updates the system’s next response, connecting digital assistance with what happens in the physical or virtual environment.

Research and demonstrations

The current examples cover situated data, machine control, movement measurement, conversational augmented reality (AR) and robot operation. They provide separate foundations for a combined system. Connecting them into one adaptive feedback loop would require further research and integration.

Adapting it to your workflow

A project would develop data synchronisation, multimodal reasoning and adaptation across the selected components. Evaluation would examine the integrated workflow rather than assuming that individually working parts establish a complete system.

What a project could evaluate

A project could assess the relevance and timing of assistance, consistency across information sources, operator control and overall task performance across the feedback loop.

↑ Top