Original data observed from sensors is rarely useful for taking actions. It has to be refined through interpretation to make it suitable for explaining a scene or determining the next action.
This is a very simple example of the kind of explaining that we perform to guide action.
Open the slider full-screen. Drag the slider or press space to play. Drag in the scene to orbit, scroll to zoom.
Observation → Interpretation → Explanation
The observation is a camera frame from an OAK-D SR, together with depth information. Even at this point there is some interpretation needed to turn a stereo camera pair into a depth field.
The interpretation applies a label to each point (probably part of the robot, probably a cube, probably the foam), then replaces the points with the models that explain them: the arm's CAD posed from its encoders, boxes fitted to the cubes, a plane fitted to the foam. This allows the world to be abstracted into objects that are relevant in the scene.
The abstract world can then be used to explain elements of the scene to a reasoning agent.
These layers of interpretation and their tradeoffs are not new for robotics, AI, or even for computing. However they've become essential as we attach reasoning agents to the robots. The more of this work that is done by relatively straightforward computer vision code, rather than by agentic systems, the faster everything goes.