Rethinking AI Architecture for Physical Robots

Anna Jey published a Medium article proposing an architecture for embodied AI agents that addresses a fundamental mismatch: applying chatbot-style language models directly to physical robots. The piece argues that robots operating in the real world require different design patterns than conversational AI systems.

The Core Problem

The article identifies a common mistake in robotics AI: treating physical agents as if they were text-based chatbots. Language models excel at generating coherent responses to prompts, but robots must perceive environments through sensors, maintain spatial awareness, and execute actions with real-world consequences. Simply wrapping an LLM in a robot body ignores these requirements.

According to the piece, embodied agents need architectures that account for continuous sensory input, physical state tracking, and the temporal nature of actions. A chatbot can respond instantly to text, but a robot arm moving an object takes time and must handle interruptions, collisions, and changing conditions.

Proposed Architecture Components

The article outlines several architectural elements for embodied AI:

Perception Module: Processes sensor data (cameras, LIDAR, touch sensors) into representations the agent can reason about. This differs from chatbot input processing, which handles discrete text tokens.

World Model: Maintains an internal representation of the environment and the agent’s physical state. This component tracks object positions, spatial relationships, and the agent’s own configuration - information a text-only system never encounters.

Action Planner: Translates high-level goals into sequences of physical actions. Unlike generating a text response, this requires understanding physics constraints, timing, and the reversibility (or irreversibility) of actions.

Execution Layer: Handles low-level motor control and monitors action completion. This layer bridges the gap between planned actions and actual hardware commands.

Implementation Considerations

The article suggests separating reasoning from execution. Language models can participate in high-level planning (“pick up the red block”), but shouldn’t directly control motors. A dedicated execution layer translates plans into hardware commands and provides feedback when actions succeed or fail.

Code integration might look like:

plan = reasoning_module.generate_plan(goal, perception_data)
execution_result = motor_controller.execute(plan)

The architecture emphasizes feedback loops. Physical actions produce observable results that must update the world model, creating a cycle absent in chatbot interactions.

Challenges and Constraints

The piece acknowledges several limitations. Real-time processing requirements strain current LLMs, which weren’t designed for millisecond-level responsiveness. Sensor noise and environmental unpredictability create challenges that clean text datasets never present.

Safety becomes critical when AI controls physical actuators. The article notes that hallucinations - acceptable quirks in chatbots - become dangerous when they cause a robot to misidentify obstacles or misjudge forces.

Integration complexity also increases. Embodied systems require coordinating vision models, language understanding, path planning algorithms, and control systems - a significantly more complex stack than a single language model serving text responses.

Practical Outlook

The article positions this architecture as a starting framework rather than a complete solution. Developers building physical AI agents should recognize that chatbot architectures omit essential components for embodied intelligence.

The separation of concerns - perception, world modeling, planning, and execution as distinct modules - allows each component to use appropriate techniques. Vision models handle sensors, physics engines inform planning, and control theory manages motors, while language models contribute to high-level reasoning where they excel.

This modular approach acknowledges that no single model type currently handles all aspects of embodied AI effectively. The architecture aims to combine specialized components rather than forcing language models to perform tasks they weren’t designed for.