Learning Physical Principles from Interaction: Self-Evolving Planning via Test-Time Memory

Researchers Haoyang Li, Yang You, Hao Su, and Leonidas Guibas developed PhysMem, a memory framework that enables robot planners to learn physical principles from interaction during test time, without updating model parameters, in a paper submitted to arXiv on February 23, 2026. The research addresses a critical limitation in vision-language model (VLM) planners, which can reason about physical properties like friction and stability in general terms but often fail when predicting how specific objects will interact with particular environments without direct experience. PhysMem represents a significant advancement in robotic manipulation capabilities by creating a system that records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. The central innovation of PhysMem lies in its verification-before-application approach, where the system tests hypotheses against new observations rather than applying retrieved experience directly. This design reduces rigid reliance on prior experience when physical conditions change, allowing robots to adapt to novel situations. The researchers evaluated their framework on three real-world manipulation tasks and simulation benchmarks across four VLM backbones. Results demonstrated that on a controlled brick insertion task, principled abstraction achieved a 76% success rate compared to only 23% for direct experience retrieval. Real-world experiments further showed consistent performance improvements over 30-minute deployment sessions, demonstrating the practical value of this approach. This research contributes to the growing field of embodied AI by creating a more flexible and adaptable robotic planning system. By enabling robots to learn from their interactions during deployment rather than requiring extensive pre-training or model updates, PhysMem opens new possibilities for autonomous systems operating in dynamic, unstructured environments. The framework's ability to abstract physical principles from specific experiences represents a step toward more generalizable robotic intelligence that can transfer knowledge across different scenarios while remaining responsive to immediate environmental conditions.