- PHYSICS SIMULATION UPDATE
- PHYSICS SIMULATION SOFTWARE
The computation necessary for training the agent (one simulation step, followed by one update of the agent’s neural network) is overshadowed by the computation spent packaging the data (i.e., marshalling data within the engine, then into a wire format such as protobuf, then into TCP buffers, and then undoing all these steps on the learner side).Were this experience instead already on the same device as the learner’s neural network, latency would drop to <1 nanosecond. Because the data must travel over the network within a datacenter, the learner must wait for 10,000+ nanoseconds to fetch experience from the simulator.
This layout imposes an enormous latency bottleneck. As we noted above, a typical robotics learning pipeline places a single learner in a tight feedback with many simulations in parallel, but upon analyzing this architecture, one finds that: Taking a closer look at how most physics simulation engines are designed today, there are a few large opportunities to improve efficiency. For each simulation timestep, forces and torques are integrated together to update the positions, rotations, and velocities of each physical body. Simulation of three spherical bodies, a wall, two joints, and one actuator. In order to accurately model such systems, simulators integrate forces from gravity, motor actuation, joint constraints, object collisions, and others to simulate the motion of a physical system across time. Rigid body physics are used in video games, robotics, molecular dynamics, biomechanics, graphics and animation, and other domains. Physics Simulation Engine Design Opportunities The right shows the Brax flow: learning and large batch simulation occur side by side on a single CPU/GPU chip. The middle shows a typical distributed simulation flow: training takes minutes by farming simulation out to thousands of machines. The left shows a typical workstation flow: on a single machine, with the environment on CPU, training takes hours or days. Using this new platform, we demonstrate 100-1000x faster training compared to a traditional workstation setup. We’ve open sourced the engine along with reference RL algorithms and simulation environments that are all accessible via Colab. The engine is designed to both efficiently run thousands of parallel physics simulations alongside a machine learning (ML) algorithm on a single accelerator and scale millions of simulations seamlessly across pods of interconnected accelerators. In “ Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation”, we present a new physics simulation engine that matches the performance of a large compute cluster with just a single TPU or GPU. These distributed simulation platforms yield impressive results that train very quickly, but they must run on compute clusters with thousands of CPUs or GPUs which are inaccessible to most researchers. While progress has been made to improve training efficiency by recycling simulation frames, some RL tools instead sidestep this problem by distributing the generation of simulation frames across many simulators. But this poses a challenge: it frequently takes millions to billions of simulation frames for an RL agent to become proficient at even simple tasks, such as walking, using tools, or assembling toy blocks. In order to facilitate the collection of training data in a practical amount of time, RL usually leverages simulation, where approximations of any number of complex objects are composed of many rigid bodies connected by joints and powered by actuators. Reinforcement learning (RL) is a popular method for teaching robots to navigate and manipulate the physical world, which itself can be simplified and expressed as interactions between rigid bodies 1 (i.e., solid physical objects that do not deform when a force is applied to them). PHYSICS SIMULATION SOFTWARE
Daniel Freeman, Senior Software Engineer and Erik Frey, Staff Software Engineer, Google Research
0 kommentar(er)
