CRESSim is mainly developed with Unity and NVIDIA PhysX 5. The fundamental building block is the “PhysX 5 for Unity” plugin. The plugin allows using PhysX 5 as the physics engine and adding rigid bodies, soft bodies, cloth, fluid, and kinematic trees (and robots) into Unity. Based on PhysX 5 for Unity, PSM robots are simulated as articulation robots, and ML-Agents is then used for defining RL environments. ROS-Unity integration is further realized to allow ROS communication between Unity and the dVRK software, which allows the teleoperation of the simulated robot and collecting human demonstrations.
The plugin contains native libraries (written in C++) for Windows and Linux that provide the APIs to use PhysX 5 in Unity. It also provides some functionalities for robotics, including forward and inverse kinematics. On the Unity side (C#), components are implemented based on the native APIs to allow the easy creation of simulated objects in Unity, including rigid bodies, FEM soft bodies, PBD cloth and inflatables, PBD fluid, and kinematic trees with articulation joints (including robots). For details about PhysX 5, please refer to NVIDIA’s documentation.
Since PBD fluid is used, fluid rendering is also handled in this part. A typical screen-space rendering technique is used, with a narrow-range filter. In short, each particle is drawn in screen space as an ellipsoid to obtain a fluid surface, after which the filter is applied to smooth it. We also assign a color value to each particle and achieve fluid color mixing effects by diffusing the particle colors. In total, 3 passes are needed before rendering the final scene: one depth pass to extract the fluid surface in screen space, one filtering pass to smooth the surface, and one color pass to extract the surface colors. The final scene is then rendered using Unity’s regular forward rendering path in a per-pixel manner.
This is a relatively simple and rough implementation that wraps around commonly used functionalities in PhysX 5 to add/remove PhysX objects, such as scenes (PxScene) and actors (PxActor). Collision filtering and modification have not been added. Some scene query features are also used but only because they make implementing the Unity side easier.
To achieve communication between Unity and dVRK, we follow the steps at Unity-Robotics-Hub. The dVRK software is built with the ROS-TCP-Endpoint package. On Unity’s side, the package ROS-TCP-Connector is used. DVRK-specific ROS messages are generated (we are using ROS 1 with dVRK 2.1.0).
Coroutines are extensively used in Unity’s scripts due to the need to wait for sending/receiving ROS messages across multiple frames. So far, only the MTM has been integrated and only basic functionalities have been implemented, such as homing and reading/setting poses. Also, note that Unity uses a left-handed coordinate system.
