ILLIXR plugins

This page details the structure of ILLIXR's plugins and how they interact with each other.

Default Plugins

• audio_pipeline: Launches a thread for binaural recording and one for binaural playback. Audio output is not yet routed to the system's speakers or microphone, but the plugin's compute workload is still representative of a real system. By default this plugin is enabled (see native configuration).

  Topic details:

  • Calls pose_prediction.

• debugview: Renders incoming frames from the graphics pipeline for debugging live executions of the application.

  Topic details:

  • Calls pose_prediction.
  • Asynchronously reads fast_pose on imu_raw topic. (IMU biases are unused).
  • Asynchronously reads slow_pose on slow_pose topic.
  • Synchronously reads imu on imu topic.
  • Asynchronously reads buffered cam_type on cam topic.

• gldemo: Renders a static scene (into left and right eye buffers) given the pose from pose_prediction.

  Topic details:

  • Calls pose_prediction.
  • Publishes rendered_frame on eyebuffer topic.
  • Asynchronously reads time_type on vsync_estimate topic.

• ground_truth_slam: Reads the ground truth from the same dataset as the offline_imu plugin. Ground truth data can be compared against the measurements from offline_imu for accuracy. Timing information is taken from the offline_imu measurements/data.

  Topic details:

  • Publishes pose_type on true_pose topic.
  • Asynchronously reads imu_type on imu topic.

• gtsam_integrator: Integrates over all IMU samples since the last published SLAM pose to provide a fast pose every time a new IMU sample arrives using the GTSAM library (upstream).

  Topic details:

  • Publishes imu_raw_type on imu_raw topic.
  • Synchronously reads/subscribes to imu_type on imu topic.
  • Asynchronously reads imu_integrator_input on imu_integrator_input topic.

• offline_imu: Reads IMU data files on disk, emulating a real sensor on the headset (feeds the application input measurements with timing similar to an actual IMU).

  Topic details:

  • Publishes imu_type on imu topic.

• offline_cam: Reads camera images from files on disk, emulating real cameras on the headset (feeds the application input measurements with timing similar to an actual camera).

  Topic details:

  • Publishes cam_type on cam topic.
  • pose_prediction: Uses the latest IMU value to predict a pose for a future point in time. Implements the pose_prediction service (defined in common), so poses can be served directly to other plugins.

  Topic details:

  • Asynchronously reads pose_type on slow_pose topic, but it is only used as a fallback.
  • Asynchronously reads imu_raw on imu_raw topic.
  • Asynchronously reads pose_type on true_pose topic, but it is only used if the client asks for the true pose.
  • Asynchronously reads time_type on vsync_estimate topic. This tells pose_predict what time to estimate for.

• timewarp_gl: Asynchronous reprojection of the eye buffers. The timewarp ends just after vsync, so it can deduce when the next vsync will be.

  Topic details:

  • Calls pose_prediction.
  • Asynchronously reads rendered_frame on eyebuffer topic.
  • Publishes time_type on vsync_estimate topic.
  • Publishes hologram_input on hologram_in topic.
  • Publishes texture_pose on texture_pose topic if ILLIXR_OFFLOAD_ENABLE is set in the env.

Below this point, we will use Switchboard terminology. Read the API documentation on Switchboard for more information.

• In the above figure, ovals are plugins.

• Solid arrows from plugins to topics represent publishing.

• Solid arrows from topics to plugins represent synchronous reading. Some action is taken for every event which gets published on the topic.

• Dashed arrows from topics to plugins represent asynchronous reading. Plugin readers only need the latest event on their topic.

• Imagine the topic as a trough filling with events from its publisher. Synchronous readers (AKA subscribers) drain the trough, while asynchronous readers just skim fresh events off the top of the trough.

See Writing Your Plugin to extend ILLIXR.

Other Supported Plugins

ILLIXR supports additional plugins to replace some of the default plugins.

• fauxpose: An alternate tracking implementation that simply generates "fast_pose" data from a simple mathematical algorithm (circular movement). The intent is for use when debugging other plugins and the developer wants a known pose trajectory without having to configure actual tracking.

  Topic details: - Publishes pose_position on fast_pose topic.

• hologram: Adapts the eyebuffer for use on a holographic display. By default, this plugin is disabled, since an NVIDIA GPU is currently required.

  Topic details:

  • Asynchronously reads hologram_input on hologram_in topic. Hologram is too slow to run for every input, so the plugin implements an asynchronous reader which can drop inputs.

• offload_data: Writes frames and poses output from the asynchronous reprojection plugin to disk for analysis.

  Topic details:

  • Synchronously reads texture_pose on texture_pose topic.

• open_vins: An alternate SLAM (upstream) implementation that uses a MSCKF (Multi-State Constrained Kalman Filter) to determine poses via camera/IMU.

  Topic details:

  • Publishes pose_type on slow_pose topic.
  • Publishes imu_integrator_input on imu_integrator_input topic.
  • Synchronously reads/subscribes to imu_type on imu topic.

• pose_lookup: Implements the pose_predict service, but uses ground truth from the dataset. The plugin peeks "into the future" to determine what the exact pose will be at a certain time.

  Topic details:

  • Asynchronously reads time_type on vsync_estimate topic. This tells pose_lookup what time to lookup.

• realsense: Reads images and IMU measurements from the Intel Realsense.

  Topic details:

  • Same interface as zed.

• rk4_integrator: Integrates over all IMU samples since the last published SLAM pose to provide a fast pose every time a new IMU sample arrives using RK4 integration.

  Topic details:

  • Same interface as gtsam_integrator.

• zed: Reads images and IMU measurements from the ZED Mini. Unlike offline_imu, zed additionally has RGB and depth data. Note that this plugin implements two threads: one for the camera, and one for the IMU.

  Topic details:

  • Publishes imu_type on imu topic.
  • Publishes rgb_depth_type on rgb_depth topic.

See Getting Started for more information on adding plugins to a profile file.