ILLIXR plugins

This page details the structure of ILLIXR's plugins and how they interact with each other. Each plugin is labeled with the OSs they are supported on:

Logo	OS
	Linux
	Windows
	Android

Ada

Ada’s distributed design relies on four communication plugins that coordinate data transfer between the device and server for remote scene provisioning.

• ada.device_rx: Receives processed data from the server to the device.
  • Asynchronously reads a string from the topic ada_processed, which contains protobuf packets sent by the server.
  • Publishes mesh_type to compressed_scene topic. forwards compressed mesh chunks to ada.mesh_decompression_grey
  • Publishes vb_type to VB_update_lists topic. forwards the unique voxel block list (UVBL) to ada.scene_management
• ada.device_tx: Sends encoded depth images (MSB and LSB) along with non-encoded pose information from the device to the server. The LSB encoding bitrate is configurable for bandwidth–reconstruction accuracy trade-offs.
  • Synchronously reads scene_recon_type from ScanNet_Data topic, which provides dataset input (via ada.offline_scannet).
  • Publishes encoded depth data as a string to the ada_data topic.
• ada.server_rx: Receives encoded depth images and poses (not encoded) from the device, decodes them, and feeds them to the reconstruction module (ada.infiniTAM).
  • Asynchronously reads a string from topic ada_data
  • Publishes scene_recon_type to ScanNet_Data topic. provides decoded depth frames and corresponding pose inforamtion to downstream reconstruction.
• ada.server_tx: Sends processed scene data (meshes and Unique Voxel Block Lists(UVBL)) from the server back to the device.
  • Synchronously reads vb_type from unique_VB_list topic (output of ada.infiniTAM).
  • Synchronously reads mesh_type from compressed_scene topic (output of ada.infiniTAM).
  • Publishes a string to ada_processed topic, each string is either a protobuf for vb_type topic or a protobuf for mesh_type topic.

  Details    Code

ada.infinitam

Performs scene reconstruction using incoming depth and pose data from the device, followed by on-demand or proactive scene extraction. During extraction, it generates both the updated partial mesh and the Unique Voxel Block List (UVBL), which are sent downstream for compression and scene management. Extraction frequency is configurable to balance latency and compute cost.

Topic details:

• Synchronously reads scene_recon_type from ScanNet_Data topic.
• Publishes mesh_type to requested_scene topic. Extracted mesh chunks for compression
• Publishes vb_type to unique_VB_list topic. This is the metadata for identifying updated voxel regions

  Details    Code

ada.mesh_compression

Compresses mesh chunks from ada.infinitam using a customized version of Google Draco. Compression parallelism can be tuned for different latency–power trade-offs.

Topic details:

• Synchronously reads mesh_type from requested_scene topic.
• Publishes mesh_type to compressed_scene topic. compressed mesh chunks ready for transmission to the device. Voxel block information has been attached to each encoded face.

  Details    Code

ada.mesh_decompression_grey

Decompresses the mesh chunks received from the server and performs a portion of scene management that can be parallelized. Decompression parallelism can be tuned for different latency–power trade-offs.

Topic details:

• Synchronously reads mesh_type from compressed_scene topic.
• Publishes draco_type to decoded_inactive_scene topic. decoded mesh data sent to ada.scene_management.

  Details    Code

ada.offline_scannet

Loads the ScanNet dataset for offline or reproducible experiments.

Topic details:

• Publishes scene_recon_type to ScanNet_Data topic.

  Details    Code

ada.scene_management

Integrates incremental scene updates into a maintained global mesh, merging new geometry and removing outdated regions for consistency.

Topic details:

• Synchronously reads draco_type from decoded_inactive_scene topic.
• Synchronously reads vb_type from VB_update_lists topic.

  Details    Code

android_data

Reads imu_type and binocular_cam_type from files on the Android device and publishes them to the switchboard.

Topic details:

• Publishes imu_type to imu topic.
• Publishes binocular_cam_type to cam topic.

  Details    Code

audio_pipeline

Launches a thread for binaural: recording and one for binaural playback. Audio output is not yet routed to the system's speakers or microphone, but the plugin's compute workload is still representative of a real system. By default, this plugin is enabled (see native configuration).

Topic details:

• Calls pose_prediction

  Details    Code

boba_immersive

Topic details:

• Asynchronously reads controller_input from quest_controller topic.
• Asynchronously reads openxr_view_frame from openxr_view topic.
• Publishes stereo_frame to stereo_frame topic.

  Details    Code

boba_streaming_server

Topic details:

• Asynchronously reads stereo_frame from stereo_frame topic.
• Publishes compressed_frame to compressed_frames topic.
• Publishes boba_modal_texture to boba_modal_texture topic.

  Details    Code

debugview

Renders incoming frames from the graphics pipeline for debugging live executions of the application.

Topic details:

• Calls pose_prediction
• Asynchronously reads fast_pose_type from imu_raw topic. (IMU biases are unused).
• Asynchronously reads pose_type from slow_pose topic.
• Asynchronously reads rgb_depth_type from rgb_depth topic.
• Asynchronously reads buffered binocular_cam_type from cam topic.

  Details    Code

depthai

Enables access to the DepthAI library.

Topic details:

• Publishes imu_type to imu topic
• Publishes binocular_cam_type to cam topic`
• Publishes rgb_depth_type to rgb_depth topic

  Details    Code

gldemo ¹

Renders a static scene (into left and right eye buffers) given the pose from pose_prediction.

Topic details:

• Calls pose_prediction
• Publishes rendered_frame to eyebuffer topic.
• Publishes image_handle to image_handle topic.
• Asynchronously reads time_point from vsync_estimate topic.

  Details    Code

ground_truth_slam

Reads the ground truth from the same dataset as the offline_imu plugin. Ground truth data can be compared against the head tracking results (e.g. from VIO, IMU integrator, or pose predictor) for accuracy. Timing information is taken from the offline_imu measurements/data.

Topic details:

• Publishes pose_type to true_pose topic.
• Publishes Eigen::Vector3f to ground_truth_offset topic.
• Asynchronously reads imu_type from imu topic.

  Details    Code

gtsam_integrator

Integrates over all IMU samples since the last published visual-inertial pose to provide a fast pose every time a new IMU sample arrives using the GTSAM library (upstream).

Topic details:

• Publishes imu_raw_type to imu_raw topic.
• Synchronously reads imu_type from imu topic.
• Asynchronously reads imu_integrator_input to imu_integrator_input topic.

  Details Code

hand_tracking

Detects and identifies hands in an image, CPU based calculations. The output from this plugin can be used to track hand movements and recognize hand gestures.

Topic details:

• Synchronously reads one of monocular_cam_type from webcam topic, binocular_cam_type from cam topic, or cam_type_zed from cam_zed topic. This is selectable at run time via an environment variable.
• Asynchronously reads camera_data from cam_data topic, only once as values are static
• If reading from webcam
  • Asynchronously reads pose_type from pose topic
  • Asynchronously reads one of depth_type from depth topic or rgb_depth_type from rgb_depth topic, depending on which is available
• If reading from cam
  • Asynchronously reads pose_type from pose topic
  • Asynchronously reads one of depth_type from depth topic or rgb_depth_type from rgb_depth topic, if either is available, but not required
• If reading from cam_zed, no additional data are required.
• Publishes ht_frame to ht topic.

  Details    Code

hand_tracking_gpu

Detects and identifies hands in an image, GPU based calculations. The output from this plugin can be used to track hand movements and recognize hand gestures. This plugin is currently experimental.

Topic details:

• Synchronously reads one of monocular_cam_type from webcam topic, binocular_cam_type from cam topic, or cam_type_zed from cam_zed topic. This is selectable at run time via an environment variable.
• Asynchronously reads camera_data from cam_data topic, only once as values are static
• If reading from webcam
  • Asynchronously reads pose_type from pose topic
  • Asynchronously reads one of depth_type from depth topic or rgb_depth_type from rgb_depth topic, depending on which is available
• If reading from cam
  • Asynchronously reads pose_type from pose topic
  • Asynchronously reads one of depth_type from depth topic or rgb_depth_type from rgb_depth topic, if either is available, but not required
• If reading from cam_zed, no additional data are required.
• Publishes ht_frame to ht topic.

  Details    Code

hand_tracking.viewer

Reads the output of the hand_tracking plugin and displays the results on the screen. This is most useful for debugging. The capabilities of this plugin will be merged into the debugview plugin in the future.

Topic details:

• Synchronously reads ht_frame from ht topic.

  Details    Code

lighthouse

Enables lighthouse tracking using the libsurvive library

Topic details:

• Publishes pose_type to slow_pose topic.
• Publishes fast_pose_type to fast_pose topic.

  Details    Code

native_renderer

Constructs a full rendering pipeline utilizing several ILLIXR components.

Topic details:

• Calls pose_prediction
• Calls vulkan::display_provider
• Calls vulkan::timewarp
• Calls vulkan::app
• Synchronously reads time_point from vsync_estimate topic.

  Details    Code

network_latency

This set of plugins periodically measures round-trip network latency between client and server. One way latency can be measured between devices running the same operating system and with well correlated clocks.

Topic details:

• Publishes network_latency_result to network_latency topic

  Details    Code

offline_cam

Reads camera images from files on disk, emulating real cameras on the headset (feeds the application input measurements with timing similar to an actual camera).

Topic details:

• Publishes binocular_cam_type to cam topic.

  Details    Code

offline_imu

Reads IMU data files on disk, emulating a real sensor on the headset (feeds the application input measurements with timing similar to an actual IMU).

Topic details:

• Publishes imu_type to imu topic.

  Details    Code

offload_data

Writes frames and poses output from the asynchronous reprojection plugin to disk for analysis.

Topic details:

• Synchronously reads texture_pose to texture_pose topic.

  Details    Code

offload_rendering_client

Receives encoded frames from the network, sent by offload_rendering_server

Topic details:

Linux: - Calls vulkan::display_provider - Calls pose_prediction - Asynchronously reads compressed_frame from compressed_frames topic. - Publishes fast_pose_type to render_pose topic. - Asynchronously reads network_latency_result from network_latency

Android: - Asynchronously reads compressed_frame from compressed_frames topic. - Publishes dual_frames to unity_rendered_frame topic. - Asynchronously reads network_latency_result from network_latency

  Details    Code

offload_rendering_server

Encodes and transmits frames to one of the offload_rendering_clients.

Topic details:

• Calls vulkan::display_provider
• Asynchronously reads fast_pose_type from render_pose_ topic.
• Publishes compressed_frame to compressed_frames topic.

  Details    Code

offload_vio

Four plugins which work in unison to allow head tracking (VIO) to be rendered remotely.

Topic details:

• offload_vio.device_rx
  • Asynchronously reads a string from topic vio_pose.
  • Synchronously reads imu_type from imu topic
  • Publishes pose_type to slow_pose topic.
  • Publishes imu_integrator_input to imu_integrator_input topic.
• offload_vio.device_tx
  • Asynchronously reads binocular_cam_type from cam topic
  • Publishes a string to compressed_imu_cam topic
• offload_vio.server_rx
  • Asynchronously reads a string from compressed_imu_cam topic
  • Publishes imu_type to imu topic.
  • Publishes binocular_cam_type to cam topic.
• offload_vio.server_tx
  • Asynchronously reads imu_integrator_input from imu_integrator_input topic.
  • Synchronously reads pose_type from slow_pose topic from open_vins
  • Publishes a string to vio_pose topic.

  Details    Code

openni

Enables an interface to the Openni algorithms.

Topic details:

• Publishes rgb_depth_type to rgb_depth topic.

  Details    Code

open_vins

An alternate head tracking (upstream) implementation that uses a MSCKF (Multi-State Constrained Kalman Filter) to determine poses via camera/IMU.

Topic details:

• Publishes pose_type on slow_pose topic.
• Publishes imu_integrator_input on imu_integrator_input topic.
• Synchronously reads/subscribes to imu_type on imu topic.

  Details    Code

openxr_interface

Connects to the OpenXR runtime on an Android device, gathers head and hand poses, sending them over the network. Receives decoded frames and puts them in the OpenXR swapchain for display.

Topic details:

• Publishes [combined_pose][A??] to combined_pose topic
• Asynchronously reads network_latency_result from `network_latency topic
• Asynchronously reads dual_frames from unity_rendered_frames topic

  Details    Code

openwarp_vk

Provides a Vulkan-based reprojection service.

Topic details:

• Calls vulkan::timewarp
• Calls pose_prediction

  Details    Code

orb_slam3

Utilizes the ORB_SLAM3 library to enable real-time head tracking.

Topic details:

• Asynchronously reads binocular_cam_type from cam topic.
• Synchronously reads/subscribes to imu_type on imu topic.
• Publishes pose_type to slow_pose topic.
• Publishes imu_integrator_input to imu_integrator_input topic.

  Details    Code

passthrough_integrator

Provides IMU integration.

Topic details:

• Asynchronously reads imu_integrator_input from imu_integrator_input topic.
• Synchronously reads imu_type from imu topic.
• Publishes imu_raw_type to imu_raw topic.

  Details    Code

quest3.controller

Provides an interface to the Quest 3 hand controller

Topic details:

• Publishes quest_controller_input to quest_controller topic.
• Publishes openxr_view_frame to openxr_view topic.
• Asynchronously reads stereo_frame from stereo_frame topic.

  Details    Code

quest3.unity

Captures camera and depth images from a Quest 3 headset, encodes them, and sends them over the network to a server. The camera images are captured via standard Android APIs. The depth images have to come over OpenXR, so the plugin hooks into the Unity app and uses it to acquire the depth images.

Topic details:

• Publishes [semantic_frame][A30] tosemantic_frame` topic

  Details    Code

realsense

Reads images and IMU measurements from the Intel Realsense.

Topic details:

• Publishes imu_type to imu topic.
• Publishes binocular_cam_type to cam topic.
• Publishes rgb_depth_type to rgb_depth topic.

  Details    Code

record_imu_cam

Writes imu_type and binocular_cam_type data to disk.

Topic details:

• Asynchronously reads binocular_cam_type from cam topic.
• Synchronously reads imu_type from imu topic.

  Details    Code

record_rgb_depth

Writes rgb_depth_type data to disk.

Topic details:

• Synchronously reads rgb_depth_type from rgb_depth topic.

Details Code

rk4_integrator

Integrates over all IMU samples since the last published visual-inertial pose to provide a fast pose every time a new IMU sample arrives using RK4 integration.

Topic details:

• Asynchronously reads imu_integrator_input from imu_integrator_input topic.
• Synchronously reads imu_type from imu topic.
• Publishes imu_raw_type to imu_raw topic.

  Details    Code

semantic_python

Provides reader and writer handles to a Python script, allowing the script to access the switchboard directly.

Topic details:

• Asynchronously reads semantic_frame from semantic_frame topic
• Asynchronously reads voice_query from semantic_query topic
• Publishes query_response to semantic_response networked topic

  Details    Code

tcp_network_backend

Provides network communications over TCP.

  Details    Code

timewarp_gl ¹

Asynchronous reprojection of the eye buffers. The timewarp ends right before vsync, so it can deduce when the next vsync will be.

Topic details:

• Calls pose_prediction
• Publishes hologram_input to hologram_in topic.
• If using Monado
  • Asynchronously reads rendered_frame on eyebuffer topic, if using Monado.
  • Publishes time_point to vsync_estimate topic.
  • Publishes texture_pose to texture_pose topic if ILLIXR_OFFLOAD_ENABLE is set in the env.
• If not using Monado
  • Publishes signal_to_quad to signal_quad topic.

  Details    Code

timewarp_vk

Asynchronous reprojection of the eye buffers. The timewarp ends right before vsync, so it can deduce when the next vsync will be.

Topic details:

• Calls vulkan::timewarp
• Calls pose_prediction
• Asynchronously reads time_point from vsync_estimate topic.

  Details    Code

udp_network_backend

Provides network communications over UDP.

  Details    Code

webcam

Uses a webcam to capture images for input into the hand_tracking plugin. This plugin is useful for debugging and is not meant to be used in a production pipeline.

Topic details:

• Publishes monocular_cam_type to webcam topic.

  Details    Code

zed

Reads images and IMU measurements from the ZED Mini. Unlike offline_imu, zed additionally has RGB and depth data.

Note

This plugin implements two threads: one for the camera, and one for the IMU.

Topic details:

• Publishes imu_type to imu topic.
• Publishes binocular_cam_type to cam topic.
• Publishes rgb_depth_type to rgb_depth topic.
• Publishes camera_data to cam_data topic.
• Publishes cam_type_zed on cam_zed topic.

  Details    Code

zed.data_injection

Reads images and pose information from disk and publishes them to ILLIXR.

Topic details:

• Publishes binocular_cam_type to cam topic
• Publishes pose_type to pose topic.
• Publishes camera_data to cam_data topic.

  Details    Code

Below this point, we will use Switchboard terminology. Read the API documentation on Switchboard for more information.

Current dataflow between all plugins and services. Dark blue boxes represent plugins and cyan component boxes represent services. Data types are represented with cylinders and labelled as topic <data_type>. Service data types are represented by octagons. Solid lines point from the plugin/service which publishes them to the data type. Dashed lines point from data types to the plugin/service which reads them synchronously. Dotted lines point from data types to the plugin/service which reads them asynchronously.

See Writing Your Plugin to extend ILLIXR.

Plugin Interdependencies

Some plugins require other plugins to be loaded in order to work. The table below gives a listing of the plugin interdependencies.

Plugin	Requires	Provided by plugin
debugview	pose_prediction	fauxpose, pose_lookup, pose_prediction
gldemo	pose_prediction	fauxpose, pose_lookup, pose_prediction
native_renderer	app	vkdemo
	display_sink	display_vk
	pose_prediction	fauxpose, pose_lookup, pose_prediction
	timewarp	timewarp_vk
timewarp_gl	pose_prediction	fauxpose, pose_lookup, pose_prediction
timewarp_vk	display_sink	display_vk
	pose_prediction	fauxpose, pose_lookup, pose_prediction
vkdemo	display_sink	display_vk

See Getting Started for more information on adding plugins to a profile file.

---

1. ILLIXR has switched to a Vulkan back end, thus OpenGL-based plugins may not work on every system. ↩↩