Class ILLIXR::xr_sensor_capture

ClassList > ILLIXR > xr_sensor_capture

ILLIXR plugin that co-captures HEVC-encoded RGB frames, depth frames, and per-sensor matched head poses on the Meta Quest 3.More...

#include <plugin.hpp>

Inherits the following classes: ILLIXR::threadloop

Classes

Type	Name
struct	capture_result
struct	latest_head_pose

Public Attributes

Type	Name
std::array< capture_result, CAPTURE_RESULT_CACHE_SIZE >	capture_result_cache_
std::mutex	capture_result_mutex_
size_t	capture_result_next_ = `0`
int64_t	clock_offset_ns_
latest_head_pose	latest_head_pose_
std::mutex	latest_head_pose_mutex_
double	ovr_time_offset_sec_ = `0.0`

Public Static Attributes

Type	Name
constexpr size_t	CAPTURE_RESULT_CACHE_SIZE = `16`
IUnityInterfaces *	s_unity_interfaces_ = `[nullptr](namespaceILLIXR.md#function-normalize)`
IUnityGraphicsVulkan *	s_vk_interface_ = `[nullptr](namespaceILLIXR.md#function-normalize)`

Public Functions

Type	Name
void	acquire_depth_unity_thread (int64_t predicted_display_time_ns, double ovr_plugin_time_sec, const float * rgb_camera_pose_lh, const float * head_pose_lh, const float * tracking_to_world_lh)
void	destroy_vulkan ()
bool	init_vulkan ()
void	release_depth_after_submit ()
void	submit_depth_readback ()
	xr_sensor_capture (const std::string & name, phonebook * pb)
	~xr_sensor_capture () override

Public Functions inherited from ILLIXR::threadloop

See ILLIXR::threadloop

Type	Name
virtual void	internal_stop () Stops the thread.
virtual void	start () override Starts the thread.
virtual void	stop () override Joins the thread.
	threadloop (const std::string & name, phonebook * pb)
	~threadloop () override

Public Functions inherited from ILLIXR::plugin

See ILLIXR::plugin

Type	Name
std::string	get_name () noexcept const
	plugin (std::string name, phonebook * pb)
void	spd_add_file_sink (const std::string & file_name, const std::string & extension, const std::string & log_level)
auto	spdlogger (const char * log_level)
virtual void	start () A method which Spindle calls when it starts the component.
virtual void	stop () A method which Spindle calls when it stops the component.
virtual	~plugin () = default

Public Static Functions

Type	Name
void	set_unity_interfaces (IUnityInterfaces * interfaces)

Protected Types inherited from ILLIXR::threadloop

See ILLIXR::threadloop

Type	Name
enum	skip_option

Protected Attributes inherited from ILLIXR::threadloop

See ILLIXR::threadloop

Type	Name
std::size_t	iteration_no = `0`
std::size_t	skip_no = `0`

Protected Attributes inherited from ILLIXR::plugin

See ILLIXR::plugin

Type	Name
const std::size_t	id_
std::string	name_
const phonebook *	phonebook_
std::shared_ptr< spdlog::logger >	plugin_logger_
const std::shared_ptr< record_logger >	record_logger_

Protected Functions

Type	Name
virtual void	_p_one_iteration () override Override with the computation the thread does every loop.
virtual skip_option	_p_should_skip () override Gets called in a tight loop, to gate the invocation of `_p_one_iteration()` __

Protected Functions inherited from ILLIXR::threadloop

See ILLIXR::threadloop

Type	Name
virtual void	_p_one_iteration () = 0 Override with the computation the thread does every loop.
virtual skip_option	_p_should_skip () Gets called in a tight loop, to gate the invocation of `_p_one_iteration()` __
virtual void	_p_thread_setup () Gets called at setup time, from the new thread.
bool	should_terminate () Whether the thread has been asked to terminate.

Detailed Description

Acquisition and encoding pipeline

RGB: Camera2 writes directly into MediaCodec's ANativeWindow input surface. The Snapdragon hardware handles NV12 -> HEVC without any CPU copy. The encoder is initialised first so its ANativeWindow can be used as the Camera2 capture target. Completed H.265 output buffers are drained on each threadloop tick.

RGB pose: acquire_depth_unity_thread() samples xrLocateSpace(VIEW,LOCAL) at 90Hz on Unity's main thread (the only valid context for VIEW space) and stores the result in latest_head_pose_ under a mutex. on_capture_completed() snapshots that pose into capture_result_cache_ at the moment of each Camera2 exposure. _p_one_iteration() reads the most recent capture result pose and attaches it to the outgoing frame.

Depth: XR_META_environment_depth OpenXR extension (standard Khronos openxr.h). Unity uses the Vulkan renderer, so depth swapchain images are VkImages. acquire_depth_unity_thread() is called from LateUpdate() via ILLIXRBridge while Unity's XR frame is open. It acquires the depth VkImage and copies it to a host-visible staging buffer via vkCmdCopyImageToBuffer, then maps the buffer to read the R16F pixels to CPU. Vulkan handles are obtained from IUnityGraphicsVulkan via UnityPluginLoad(). Depth format: R16F (float16), 2 bytes/pixel, top-down, inverted NDC. depth_m = near_z / r_value.

Initialisation order

UnityPluginLoad() - captures IUnityGraphicsVulkan (called by Unity at startup) * init_openxr() - reuses Unity's XrInstance/XrSession, loads depth extension * init_vulkan() - creates staging buffer, command pool, fence from Unity's VkDevice * init_encoder() - MediaCodec HEVC, creates input ANativeWindow * init_camera() - Camera2, targeting encoder's ANativeWindow

Configuration (env vars, set via Unity JNI bridge before plugin start)

ILLIXR_CAPTURE_FPS Encoder target FPS hint (default: 2) ILLIXR_ENCODER_BITRATE_BPS HEVC bitrate in bits/s (default: 5000000) ILLIXR_ENCODER_IFRAME_SEC IDR period in seconds (default: 1) ILLIXR_DEPTH_DISABLED Set to 1 to skip depth (default: 0) ILLIXR_MAX_DEPTH_M Far-depth cap in metres (default: 0) ILLIXR_CAMERA_WIDTH Camera capture width (default: 960) ILLIXR_CAMERA_HEIGHT Camera capture height (default: 960)

Publishes: "semantic_frame" (ILLIXR::data_format::semantic_frame)

Public Attributes Documentation

variable capture_result_cache_

std::array<capture_result, CAPTURE_RESULT_CACHE_SIZE> ILLIXR::xr_sensor_capture::capture_result_cache_;

variable capture_result_mutex_

std::mutex ILLIXR::xr_sensor_capture::capture_result_mutex_;

variable capture_result_next_

size_t ILLIXR::xr_sensor_capture::capture_result_next_;

variable clock_offset_ns_

int64_t ILLIXR::xr_sensor_capture::clock_offset_ns_;

variable latest_head_pose_

latest_head_pose ILLIXR::xr_sensor_capture::latest_head_pose_;

variable latest_head_pose_mutex_

std::mutex ILLIXR::xr_sensor_capture::latest_head_pose_mutex_;

variable ovr_time_offset_sec_

double ILLIXR::xr_sensor_capture::ovr_time_offset_sec_;

Public Static Attributes Documentation

variable CAPTURE_RESULT_CACHE_SIZE

constexpr size_t ILLIXR::xr_sensor_capture::CAPTURE_RESULT_CACHE_SIZE;

variable s_unity_interfaces_

IUnityInterfaces * xr_sensor_capture::s_unity_interfaces_;

variable s_vk_interface_

IUnityGraphicsVulkan * xr_sensor_capture::s_vk_interface_;

Public Functions Documentation

function acquire_depth_unity_thread

void ILLIXR::xr_sensor_capture::acquire_depth_unity_thread (
    int64_t predicted_display_time_ns,
    double ovr_plugin_time_sec,
    const  float * rgb_camera_pose_lh,
    const  float * head_pose_lh,
    const  float * tracking_to_world_lh
)

function destroy_vulkan

void ILLIXR::xr_sensor_capture::destroy_vulkan ()

function init_vulkan

bool ILLIXR::xr_sensor_capture::init_vulkan ()

function release_depth_after_submit

void ILLIXR::xr_sensor_capture::release_depth_after_submit ()

function submit_depth_readback

void ILLIXR::xr_sensor_capture::submit_depth_readback ()

function xr_sensor_capture

ILLIXR::xr_sensor_capture::xr_sensor_capture (
    const std::string & name,
    phonebook * pb
)

function ~xr_sensor_capture

ILLIXR::xr_sensor_capture::~xr_sensor_capture () override

Public Static Functions Documentation

function set_unity_interfaces

static void ILLIXR::xr_sensor_capture::set_unity_interfaces (
    IUnityInterfaces * interfaces
)

Protected Functions Documentation

function _p_one_iteration

Override with the computation the thread does every loop.

virtual void ILLIXR::xr_sensor_capture::_p_one_iteration () override

This gets called in rapid succession.

Implements ILLIXR::threadloop::_p_one_iteration

function _p_should_skip

Gets called in a tight loop, to gate the invocation of _p_one_iteration() __

virtual skip_option ILLIXR::xr_sensor_capture::_p_should_skip () override

Implements ILLIXR::threadloop::_p_should_skip

The documentation for this class was generated from the following file /home/friedel/devel/ILLIXR/illixr_release/ILLIXR/plugins/quest3/unity/plugin.hpp