Skip to content

offload_rendering_server

Summary

offload_rendering_server encodes frames using FFMPEG (on Linux, this is also the default) and NVENC (on Windows and Linux) and transmits them to the client with the network_backend from switchboard. In addition to the frame, some additional information is also transmitted, e.g., the pose used for the rendered frame so that the client can reprojected the decoded frame accordingly.

Relevant environment variables include: - ILLIXR_USE_DEPTH_IMAGES set to non-zero will encode/transmit depth images. - ILLIXR_OFFLOAD_RENDERING_BITRATE sets the encoding bitrate. - ILLIXR_OFFLOAD_RENDERING_FRAMERATE sets the encoding framerate. - ILLIXR_OFFLOAD_RENDERING_NALU_ONLY set to non-zero indicates a Jetson client.

When the client is an Android device with its own OpenXR runtime (e.g., Quest 3) you can enable overscanning to allow the onboard timewarp to produce complete images for display. To enable this feature set the ILLIXR_OVERSCAN environment variable in main.cpp to the scale factor you want to use. Testing has found a value of 1.1 works well for the Quest 3. You will also need to supply the same environment variable and value on the server side.

Please refer to the README in network backends for setting the server and client IP address and port number.

NVENC Encoding

NVENC encoding uses the onboard encoders of NVIDIA GPU cards. It is intended to be used when the offloading client is an Android device. ILLIXR uses either AV1 or HEVC encoding with this method. The GeForce RTX 4090 and 5090 cards are known to work with both encodings, while the GeForce RTX 3090 card only supports HEVC encoding. Using the NVENC encoder requires a few additional components installed, that must be manually done. It requires an NVIDIA GPU running CUDA 12.8+ or 13.0+. The following need to be installed on your machine:

  • CUDA toolkit installed: this can be downloaded from here
  • NVIDIA video codec SDK: this can be downloaded from here; this SDK is just some headers, so when you unzip the archive, note the location of the files.

Note

Please be sure to match the version of the SDK's with the CUDA version and NVIDIA driver versions. For the NVIDIA SDK there is a system requirements section on the page stating the minimum driver version. Older version of the NVIDIA SDK can be found under the "Additional Resources" section on the page.

To enable NVENC encoding can be enabled with the -DUSE_NVENC_ENCODER=ON CMake argument. The different encoding schemes can be selected with the -DUSE_AV1_ENCODING and -DUSE_HEVC_ENCODING CMake arguments, setting one to ON and the other OFF.

Since this is intended to be used with Android based clients with an OpenXR runtime it is recommended to also use the -DUSE_OPENXR_DEFINITIONS=ON and -DUSE_COMBINED_ENCODING=ON CMake arguments. In addition to receiving head poses from an OpenXR based client, you can also receive hand poses and hand interactions. To enable the use of these data on the server use the ILLIXR_USE_HAND_TRACKING and ILLIXR_USE_HAND_INTERACTIONS environment variables, set to 1, when launching the server. The hand based data will only be useful if your OpenXR server app requests them.

Note

Note that at the moment, the offload_rendering_server only supports Monado + OpenXR apps, and does not offload the native demos.

Windows builds

The Windows version of the offload_rendering_server has only been tested with an Android client.