How to enable client-side video upscaling

The Anbox Streaming SDK offers experimental video upscaling features for client side streaming, leveraging AMD FidelityFX Super Resolution 1.0. This feature allows video to be transmitted at lower resolutions, saving bandwidth without losing too much detail. The SDK sets up a WebGL context and utilises a fragment shader based on the FSR 1.0 algorithm for post-processing, producing clearer and sharper video frames before they are displayed on the screen.

Initiate the AnboxStream object

To enable client-side video upscaling, initiate the AnboxStream object as follows:

<!DOCTYPE html>
    stream = new AnboxStream({
        targetElement: "anbox-stream",
        experimental: {
            upscaling: {
              enabled: true,
            },
        },
        ...
    });

Use target frame rate

The option experimental.upscaling.useTargetFrameRate allows canvas to use the refresh frame rate setting in the Anbox container for rendering video frames. This avoids relying on HTMLVideoElement.requestVideoFrameCallback function, which can occasionally be fired one v-sync late. This option is off by default and requires explicit activation.

Note

On a WebView where the HTMLVideoElement.requestVideoFrameCallback function is not supported, this option will be enabled by default.

Use custom fragment shader

The option options.experimental.upscaling.fragmentShader allows the use of a custom fragment shader-based upscaling algorithm for video streaming. By providing the fragment shader source code in this option, the canvas can use the custom fragment shader for video upscaling instead of the default FSR 1.0 shader. Currently, the implementation supports only single-pass rendering, which uses a single fragment shader to perform the upscaling.

When using a custom fragment shader in WebGL, you need to manage the following pre-defined variables to interact with textures and control rendering:

  • uSampler Type: sampler2D Description: this uniform represents the texture sampler for your fragment shader. It allows the shader to sample from the texture bound to texture unit 0.

  • vTextureCoord Type: varying vec2 Description: This varying variable is passed from the vertex shader to the fragment shader and represents the texture coordinates for the current fragment. It is used to sample the texture at specific coordinates.

  • uResolution Type: vec2 Description: This uniform contains the dimensions of the rendering canvas, typically set to the width and height of the canvas in pixels. It helps in normalising texture coordinates for effects like scaling and post-processing.