Skip to main content
Version: Next

Custom sources on the web Web

note

This guide is exclusively for Web (React) applications. For React Native, see Custom sources in React Native.

This guide demonstrates how to stream non-standard video or audio to other peers in your web app. The utilities in this section allow you to integrate Fishjam with powerful browser APIs such as WebGL and WebGPU, or higher level libraries, which leverage these APIs, such as Three.js, Smelter or PixiJS. For a complete WebGPU render-and-publish pipeline, see WebGPU effects.

Creating a custom source - useCustomSource()​

To create a custom source, you only need to do two things:

  1. Call the useCustomSource hook.
  2. Call the setStream callback provided by the useCustomSource hook with a MediaStream object.

Usage Example​

import React, { useEffect } from "react";
import { useCustomSource } from "@fishjam-cloud/react-client";

const stream: MediaStream | null = null; // Replace with your MediaStream object
const { setStream } = useCustomSource("my-custom-source");

useEffect(() => {
  setStream(stream);
}, [stream, setStream]);

Using a created custom source​

Once you have called setStream for a given source ID (in the above example, "my-custom-source"), any subsequent calls to useCustomSource with the same ID will return the same state. This enables multiple components to control and read a shared custom source.

Deleting a custom source​

If you wish to remove a custom source, then you should call the setStream callback with null as its argument.

Usage Example​

const { setStream } = useCustomSource("my-custom-source");
// ...
await setStream(null);

How to get a MediaStream object?​

Depending on your use-case, the way you can obtain a MediaStream will vary. Below are some common examples of how to obtain one from an existing video or audio source.

Canvas​

If you have a <canvas> HTML element, then you will need to call the captureStream method of the given canvas. This is particularly useful if you are using Three.js or PixiJS to render complex animations and more.

Usage Example​

import React, { useCallback, useState } from "react";

export function CanvasExample() {
  const [canvasStream, setCanvasStream] = useState<MediaStream>();

  const streamFromCanvas = useCallback((canvas: HTMLCanvasElement | null) => {
    if (!canvas) return;
    setCanvasStream(canvas.captureStream());
  }, []);

  // use canvasStream as desired

  return <canvas ref={streamFromCanvas} />;
}

Smelter​

If you are using Smelter, then all you need to do is register an output via registerOutput.

Usage Example​

tip

If you want to see a full example React app which uses Fishjam with Smelter, then you can check out our full example on GitHub.

import React from "react";
import Smelter, { setWasmBundleUrl } from "@swmansion/smelter-web-wasm";
import { View } from "@swmansion/smelter";

setWasmBundleUrl("/assets/smelter.wasm");

async function run() {
  const smelter = new Smelter({ framerate: 30 });
  const { stream } = await smelter.registerOutput("example-output", <View />, {
    type: "stream",
    video: {
      resolution: { width: 1920, height: 1080 },
    },
  });
  await smelter.init();
}

Video/Audio HTML tag​

If you have a <video> or <audio> element, then you need to call their captureStream method, analogously as you would with a canvas.

Usage Example​

import React, { useCallback, useState } from "react";

const hasCaptureStream = (video: HTMLVideoElement): video is HTMLVideoElement & { captureStream: () => MediaStream } => "captureStream" in video && typeof video.captureStream === "function";

export function CanvasExample() {
  const [stream, setStream] = useState<MediaStream>();

  const streamFromVideo = useCallback((video: HTMLVideoElement | null) => {
    if (!video) return;
    if(!hasCaptureStream(video)) {
      console.warn("captureStream is not supported on this browser");
      return;
    }

    setStream(video.captureStream());
  }, []);

  // use stream as desired

  return <video ref={streamFromVideo} /* ... */ />;
}