Skip to main content
Version: Next

Process camera frames in a worklet Mobile

note

This guide is exclusively for Mobile (React Native) applications.

@fishjam-cloud/react-native-worklets runs a worklet on every frame the Fishjam camera captures. The worklet gets the frame's native buffer, so you can hand it to on-device inference or other native frame-processing code, while the camera keeps publishing as usual.

Use this guide when you want to read frames. To change the published video, use background effects or your own WebGPU effects, which are built on the same mechanism.

Prerequisites​

  • @fishjam-cloud/react-native-client 0.30.2 or newer, with your app wrapped in FishjamProvider (see Installation)
  • React Native 0.86 with the New Architecture enabled
  • Android 8.0 (API 26) or newer on Android devices
npm install @fishjam-cloud/react-native-worklets react-native-worklets

The minor version of @fishjam-cloud/react-native-worklets follows react-native-worklets: 0.12.x works with react-native-worklets 0.12.x. Add the worklets Babel plugin as the last entry in your Babel plugins, then rebuild the native app:

module.exports = {
  presets: ["babel-preset-expo"],
  plugins: ["react-native-worklets/plugin"],
};

Attach a frame callback​

A frame callback attaches to a camera track through its frame processor: getCameraFrameProcessor from @fishjam-cloud/react-native-webrtc returns the processor, and attachCameraFrameCallback starts calling your worklet.

The easiest place to do this is a camera track middleware that returns the camera track unchanged. The middleware receives the raw camera track, and Fishjam runs it again whenever the camera restarts or you switch cameras, so the callback always follows the current camera:

import type { TrackMiddleware } from "@fishjam-cloud/react-native-client";
import { getCameraFrameProcessor } from "@fishjam-cloud/react-native-webrtc";
import { attachCameraFrameCallback } from "@fishjam-cloud/react-native-worklets";
import { scheduleOnRN } from "react-native-worklets";

function reportFrameRate(framesPerSecond: number) {
  console.log(`Camera: ${framesPerSecond} fps`);
}

export const frameRateMonitor: TrackMiddleware = async (track) => {
  const processor = await getCameraFrameProcessor(track);

  const stats = { windowStartNs: 0, frames: 0 };
  const subscription = await attachCameraFrameCallback(processor, (frame) => {
    "worklet";
    stats.frames += 1;
    const elapsedNs = frame.timestampNanoseconds - stats.windowStartNs;
    if (elapsedNs >= 1_000_000_000) {
      scheduleOnRN(
        reportFrameRate,
        Math.round((stats.frames * 1e9) / elapsedNs),
      );
      stats.windowStartNs = frame.timestampNanoseconds;
      stats.frames = 0;
    }
  });

  // Publish the camera unchanged; stop the callback when the middleware is removed.
  return { track, onClear: () => subscription.remove() };
};

Switch it on with setCameraTrackMiddleware:

import { useCamera } from "@fishjam-cloud/react-native-client";

const { setCameraTrackMiddleware } = useCamera();

await setCameraTrackMiddleware(frameRateMonitor);

The callback runs on a dedicated camera frame thread, never on the JS thread. Use scheduleOnRN from react-native-worklets to send results back to the JS thread, as in the example.

What a frame carries​

FieldDescription
nativeBufferCVPixelBufferRef on iOS, AHardwareBuffer* on Android, as a bigint pointer
width, heightSize of the buffer, in pixels, before rotation
rotationDegreesClockwise rotation (0, 90, 180 or 270) that brings the frame upright
isFrontCameraWhether the frame comes from the front camera
timestampNanosecondsPresentation timestamp of the frame
pixelFormat"nv12" or "bgra8" on iOS, depending on the camera; "rgba8" on Android
release()Hands the buffer back to the camera before the callback returns. isReleased tells if it was.

On Android, the camera tap converts each camera image to RGBA before your callback runs, so the buffer is always rgba8.

Frame lifetime

nativeBuffer is valid only until your callback returns or you call release(). Don't store the pointer for later use; copy what you need inside the callback.

How frames are delivered​

  • Frames are dropped, never queued. While your callback holds a frame, newer frames are dropped. A slow callback lowers the rate at which you see frames, not the rate at which the camera publishes.
  • One callback per camera track. Remove the subscription before you attach another callback to the same track. The frame processor is also what background effects and WebGPU effects use, so don't attach your own callback to a camera that already runs an effect.
  • One worklet runtime for the app. All frame callbacks share one camera frame runtime, created the first time you attach a callback. Its closure copies follow the usual worklet rules: capture plain values and other worklets, and mutate objects captured by the worklet, not variables from the JS thread.

Troubleshooting​

  • Camera frame worklets require the New Architecture. Turn on the New Architecture and rebuild the native app.
  • Camera frame worklets require Android 8.0 (API 26). The device is too old for hardware buffers. Skip frame processing on such devices.
  • @fishjam-cloud/react-native-worklets is not linked. Rebuild the native app after installing the package; a JS reload is not enough. On Android, if an older version was installed before, also delete android/build/generated/autolinking in your app.
  • A camera frame callback is already attached. Remove the previous subscription first, or check that no effect middleware is running on the camera.