📘
Video data streaming is currently supported in PNG and H264 formats

📘
H264 video per participant is currently only available on bots with the web4core variant

Receive separate live videos of each participant in PNG or H264 Formats over websocket in realtime

Platforms Support

Supported	Platform	Number of concurrent streams
✅	Zoom	16
✅	Microsoft Teams	9 - Teams web only supports up to 9 video streams at once
✅	Google Meet	16
❌	Webex
✅	Slack Huddles (Beta)
✅	Go-To Meeting (Beta)

PNG Recording Specifications

PNG Resolution	Resolution	Frame Rate
Screen Share	360p	2 frames per second
Participant	360p	2 frames per second

H264 Recording Specifications

H264 buffers retrieved are at the resolution that we receive from the meeting platform, which can change during the meeting based on the number of participants and the connection quality of each participant's connection.

H264 Resolution	Resolution	Frame Rate
Screen Share	240px-1280px typical	10-30 frames per second typical
Participant	200px-1000px typical	10-30 frames per second typical

Implementation

Step 1: Create a bot

To get separate video per participant, you must set recording_config.video_mixed_layout = "gallery_view_v2". Below is an example of what it would look like in your request

PNG Format

curl --request POST \
     --url https://us-west-2.recall.ai/api/v1/bot \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --header 'authorization: YOUR_RECALL_API_KEY' \
     --data '
{
  "meeting_url": "YOUR_MEETING_URL",
  "recording_config": {
    "video_mixed_layout": "gallery_view_v2",
    "video_separate_png": {},
    "realtime_endpoints": [
      {
      	type: "websocket",
        url: YOUR_WEBSOCKET_RECEIVER_URL,
        events: ["video_separate_png.data"]
      }
    ]
  }
}
'

const response = await fetch("https://us-west-2.recall.ai/api/v1/bot", {
  method: "POST",
  headers: {
    "accept": "application/json",
    "content-type": "application/json"
    "authorization": YOUR_RECALL_API_KEY
  },
  body: JSON.stringify({
    meeting_url: YOUR_MEETING_URL,
    recording_config: {
      video_mixed_layout: "gallery_view_v2",
      video_separate_png: {},
      realtime_endpoints: [
        {
          type: "websocket",
          url: YOUR_WEBSOCKET_RECEIVER_URL,
          events: ["video_separate_png.data"]
        }
      ]	
    }
  })
});

if (!response.ok) {
  throw new Error(`Error: ${response.status} ${response.statusText}`);
}
const data = await response.json();

import requests

response = requests.post(
    "https://us-west-2.recall.ai/api/v1/bot",
    json={
      "meeting_url": "YOUR_MEETING_URL",
      "recording_config": {
	      "video_mixed_layout": "gallery_view_v2"
        "video_separate_png": {},
  			"realtime_endpoints": [
        	{
            type: "websocket",
            url: YOUR_WEBSOCKET_RECEIVER_URL,
            events: ["video_separate_png.data"]
          }
      	]
      }
    },
    headers={
      "accept": "application/json",
      "content-type": "application/json",
    	"authorization": YOUR_RECALL_API_KEY
    }
)

if not response.ok:
 	errorMessage = f"Error: {response.status_code} - {response.text}"
  raise requests.RequestException(errorMessage)
result = response.json()

H264 Format

curl --request POST \
     --url https://us-west-2.recall.ai/api/v1/bot \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --header 'authorization: YOUR_RECALL_API_KEY' \
     --data '
{
  "meeting_url": "YOUR_MEETING_URL",
  "variant": {
    "zoom": "web_4_core",
    "google_meet": "web_4_core",
    "microsoft_teams": "web_4_core"
    },
  "recording_config": {
    "video_mixed_layout": "gallery_view_v2",
    "video_separate_h264": {},
    "realtime_endpoints": [
      {
      	type: "websocket",
        url: "YOUR_WEBSOCKET_RECEIVER_URL",
        events: ["video_separate_h264.data"]
      }
    ]
  }
}
'

const response = await fetch("https://us-west-2.recall.ai/api/v1/bot", {
  method: "POST",
  headers: {
    "accept": "application/json",
    "content-type": "application/json"
    "authorization": YOUR_RECALL_API_KEY
  },
  body: JSON.stringify({
    meeting_url: YOUR_MEETING_URL,
    variant: {
      zoom: "web_4_core",
      google_meet: "web_4_core",
      microsoft_teams: "web_4_core"
    },
    recording_config: {
      video_mixed_layout: "gallery_view_v2",
      video_separate_h264: {},
      realtime_endpoints: [
        {
          type: "websocket",
          url: YOUR_WEBSOCKET_RECEIVER_URL,
          events: ["video_separate_h264.data"]
        }
      ]	
    }
  })
});

if (!response.ok) {
  throw new Error(`Error: ${response.status} ${response.statusText}`);
}

const data = await response.json();

import requests

response = requests.post(
    "https://us-west-2.recall.ai/api/v1/bot",
    json={
      "meeting_url": YOUR_MEETING_URL,
      "variant": {
        "zoom": "web_4_core",
        "google_meet": "web_4_core",
        "microsoft_teams": "web_4_core"
      },
      "recording_config": {
        "video_mixed_layout": "gallery_view_v2"
        "video_separate_h264": {},
        "realtime_endpoints": [
          {
            "type": "websocket",
            "url": YOUR_WEBSOCKET_RECEIVER_URL,
            "events": ["video_separate_h264.data"]
          }
        ]
      }
    },
    headers={
      "accept": "application/json",
      "content-type": "application/json",
    	"authorization": YOUR_RECALL_API_KEY
    }
)

if not response.ok:
 	errorMessage = f"Error: {response.status_code} - {response.text}"
  raise requests.RequestException(errorMessage)
  
result = response.json()

Step 2: Receive websocket messages with video data

Setup a websocket server and ensure it is publicly accessible. You will receive messages in the following payload format:

PNG Format

{
  "event": "video_separate_png.data", 
  "data": {
    "data": {
      "buffer": string, // base64 encoded png at 2fps with resolution 360x640
      "timestamp": {
      	"relative": float,
        "absolute": string
    	},
      "type": "webcam" | "screenshare",
      "participant": {
      	"id": number,
      	"name": string | null,
        "is_host": boolean,
        "platform": string | null,
        "extra_data": object,
        "email": string | null
      }
    },
    "realtime_endpoint": {
      "id": string,
      "metadata": object,
    },
    "video_separate": {
      "id": string,
      "metadata": object
    },
    "recording": {
      "id": string,
      "metadata": object
    },
    "bot": {
      "id": string,
      "metadata": object
    },
  }
}

H264 Format

{
  "event": "video_separate_h264.data", 
  "data": {
    "data": {
      "buffer": string, // base64 h264 at a resolution and framerate set by the platform
      "timestamp": {
      	"relative": float,
        "absolute": string
    	},
      "type": "webcam" | "screenshare",
      "participant": {
      	"id": number,
      	"name": string | null,
        "is_host": boolean,
        "platform": string | null,
        "extra_data": object,
        "email": string | null
      }
    },
    "realtime_endpoint": {
      "id": string,
      "metadata": object,
    },
    "video_separate": {
      "id": string,
      "metadata": object
    },
    "recording": {
      "id": string,
      "metadata": object
    },
    "bot": {
      "id": string,
      "metadata": object
    },
  }
}

Step 3: Decode Video

For H264 live video, checkout our samples:

☕️ Javascript Browser Sample

🐍 Python Samples

Troubleshooting H264 Live Video

The participant video is blurry

The resolution of per-participant video is determined by the meeting platform. They typically decide on the resolution to use based on the number of participants in a meeting, the network conditions and available compute of the participant, and whether or not the participant is judged to be of interest.

One way to normally convince the meeting platform to provide a higher resolution for a single participant is to use the pin participant endpoint. This has the effect of pinning a single participants video, which normally provides a higher resolution.

The framerate is low

The framerate is largely determined by the meeting provider. However, there are a few things you can do to ensure that you're getting the highest framerate that the meeting provider will allow:

Make sure that the websocket receiver that handles the video_separate_h264.data event is not blocked. This means that the base64 decoding, h264 decoding, and any muxing should be handled in separate thread(s) whenever possible. The websocket receiver will ideally only be in charge of receiving the message and then passing this data to another thread for processing.
Ensure that you have a high bandwidth connection to our servers. Ngrok works great for testing, but for large meetings there might not be enough bandwidth to receive the video at full framerate. The best solution is to deploy your server in the same region as the Recall.ai region for your bot, ideally within AWS for the maximum throughput.