Recall supports streaming a webpage's audio and video to meetings via the bot's camera and microphone. This set of APIs is the perfect combination to build interactive AI agents that listen to the meeting and react in realtime. Some of our customers use this functionality to build AI-powered sales agents, coaches, recruiters, meeting planners and more.

For example implementations/use cases, see our demo repos:

• AI Voice Agent Demo
• Real-time Translator Demo

A specific example

The following video demonstration shows an example of a bot reacting in real time to the conversation in the meeting.

Platform Support

*Zoom native bot not supported

🚧

Streaming Media incompatible with automatic_video_output and automatic_audio_output

Bot Media Output cannot currently be used with automatic_video_output or automatic_audio_output.

The Output Video and Output Audio endpoints must also not be used if your bot is streaming a webpage's contents to the meeting.

Quickstart

Streaming a webpage's audio/video to a meeting

Method 1: Using output_media in Create Bot

You can use the output_media configuration in the Create Bot endpoint to stream the audio and video contents of a webpage to your meeting.

output_media takes the following parameters:

• kind: The type of media to stream (currently only webpage is supported)
• config: The webpage configuration (currently only supports url)

Let's look at an example call to Create Bot:

// POST /api/v1/bot/
{
  "meeting_url": "https://us02web.zoom.us/j/1234567890",
  "bot_name": "Recall.ai Notetaker",
  "output_media": {
    "camera": {
      "kind": "webpage",
      "config": {
        "url": "https://www.recall.ai"
      }
    }
  }
}

The example above tells Recall to create a bot that will continuously stream the contents of recall.ai to the provided meeting URL.

Method 2: Using the Output Media endpoint

You can choose to start outputting media by calling the Output Media endpoint at any point when the bot is in a call.

The parameters for the request are the same as the output_media configuration.

Example cURL:

curl --request POST \
     --url https://api.recall.ai/api/v1/bot/{bot_id}/output_media/ \
     --header 'Authorization: ${RECALL_API_KEY}' \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data-raw '
{
	"camera": {
    "kind": "webpage", 
    "config": {
      "url": "https://recall.ai"
    }
  }
}
'

Stopping media output

You can stop the bot media output at any point while the bot is streaming media to a call by calling the Stop Output Media endpoint.

Example cURL:

curl --request DELETE \
     --url https://api.recall.ai/api/v1/bot/{bot_id}/output_media/ \
     --header 'Authorization: ${RECALL_API_KEY}' \
     --header 'accept: application/json' \
     --header 'content-type: application/json'
     --data-raw '{ "camera": true }'

Accessing realtime meeting data

The bot exposes a Websocket endpoint to retrieve realtime meeting data while the webpage is streaming audio and video to the call. Right now, only realtime transcripts are supported. You can connect to the realtime API from your webpage with the following example:

const ws = new WebSocket('wss://meeting-data.bot.recall.ai/api/v1/transcript');

ws.onmessage = (event) => {
  const message = JSON.stringify(event.data);

  // .. your logic to handle realtime transcripts
};

ws.onopen = () => {
  console.log('Connected to WebSocket server');
};

ws.onclose = () => {
  console.log('Disconnected from WebSocket server');
};

The websocket messages coming from the /api/v1/transcript endpoint have the same shape as the data object in Real-time transcription .

Piping the meeting audio to the webpage

Output media bots configure an input device inside the running webpage that receives the mixed meeting audio of all participants.

You can access a MediaStream object and its audio track from the webpage running inside the bot. The following example shows how to get samples of the meeting audio in AudioData objects:

const mediaStream =   await navigator.mediaDevices.getUserMedia({ audio: true });
const meetingAudioTrack = mediaStream.getAudioTracks()[0];

const trackProcessor = new MediaStreamTrackProcessor({ track: meetingAudioTrack });
const trackReader = trackProcessor.readable.getReader();

while (true) {
  const { value, done } = await trackReader.read();
  const audioData = value;
  ... // Do something with the audio data
}

Debugging

Debugging your webpage with remote Devtools

During the development process, you may need to debug your output media bot's webpage. Recall provides an easy way to connect to the webpage's Chrome Devtools while the bot is running. Check the video demo below and read the following instructions to learn how to access your bot's Devtools.

1. Send an output media bot to your meeting, and wait for its output media stream
2. Log in to your Recall.ai dashboard
3. Select Bot Explorer in the sidebar
4. In the Bot Explorer app, search for your bot by ID

5. Open the "Bot Details" for your bot and click the "Output Media Devtools" button: a devtools inspector connected to your live bot will open in a new tab.

📘

Since Output Media Devtools are exposed by the bot itself, they are only available when the bot is actively in a call.

You can also view the CPU usage for an individual bot in the "Bot Details" section. You can use this graph to uncover any performance bottlenecks with your webpage which might be causing the webpage to lag or perform poorly.

Addressing audio and video issues: 4 core bots

While we expose CPU metrics to help you identify and address any performance issues on your end, sometimes this is out of your control and you just need more CPU power.

To enable this, you can specify 4-core bots in your Create Bot request by using the variant parameter:

{
  ...
  "variant": {
    "zoom": "web_4_core",
    "google_meet": "web_4_core",
    "microsoft_teams": "web_4_core"
  }
}

These bots run on larger machines, which can help address any CPU bottlenecks hindering the audio & video quality of your Output Media feature.

❗️

Important

Due to the inherent cost of running larger machines, 4-core bots are an additional $0.10/hour on top of the regular bot usage rate.

FAQ

What are the browser dimensions of the webpage?

1280x720px