Build a custom meeting agent
Build an agent or avatar that receives meeting audio and responds through a webpage.
Use a webpage you control to build a custom meeting agent or avatar. For starting, stopping, and configuring the webpage output, see Share and control a webpage.
To use a third-party integration, follow an implementation guide.
Setup
Setting up your development environment
For development, you'll want to be able to iterate quickly on your webpage and see changes reflected immediately in the bot. The easiest way to do this is:
- Run a local development server with your webpage. You can either create your own or clone one of our sample repos (Real-Time Translator, Voice Agent)
- Expose it publicly using a tunneling service like Ngrok
- Point your bot to the public tunnel URL
This lets you edit your code locally and instantly see the results in your meeting bot. Once everything is configured, here's what your Create Bot request should look like:
POST /api/v1/bot/
Traffic to the example tunnel URL is forwarded to your localhost.
{
"meeting_url": "https://us02web.zoom.us/j/1234567890",
"bot_name": "Recall.ai Notetaker",
"variant": {
"zoom": "web_4_core",
"google_meet": "web_4_core",
"microsoft_teams": "web_4_core",
"webex": "web_4_core"
},
"output_media": {
"camera": {
"kind": "webpage",
"config": {
"url": "https://my-static-domain.ngrok-free.app"
}
}
}
}Making the bot interactive
So far you've seen how to stream any webpage to a meeting. But to build a real AI agent, you need a webpage that can listen to the information coming from the meeting and respond dynamically. This means creating a webpage that can receive live meeting data and update in real time.
Listening to meeting audio from the webpage
When your bot starts streaming your webpage, the webpage automatically gets access to the live audio from the meeting.
No User Interaction RequiredNormally, accessing microphone audio requires user permission and interaction (like a button click). However, the bot automatically grants microphone permissions, so your webpage will be able to access audio immediately without user prompts or click events.
You can access a MediaStream object and its audio track from the webpage running inside the bot. The following example shows how to get samples of the meeting audio in AudioData objects:
const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
const meetingAudioTrack = mediaStream.getAudioTracks()[0];
const trackProcessor = new MediaStreamTrackProcessor({ track: meetingAudioTrack });
const trackReader = trackProcessor.readable.getReader();
while (true) {
const { value, done } = await trackReader.read();
if (done) break;
const audioData = value;
try {
// Do something with the audio data before closing it.
} finally {
audioData.close();
}
}Generate and play responses
From here, you can process the audio however you need. For example, pipe it to OpenAI's Realtime API for speech-to-speech processing, then output the AI's audio response back to the meeting participants through your webpage's audio elements. This creates a fully interactive voice agent that can have natural conversations with meeting attendees.
You can also use a speech-to-text → LLM → text-to-speech pipeline. Playing audio on the page does not require a separate Recall API call for each spoken response. Keep model-provider credentials on your backend.
Accessing real-time meeting data from the webpage
The bot exposes a websocket endpoint to retrieve real-time meeting data while the webpage is streaming audio and video to the call. This endpoint is available to the webpage running inside the bot. Right now, only real-time transcripts are supported. You can connect to the real-time API from your webpage with the following example:
const ws = new WebSocket('wss://meeting-data.bot.recall.ai/api/v1/transcript');
ws.onmessage = (event) => {
const message = JSON.parse(event.data).transcript?.words?.map(l => l.text)?.join(' ');
// .. your logic to handle realtime transcripts
};
ws.onopen = () => {
console.log('Connected to WebSocket server');
};
ws.onclose = () => {
console.log('Disconnected from WebSocket server');
};Configure real-time transcription before using this input. The websocket messages coming from the /api/v1/transcript endpoint have the same shape as the data object in Real-time transcription.
FAQ
How do I debug my agent's webpage?
See the debugging steps and the webpage FAQ.
Why is the agent's audio or video choppy?
See bot variants and streaming quality.
How do I include the agent's audio in the recording?
Follow the recording configuration. See the recording FAQ for related questions.
How do I transcribe the agent's voice?
See Including Bot Audio in Transcript for real-time and post-meeting transcription.
How do I make the agent respond only when addressed?
See response controls.
How do I receive audio from each participant?
See participant-specific real-time audio.
For actions such as sending chat messages or sharing content, follow Add meeting tools. See the Agents FAQ for shared questions.
Updated about 1 hour ago
