Dynamic 2D Map with Breakout Rooms and Spatial Audio

React web application to demonstrate a conference where participants can move themselves around a 2D map of an office floor plan supporting spatial audio.

12001200

2D Map with dynamic layout

Overview

This project demonstrates a top-down 2D view of a video conference where participants can move their avatar around the screen and audio from participants comes from their virtual placement in a scene.

Features Tech Stack
  • Voice and video conferencing
  • Audio and video with start and stop
  • Audio and video device selection
  • Keyboard controls to move on screen
  • Spatial audio placement for position of participants
  • Muting participants to create breakout rooms
  • HTML / CSS
  • JavaScript / ReactJS
  • Firebase Realtime Database

Getting Started

Clone the Repository

Run the following for the basic version:

git clone https://github.com/dolbyio-samples/meet-dolbyio-water-cooler.git
cd meet-dolbyio-water-cooler

for the spatial sample code you'll want to view the v2.0 branch.

git checkout v2.0

Follow setup instructions

You'll need to complete a few setup steps as described in the README.

✓ Set your Dolby.io App key and App secret in voxeetUtils.js
✓ Setup Firebase Realtime database and update firebaseConfig.js

Key Concepts

This app illustrates meeting around a virtual water cooler. Participant video is rendered in a circle that can be moved around the screen using the keyboard arrow keys. The background image is a custom office floor plan.

There are two versions of this application and source code:

  1. The main branch features breakout rooms represented over a virtual grid. When participants occupy the same hex area they can only speak and listen to other participants in the same region.
  2. The v2.0 branch features spatial audio where participant audio originates from the direction of a participant relative to other participants location on screen.

Breakout Rooms

To encourage both social interactions and private conversations a video conferencing application can support breakout rooms. To support this scenario, participants should only be able to hear and speak to others in a common room.

One way to accomplish this would be to initialize an individual conference for each breakout room. Initializing a new conference and establishing a websocket connection does come with some performance overhead. If participants need to be able to rapidly switch between rooms such as changing channels or to support dynamic movement on screen, the setup and teardown of a conference may be a bottleneck.

A common trick to overcome this constraint is to place all participants in the same conference but selectively mute audio streams. The effect is that participants can see the video of everybody across an entire map, but only hear other participants that share a common room. These rooms have been represented on the map as hex cells.

Selective Muting

Enabling control over the conference audio experience by muting the other participants audio when the user leaves a conversation area is accomplished by the useStopAudio method located in components/WaterCooler/useStopAudio.js in the project.

The core code to control the conference audio is seen below in lines 21-27. To manage the audio of a participant you pass the participant object to the conference.stopAudio(participant) and conference.startAudio(participant) methods of the Dolby.io Communications API.

import { useEffect } from 'react';
import { conference } from '@voxeet/voxeet-web-sdk';

// Stop audio for remote users who are not in the same hexagon.
// Start audio for remote users who are in the same hexagon.
// See reference:
// https://docs.dolby.io/communications-apis/docs/js-client-sdk-conferenceservice#stopaudio
export const useStopAudio = ({ joinInfo, activeHexId, remotePositions }) => {
  useEffect(() => {
    // Don't try to mute/unmute until after we've joined the meeting.
    if (!joinInfo) return;

    // Figure out who to mute and who not to mute.
    for (const { hexId, id } of remotePositions) {
      const participant = conference.participants.get(id);
      // Guard against stale Firebase entries.
      if (participant) {
        // If the remote participant is in the same hexagon,
        if (hexId === activeHexId) {
          // let their audio through.
          conference.startAudio(participant);
        } else {
          // Otherwise stop their audio.
          conference.stopAudio(participant);
        }
      }
    }
  }, [joinInfo, activeHexId, remotePositions]);
};

Manual Spatial Audio Placement

Spatial Audio allows you to place audio for conference participants spatially in a 3D rendered audio scene.

Using this technique, all participants can hear all other participants along two dimensions within the space. For example, if Participant A is on the left side of the room and Participant B is on the right side of the room, Participant A should hear Participant B in their right ear, and Participant B should hear Participant A in their left ear.

You can find the code from this version in the v2.0 branch in the project repo:
https://github.com/dolbyio-samples/meet-dolbyio-water-cooler/tree/v2.0

Enable Spatial Audio

In order to use manual spatial audio placement:

✓ Enable spatial audio when joining conference
✓ Set spatial environment scene

The scene defines the scale and direction for dimensions. You can see how this is setup in components/WaterCooler/useMeeting.js:

// ...
const joinOptions = {
    constraints: {
    audio: true,
    video: false,
    },
    preferRecvMono: false,
    preferSendMono: false,
    spatialAudio: true, // Turn on Spatial Audio
};

// join conference
await conference.join(conf, joinOptions);
setJoinInfo({
    id: session.participant.id,
});

// configure spatial environment and set on conference, [1480, 700] is the pixel size of the floor plan image container
const scale = { x: 1480 / 16, y: 700 / 12, z: 1 }; // scale room to 16 meters x 12 meters
const forward = { x: 0, y: -1, z: 0 };
const up = { x: 0, y: 0, z: 1 };
const right = { x: 1, y: 0, z: 0 };
conference.setSpatialEnvironment(scale, forward, up, right);

Set Spatial Position for Local

The helper function useAvatarPosition returns the pixel coordinate location of the user which changes as the keyboard arrow keys are pressed. When joining the conference, we need to set the initial spatial position for the local user, that is the individual (ie. you) as a self-view.

You can see this in the following snippet taken from components/WaterCooler/index.js:

// ...  
  // The position of the avatar representing the user.
  // This hook implements the arrow key interactions,
  // and returns the [px,px] coordinate position of the particiapnt.
  const position = useAvatarPosition({ initialPosition });

  // once the local participant has joined,
  if (joinInfo) {
    // set the spatial positioning for the local particiapant
    conference.setSpatialPosition(session.participant, {
      x: position[0],
      y: position[1],
      z: 1,
    });
  }
// ...

In addition, the spatial position of each remote participant must be set. Conveniently, the coordinate positioning is tracked and kept in sync using the Firebase Realtime Database. The pixel coordinates of the user's video feed on screen in the browser window correlates directly with the coordinates that can be used for spatial positioning.

This snippet demonstrates that loop for each participant and using it to set the spatial position along the two-dimensional x,y space. This can be found in components/WaterCooler/index.js:

// ...
if (remotePositions && conference.participants) {
    // For each user we are tracking in Firebase,
    for (const p of remotePositions) {
      // see if there is a corresponding participant in the Dolby.io conference
      const participant = conference.participants.get(p.id);
      if (participant) {
        // set the spatial position of the remote participant,
        // based on the object that contains position in Firebase for that user (which is kept in sync with useBroadCastPosition)
        conference.setSpatialPosition(participant, {
          x: p.position[0],
          y: p.position[1],
          z: 1,
        });
      }
    }
  }
// ...

The hexagon grid used with the breakout rooms is no longer needed using this approach so can be removed. It is no longer needed to stop audio for individual participants and instead volume attenuation is sufficient.

For best experience with spatial audio, use wired headphones.

Demo

To try out the experience for yourself, visit the watercooler demo and invite a friend to join you at that same location.

❗️

Event Code

You'll need an invitation code to access this demo website. Please contact [email protected] to obtain instructions on how to access this site.

17301730

an office floor plan with virtual breakout rooms


Did this page help you?