ProductsDocumentationBlogGitHubForums
Documentation
Login >SIGN UP

Audio Multiplexing

Suggest Edits

The Dolby.io platform supports Audio Multiplexing, a feature that allows viewers to receive multiple audio streams in a conference-like experience, where each audio stream is emphasized or deemphasized based on activity.

👍

Building a Conference Application?

Dolby.io Communications includes SDKs and APIs explicitly for building conferencing experiences. Dolby.io Communications and Streaming intergrate to help scale conferences for interactive broadcast solutions.

Understanding Audio Multiplexing

To first understand Audio Multiplexing, we need to understand how the Dolby.io servers ingest feeds. When Broadcasting, the Dolby.io Streaming servers will assign the most recent published stream source as the "Main Source", a function that allows broadcasters to seamlessly overwrite streams or add Redundant Streams. For basic broadcasts, this is sufficient, for more advanced broadcasts Dolby.io provides two features that are exceptions:

  1. Creating a Multisource stream for Multi-view experiences.
  2. Audio Multiplexing, for experiences with multiple audio streams playing at once, such as that seen on "Clubhouse" type platforms and apps.

If you've reviewed the Multi-source Broadcasting and Multi-view documentation, you may be confused as to why you would use Audio Multiplexing instead of Multiview. Audio multiplexing allows audio sources to overlap, a feature that is useful for

Audio Multiplexing
Let users hear multiple audio sources at once that overlap.
Clubhouse-like apps where hosts chat virtually
Townhall apps or platforms where people may periodically ask questions or present.
Immersive sports apps where a commentary track is combined with an in-stadium ambient noise track.

Using Audio Multiplexing

To get started using Audio Multiplexing, you first need to create a Publishing token with Multisource and have multiple audio sources ready to test, each assigned a unique sourceID at the publisher.

📘

Not familar with our JavaScript SDK?

Audio Multiplexing is a complex feature made availible through our Client SDKs.

Once you're streaming multiple audio sources, the next step is to set up the View so that the incoming audio sources can be correctly multiplexed. When connecting the View instance, there are a number of parameters available in the SDK you can adjust depending on your workflow. Some parameters of note for audio multiplexing include:

  • multiplexedAudioTracks: This is required to enable multiplexing. It denotes the number (int) of audio tracks to receive as Voice Activity Detection (VAD) multiplexed audio. This value must be greater than or equal to the number of audio tracks on the stream. Additional audio tracks will overwrite existing audio tracks. There isn't a limit to the number of audio tracks that can be rendered in the browser, only the amount of data. The current limit is a bitrate of 12 Mbps.
  • dtx: Discontinuous transmission or DTX is a boolean value that signals to the server to only deliver audio packets when a signal is detected, such as when a person is talking. Enabling DTX will reduce bandwidth costs for audio transmission but may cause non-voice audio such as instruments to become choppy.
  • pinnedSourceId: Pinned Source ID is a String that denotes the main source that will be received by the default MediaStream. This value is useful for denoting your default audio stream that will always load regardless of how many other audio channels are present.
  • excludedSourceIds: Exclude Source IDs is an Array of Strings that denotes audio streams that should not be included in the multiplex. This feature is useful for conference-type applications where a user's own audio shouldn't be heard.
// Your stream Credentials
const streamName = "YOUR STREAM NAME";
const streamAccountId = "YOUR ACCOUNT ID";

// Number of audio elements to generate on the web page
const NUMBER_OF_AUDIO_ELEMENTS = 3;

var viewer;

const tokenGenerator = () => millicast.Director.getSubscriber({
    streamName: streamName,
    streamAccountId: streamAccountId,
});

viewer = new millicast.View(undefined, tokenGenerator);

// Connect to the stream
await viewer.connect({
    pinnedSourceId: 'main',
    multiplexedAudioTracks: NUMBER_OF_AUDIO_ELEMENTS + 1,
    excludeSourceIds: ['audience1', 'audience2'],
    disableVideo: true,
    dtx: true,
});

With the Viewer defined, we can assign the incoming tracks. This can be done in a variety of ways, but the simplest is to grab both the track event as it is created by the Viewer node and the broadcastEvent as it is created by the Publisher node. Additionally, we also want to create <audio> elements. These <audio> tags will eventually be what plays the audio stream.

// Initialize the audio elements
for (let i = 0; i < NUMBER_OF_AUDIO_ELEMENTS; i++) {
    const audioElement = document.createElement("audio");
    audioElement.controls = true;
    audioElement.autoplay = true;

    audioTrackDiv.appendChild(audioElement);
    audioTrackDiv.appendChild(document.createElement("br"));
    audioTrackDiv.appendChild(document.createElement("br"));
}

viewer.on("broadcastEvent", async (event) => {
    console.log("broadcastEvent", event);

    if (event.name === "active") {
        await addEvent(event, undefined);
    }
});

viewer.on("track", async (event) => {
    console.log("track", event);
    await addEvent(undefined, event);
});

🚧

Viewer on "track" events

The Dolby.io SDKs offer .on("track", async (event) => {}) functionality for triggering events as tracks are added. When using Audio Multiplexing this event will trigger a number of times equal to the multiplexedAudioTracks value, regardless of if those tracks actually contain data.

This means that if multiplexedAudioTracks is set to 5 it will trigger once for the first track and five additional times for each multiplexed audio track, regardless of whether there are only two tracks broadcasting or twenty.

The broadcastEvent will contain the feeds as they are being published. These feeds need to be linked to Viewer tracks to be delivered via a function called project which projects the media onto the track. This is because different feeds maybe be coming from different sources and hence may connect or disconnect at different intervals. The relationship between a feed and a track is organized this way so that as feeds disconnect they can be swapped or removed without all the streams being interrupted.

The function below also establishes the relationship between the <audio> tag we created above and the track. Allowing the <audio> tag to be rendered on a page for the listener to hear. Both the source event and the track event need to be available before starting the projection.

async function addEvent(sourceEvent, trackEvent) {
    for (let i = 0; i < events.length; i++) {
        const event = events[i];
        if (!event.source || !event.track) {
            if (sourceEvent) event.source = sourceEvent;
            if (trackEvent) event.track = trackEvent;
            
            if (event.source && event.track) {
                // Both source and track events are available
                // Start the projection
                await project(event.source, event.track);
                return;
            }
        }
    }

    // Save the new entry
    events.push({source: sourceEvent, track: trackEvent});
}

async function project(sourceEvent, trackEvent) {
    // Add the track to the audio element to start playing
    const audioElements = audioTrackDiv.getElementsByTagName("audio");
    const mid = parseInt(trackEvent.transceiver.mid);
    const audioElement = audioElements[mid];
    audioElement.srcObject = new MediaStream([trackEvent.track]);
    audioElement.play();
    
    // Start the projection
    console.log("About to project", sourceEvent, trackEvent);
    await viewer.project(sourceEvent.data.sourceId, [
        {
            media: sourceEvent.data.tracks[0].media,
            trackId: sourceEvent.data.tracks[0].trackId,
            mediaId: mid,
        },
    ]);
}

To recap the above workflow:

  1. The app authenticates and connects with the Dolby.io Streaming (Millicast) Director.
  2. As the app connects the Director creates a main track plus an additional number of tracks equal to the multiplexedAudioTracks value. This triggers a track event for each track added.
  3. An audio feed is connected to the Publisher Node triggering a Broadcast event.
  4. The broadcast event creates an <audio> tag and triggers the projectOn function.
  5. The projectOn function adds a track to the <audio> tag and then calls the viewer.project function to project the media onto the track. Allowing the audio feed to be rendered in the <audio> tag.
  6. Steps 3-5 are repeated as more feeds are added.

To help with understanding and implementing the Audio Multiplexing feature is included:

<html>
<head>
    <title>Dolby Millicast Audio Multiplexing Demo</title>
    <!-- Dolby Millicast JavaScript SDK -->
    <script src="https://cdn.jsdelivr.net/npm/@millicast/sdk/dist/millicast.umd.min.js"></script>
    <link
        href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css"
        rel="stylesheet"
        integrity="sha384-1BmE4kWBq78iYhFldvKuhfTAU6auU8tT94WrHftjDbrCEXSU1oBoqyl2QvZ6jIW3"
        crossorigin="anonymous"
    />
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/font/bootstrap-icons.css" />
</head>
<body>
    <div class="container bg-dark text-white gx-4 px-4 py-4 mt-3 rounded">
        <h1>Dolby Millicast Audio Multiplexing Demo</h1>
    </div>
    <div class="container px-4 mt-4">
        <div class="row justify-content-around mt-3">
            <div class="col-8 shadow p-3 mb-5 bg-body rounded text-center">
                <button onclick="startStream()" id="startBtn" style="height: 50px; width: 150px">Start</button>
                <button onclick="resetStream()" id="resetBtn" style="height: 50px; width: 150px" disabled>Reset</button>
                <br />
                <br />
                <div id="audioTrackDiv" color="grey"></div>
            </div>
        </div>
    </div>

    <script>
        // Your stream credentials
        const streamName = "YOUR STREAM NAME";
        const streamAccountId = "YOUR ACCOUNT ID";

        // Number of audio elements to generate on the web page
        const NUMBER_OF_AUDIO_ELEMENTS = 3;

        var viewer;
        const events = [];
        const audioTrackDiv = document.getElementById("audioTrackDiv");

        async function startStream() {
            document.getElementById("startBtn").disabled = true;
            document.getElementById("resetBtn").disabled = false;

            // Initialize the audio elements
            for (let i = 0; i < NUMBER_OF_AUDIO_ELEMENTS; i++) {
                const audioElement = document.createElement("audio");
                audioElement.controls = true;
                audioElement.autoplay = true;

                audioTrackDiv.appendChild(audioElement);
                audioTrackDiv.appendChild(document.createElement("br"));
                audioTrackDiv.appendChild(document.createElement("br"));
            }

            const tokenGenerator = () =>
                millicast.Director.getSubscriber({
                    streamName: streamName,
                    streamAccountId: streamAccountId,
                });

            viewer = new millicast.View(undefined, tokenGenerator);

            viewer.on("broadcastEvent", async (event) => {
                console.log("broadcastEvent", event);

                if (event.name === "active") {
                    await addEvent(event, undefined);
                }
            });

            viewer.on("track", async (event) => {
                console.log("track", event);
                await addEvent(undefined, event);
            });

            // Connect to the stream
            await viewer.connect({
                pinnedSourceId: "main",
                multiplexedAudioTracks: NUMBER_OF_AUDIO_ELEMENTS + 1,
                disableVideo: true,
                dtx: true,
            });
        }

        async function addEvent(sourceEvent, trackEvent) {
            for (let i = 0; i < events.length; i++) {
                const event = events[i];
                if (!event.source || !event.track) {
                    if (sourceEvent) event.source = sourceEvent;
                    if (trackEvent) event.track = trackEvent;
                    
                    if (event.source && event.track) {
                        // Both source and track events are available
                        // Start the projection
                        await project(event.source, event.track);
                        return;
                    }
                }
            }

            // Save the new entry
            events.push({source: sourceEvent, track: trackEvent});
        }

        async function project(sourceEvent, trackEvent) {
            // Add the track to the audio element to start playing
            const audioElements = audioTrackDiv.getElementsByTagName("audio");
            const mid = parseInt(trackEvent.transceiver.mid);
            const audioElement = audioElements[mid];
            audioElement.srcObject = new MediaStream([trackEvent.track]);
            audioElement.play();
            
            // Start the projection
            console.log("About to project", sourceEvent, trackEvent);
            await viewer.project(sourceEvent.data.sourceId, [
                {
                    media: sourceEvent.data.tracks[0].media,
                    trackId: sourceEvent.data.tracks[0].trackId,
                    mediaId: mid,
                },
            ]);
        }

        function resetStream() {
            // Refresh the page
            location.reload();
        }
    </script>
</body>
</html>

Troubleshooting

Are audio streams encrypted

One of the advantages of using Multiplexing versus audio mixing is the end-to-end encryption as the audio never needs to be processed or re-encoded.

Only one audio track is loading

There are a number of reasons why one audio track may only be loading:

  • Check that the Publishing token is Multisource enabled.
  • Check that each feed has a different source ID.
  • Check that the media stream is being projected onto the correct track.

Audio is dropping in and out

There are a number of reasons why audio may be choppy:

  • dtx may be interfering with what audio is rendered. It is designed for voice so background noise or instruments may be cut out.
  • If the sum total bitrate of all audio tracks exceeds 12 Mbps audio may become choppy or drop out.

Updated 7 days ago