Millicast Player Plugin

MillicastPlayer plugin for Unreal Engine 4

  • Supported Unreal Engine version 4.27
  • Supported on Windows and Linux

This plugin enables you to play a real time stream from Dolby.io Real-time Streaming in your Unreal Engine game.
You can configure your credentials and configure your game logic using unreal object and then render the video in a texture2D.

Installation

You can install the plugin from the source code.
Follow these steps :

  • Create a project with the Unreal Engine (UE) editor.
  • Close the editor
  • Go at the root of your project folder (C:\Users\User\Unreal Engine\MyProject)
  • Create a new directory "Plugins" and move into it
  • Clone the MillicastPlayer repository : git clone https://github.com/millicast/millicast-player-unreal-engine-plugin.git MillicastPlayer
  • Open your project with UE

It will prompt you, saying if you want to re-build MillicastPlayer plugin, say yes.
You are now in the editor and can build your game using MillicastPlayer.

Note: After you package your game, it is possible that you will get an error when launching the game :

"Plugin MillicastPlayer could not be load because module MillicastPlayer has not been found"

And then the game fails to launch.
That is because Unreal has excluded the plugin.
If that is the case, create an empty C++ class in your project. This will force Unreal to include the plugin. Then, re-package the game, launch it, and it should be fixed.

Enable the plugin

To enable the plugin, open the plugin manager in Edit > Plugins.

527527

Then search for MillicastPlayer. It is in the category "Media Players". Tick the "enabled" checkbox to enable the plugin. It will prompt you if you are sure to use this plugin, because it is in beta, just accept. After that Unreal will reboot in order to load the plugins.

12421242

If it is already enabled, just leave it like this.

Setup your stream in the editor

Basically, you have several Unreal objects to configure in order to view a stream in your game.

First, we will see how to configure a media source to setup your Dolby.io Real-time Streaming credentials. Then how to render the video in a texture 2D, by attaching it to an object in the World. Finally, implement the logic of the game using a blueprint class.

MillicastMediaSource

The media source object allow you to configure your credentials and is the source of the webrtc video stream.
To add a MillicastMediaSource object, add a new asset by clicking "Add/Import", and you will see the object in the "media" category.

431431

After that, double click on the asset you just created so you can start configuring it.
The menu is divided in two categories. First, the Dolby.io Real-time Streaming credentials:

  • the stream name you want to subscribe to
  • your account id
  • Whether you need to use the subscribe token or not. If you are using a secure viewer, enable it and enter your subscribe token. Otherwise, leave it to the default.
  • The subscribe api url, which usually is https://director.millicast.com/api/director/subscribe
18661866

MillicastTexture2D Player

This object is a receiver/consumer for webrtc video frames and will render the frames in a texture 2D.
You can create though the asset panel.
You can also create a MillicastTexture2D from it.

22892289

MillicastTexture2D

The MillicastTexture2D is a texture you can apply to an object in your World. Once it is created, drag'n drop it in the editor, on the object that will be used to render the video.
In the image below, I created a plane and apply the texture to it. You can see the plane is now black because no frame are being rendered right now. When you apply the texture, you will see that a material object is created in your assets.

12471247

Millicast Audio Actor

This object is used to render audio track in you scene. It holds an AudioComponent instance and you can configure it to enable spatial sound. If you want spatial sound to work, you can just go in the Attenuation menu, enable the spatialization and override the attenuation settings.

740740

Like any other actor, you can search it in the editor among the actors object and drag'n drop it into your scene.

Blueprint

We will now see how to implement the logic using the blueprint.
To do something really simple, the game will subscribe to Dolby.io Real-time Streaming when it starts playing and unsubscribe when it ends.
To do that, the two important components are :

  • MillicastDirectorComponent : this object is used to authenticate to Dolby.io Real-time Streaming using the credentials you provided in the Dolby.io Real-time Streaming media source and get the websocket url and jwt.
  • MillicastSubscriberComponent: this object subscribes to the WebRTC stream by using the WebSocket url and the Jwt.

To do all this, first create a blueprint class. It will open a window, select actor.

243243 508508

Add it in your World (by drag'n drop, it is not visible when the game is playing) and double click on it.
Go in the event graph.
Add the Director and subscriber component.

291291

Both component need the instance of the MillicastMediaSource object. Assign it by clicking one component and go in the panel on the right. Be careful and assign the one you already created. Do not create a new one.

479479

Director Component

The important method here is "Authenticate". Drag'n drop your object in the graph and make a connection with this method. It returns a boolean to say if the request has been made or not.

If the request is successful, it will fire an event, "OnAuthenticated". That event forward a parameter, the Signaling information, which are the websocket url and the jwt.

If the request is not successful, it fires the event, "OnAuthenticationFailure" with the HTTP error code and the status message.

Connect the event "begin play" to the "Authenticate" method so the game will make a request to Dolby.io Real-time Streaming when starting.

Subscriber Component

Connect the event "OnAuthenticated" with the signaling info to the method Subscribe. This method will subscribe to the Dolby.io Real-time Streaming stream and receive WebRTC audio and video tracks.

Then add a new event, "Event End Play" which is an event fired when the game ends, and connect it to the method "Unsubscribe" to unsubscribe from the stream.

When you are subscribed, if you receive video or audio tracks, the event On Video Track and On Audio Track will be fired. It will give you the corresponding track. This object has a AddConsumer method which is used to add either a MillicastAudioActor for an audio track and a MillicastTexture2DPlayer for a video track. If a track has a consumer it will call a callback to feed audio/video data. The consumer itself will then render the data in the scene.

Example

Here is an example of how to connect everything.

11451145 13771377 13881388

Launch the game

First broadcast some media to Dolby.io Real-time Streaming from Millicast Studio, OBS, your browser ...
Note that for now, only VP8 and VP9 are supported by the plugin.
You can quickly test if everything works by playing the game in the editor.
You can subscribe to Dolby.io Real-time Streaming and render the video track in the texture.
However, the audio will not play in the editor.
You need to launch the game on your platform, or package it to hear audio.

Setup the stream name dynamically

In the above example, the credentials were configured in the Media Source, but we couldn't change them directly from the game. The game was just launching and subscribing to Dolby.io Real-time Streaming with the stream name we configured from the editor.
Let's see now how to modify the stream name on a given event that happens in the game.
To illustrate this, we will walk through an example project, where when we shoot a cube, the game starts subscribing to a specific stream, and if we shoot on another cube, the game subscribes to another stream. Each cube is associated with a cube.

804804

Basically, we must be able to detect a collision event on the cube, which will set the stream name on the Millicast Source and then call an event in our main blueprint to start the subscribing. To modify the Millicast Source object's stream name, we must get a reference on the object, either in the blueprints or in a C++ class. We will see both methods.

First let's see the main blueprint. Create a blueprint and add it in your game environment. Open the event graph.

11411141

The logic here is : we create a custom event "EventStartViewing". This event will be called by the blueprint associated with the cube, when a collision occurs. When it is fired, we check if we are already subscribed. If so, we stop subscribing and stop the viewer. Otherwise, we start it.
Let's see how to trigger a collision. Create a blueprint, or add a box trigger around the cube and replace the blueprint with a custom one. Check "Simulation Generates hit" to generate hit triggers, and "Block All" collision presets.

361361

Now we can open the associated blueprint. Add an "Event Hit". Now let's see how to configure the Media Source object.

Using only blueprint

When using only the blueprint, add a public variable of type "MillicastMediaSource". Then, you can just set the stream name like the event graph below.

11451145

Close the event graph editor, in the settings of the box trigger, add the instance of the MillicastMediaSource object you already own, and the main blueprint instance. All public members are visible in the settings.

365365

Using a C++ class

Create a C++ class (actor component in this example).
When Visual Studio opens, first add MillicastPlayer as a dependency in your Project.build.cs file.

PrivateDependencyModuleNames.AddRange(new string[] { "MillicastPlayer" });

Now, you can include and link against Millicast player public object.
The header file below has an instance of a UMillicastMediaSource object. It is set when Initialize is called.
ConfigureSource is the method we will call from the blueprint event graph to configure the stream name.

// USourceConfiguratorComponent Header

#pragma once

#include "CoreMinimal.h"
#include "Components/ActorComponent.h"
#include "MillicastMediaSource.h"


#include "SourceConfiguratorComponent.generated.h"

UCLASS( ClassGroup=(Custom), meta=(BlueprintSpawnableComponent) )
class MYPROJECT2_API USourceConfiguratorComponent : public UActorComponent
{
    GENERATED_BODY()

private:
    UPROPERTY(EditDefaultsOnly, Category = "Properties", META = (DisplayName = "Millicast Media Source", AllowPrivateAccess = true))
        UMillicastMediaSource* MillicastMediaSource = nullptr;

public: 
    // Sets default values for this component's properties
    USourceConfiguratorComponent();

    bool Initialize(UMillicastMediaSource* Source = nullptr);

    UFUNCTION(BlueprintCallable, Category = "Component", META = (DisplayName = "ConfigureSource"))
    void ConfigureSource(FString StreamName, FString AccountId);

    UFUNCTION(BlueprintCallable, Category = "Component", META = (DisplayName = "ConfigureSourceWithUrl"))
    void ConfigureSourceWithUrl(FString StreamName, FString AccountId, FString ApiUrl);

    UFUNCTION(BlueprintCallable, Category = "Component", META = (DisplayName = "ConfigureSecureSourceWithUrl"))
    void ConfigureSecureSourceWithUrl(FString StreamName, FString AccountId, FString ApiUrl, FString SubscribeToken);

protected:
    // Called when the game starts
    virtual void BeginPlay() override;

public: 
    // Called every frame
    virtual void TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction) override;

        
};
// USourceConfiguratorComponent source

bool USourceConfiguratorComponent::Initialize(UMillicastMediaSource* InMediaSource)
{
    if (MillicastMediaSource == nullptr && InMediaSource != nullptr)
    {
        MillicastMediaSource = InMediaSource;
    }

    return InMediaSource != nullptr && InMediaSource == MillicastMediaSource;
}

void USourceConfiguratorComponent::ConfigureSource(FString StreamName, FString AccountId)
{
    MillicastMediaSource->StreamName = std::move(StreamName);
    MillicastMediaSource->AccountId = std::move(AccountId);
}

Now, build your solution.
Return in unreal engine editor. Open the blueprint associated in the box trigger. Add the Source configurator component.

284284

Don't forget to set up the MillicastMediaSource instance in the right panel. This is the step that will call our Initialize function.

476476

Now, you can make the event graph :

11461146

Now, just launch the game, and shoot on one of the cube to start one stream or another.
This is a simple example, where we have hardcoded stream names in the blueprint event graph. But you can do something more complex, like some REST API call in your C++ class when the collision occurs and then configure the MediaSource object.

Subscriber's event

In the Subscriber's component, you can connect several events received from the Dolby.io Real-time Streaming media server.

On Active event

This event is called when a new track is being published in the stream.
You will get the stream id, the source id and an array of track information corresponding to the tracks being pushed by the publisher.

On Inactive event

This event is called when a source has been unpublished within the stream.

On Stopped

This event is called when the stream is no longer available.

On Vad

This event is called when an audio track is being multiplexed.
You will get the mid of the track and the source id of the publisher.

On Layers

This event is called when simulcast/SVC layers are available.
You will get the mid of the track and an array of the layers.

On Viewer count

This event is called when a viewer starts or stops viewing.
You get the new viewer count each time it is called.

Receive multisource stream

Now that we have seen all the subscribers event, you are now able to receive multisource streams.
Typically, you will get the tracks and source ids of the publishers in the active event.
When you get that information, you must use the project method from the subscriber to forward the media from a track of a given source into a given track.
Since, you don't know how much track you will receive in the active event, and as default there is only one audio/video track negociated in the peerconnect, you can use the subscriber's method addRemoteTrack to dynamically add a new track to the peerconnection by renegociating the SDP.

As an example blueprint :

23082308 20282028

Did this page help you?