Android
The Native SDK for creating Android applications. You may use it in your Android project to connect, capture, publish, or subscribe to streams using the Dolby.io Streaming Platform.
The package name of the Native SDK for Android is millicast-native-sdk-version-Android.tar.gz.
Note
If you want to use Streaming APIs on Android and iOS, you can also use the Flutter SDK or the React Native plugin.
This guide explains the basic use of the SDK by showing how to start publishing a stream and how to subscribe to the streamed content to render audio and video.
Requirements
Before you start:
- Sign up for a free Dolby.io account.
- Make sure that you have a working video camera and microphone.
- Make sure that you use the Android API 24 or later.
Getting started with publishing
Follow these steps to add the publishing capability to your application.
1. Initialize the SDK
Call the initMillicastSdk method to initialize the SDK with your application context.
import com.millicast.Client;
public class MainActivity extends AppCompatActivity implements NavigationView.OnNavigationItemSelectedListener {
public static final String TAG = "MainActivity";
private static Context context;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
// Initialize when the application starts
if (savedInstanceState == null) {
context = getApplicationContext();
// Load the native shared library and initialize the SDK with the application context
Client.initMillicastSdk(context);
}
}
}
2. Capture audio and video
To capture media, get an array of available audio and video sources and choose the preferred sources from the list. After you start capturing audio and video, the SDK will return an audio and video track that you can add to the publisher later.
// Get an array of audio sources
Media media = Media.getInstance(context); // Get the instance with your application context
ArrayList<AudioSource> audioSources = media.getAudioSources();
// Choose the preferred audio source from the array
AudioSource audioSource = audioSources.get(0); // Get the first audio source
AudioTrack audioTrack;
try {
audioTrack = (AudioTrack) audioSource.startCapture();
} catch(RuntimeException e) {
// Problem when starting the audio capture
}
// Get an array of available video sources
Media media = Media.getInstance(context); // Get the instance with your application context
ArrayList<VideoSource> videoSources = media.getVideoSources();
// Choose the preferred audio source from the array
VideoSource videoSource = videoSources.get(0); // Get the first source
// Get capabilities of the available video sources, such as width, height, and frame rate
ArrayList<VideoCapabilities> capabilities = videoSource.getCapabilities();
Camera.Parameters parameters = videoSource.getParameters(); // Optionally, get the camera parameters
// Set the preferred capability; not setting any capability object results in setting the first one from the list
videoSource.setCapability(capabilities.get(0)); // Set the first capability from the list
// Start capturing video
VideoTrack videoTrack;
try {
videoTrack = (VideoTrack) videoSource.startCapture();
} catch(RuntimeException e) {
// Problem when starting the video capture
}
// Handle switching between cameras
class SwitchHdl implements VideoSource.SwitchCameraHandler {
@Override
public void onCameraSwitchDone(boolean b) {}
@Override
public void onCameraSwitchError(String s) {}
}
videoSource.switchCamera(new SwitchHdl());
videoSource.changeFormat(width, height, fps);
3. Set logger
Optionally, set your own logger function to print Dolby.io Real-time Streaming logs according to the severity. By default, the SDK prints the standard output, where the severity is displayed first and then the message.
Logger.setLoggerListener((String msg, LogLevel level) -> {
String logTag = "[SDK][Log][L:" + level + "] ";
logD(TAG, logTag + msg);
});
4. Publish a stream
Create a publisher object and set a listener object to the publisher to receive proper events. This requires creating a class that inherits the publisher's listener interface. Then, create a stream in your Dolby.io developer dashboard or using the Dolby.io Streaming REST API and set your credentials.
PubListener listener = new PubListener();
Publisher publisher = Publisher.createPublisher(listener);
// Set a listener object to the publisher to receive proper events
public class PubListener implements Publisher.Listener {
public PubListener() {}
@Override
public void onPublishing() {}
@Override
public void onPublishingError(String s) {}
@Override
public void onConnected() {}
@Override
public void onConnectionError(String reason) {}
@Override
public void onSignalingError(String s) {}
@Override
public void onStatsReport(RTCStatsReport statsReport) {}
@Override
public void onViewerCount(int count) {}
@Override
public void onActive() {}
@Override
public void onInactive() {}
}
// Get the credentials structure from your publisher instance, fill it in, and set the modified credentials
Publisher.Credential creds = publisher.getCredentials();
creds.streamName = "streamName"; // The name of the stream you want to publish
creds.token = "aefea56153765316754fe"; // The publishing token
creds.apiUrl = "https://director.millicast.com/api/director/publish"; // The publish API URL
publisher.setCredentials(creds);
5. Configure your publishing session
Get a list of the available codecs and set the codecs that you want to use. By default, the SDK uses VP8 as the video codec and Opus as the audio codec.
Additionally, to publish several sources from the same application, create a publisher instance for each source. We recommend enabling discontinuous transmission that detects audio input and sends audio only when it is detected.
Publisher.Option publisherOption = new Publisher.Option();
// Get a list of codecs
Media media = Media.getInstance(context); // Get the media instance with the application context
ArrayList<String> videoCodecs = media.getSupportedVideoCodecs();
ArrayList<String> audioCodecs = media.getSupportedAudioCodecs();
// Choose the preferred codecs
publisherOption.videoCodec = Optional.of(videoCodecs.get(0)); // Set the first video codec
publisherOption.audioCodec = Optional.of(audioCodecs.get(0)); // Set the first audio codec
// If you want to support multi-source, set a source ID of the publisher
publisherOption.sourceId = "sourceId";
// To publish several sources from the same application, create a publisher instance for each source
publisherOption.dtx = true;
// Enable stereo
publisherOption.stereo = true;
// Set the selected options to the publisher
publisher.setOptions(publisherOption);
6. Add the audio and video track
Add the audio and video track that you created earlier when you started capturing media.
publisher.addTrack(videoTrack);
publisher.addTrack(audioTrack);
7. Authenticate using the Director API
Authenticate to access Dolby.io Real-time Streaming through the Director API.
publisher.connect();
Successful authentication results in opening a WebSocket connection that allows using the Dolby.io Real-time Streaming server and calling the listener's on_connected
method.
8. Start publishing
publisher.publish();
Once the publisher starts sending media, the SDK calls the listener's on_publishing
method.
Getting started with subscribing
Follow these steps to add the subscribing capability to your application.
1. Create a subscriber object
SubListener listener = new SubListener();
Subscriber subscriber = Subscriber.createSubscriber(listener);
2. Create a listener class
Create a viewer's listener class by inheriting the viewer listener's interface.
public class SubListener implements Subscriber.Listener {
public SubListener() {}
@Override
public void onSubscribed() {}
@Override
public void onSubscribedError(String s) {}
@Override
public void onConnected() {}
@Override
public void onConnectionError(String reason) {}
@Override
public void onStopped() {}
@Override
public void onSignalingError(String s) {}
@Override
public void onStatsReport(RTCStatsReport statsReport) {}
@Override
public void onTrack(VideoTrack videoTrack, Optional<String> mid) {}
@Override
public void onTrack(AudioTrack audioTrack, Optional<String> mid) {}
@Override
public void onActive(String streamId, String[] tracks, Optional<String> sourceId) {}
@Override
public void onInactive(String streamId, Optional<String> sourceId) {}
@Override
public void onLayers(String mid, LayerData[] activeLayers, LayerData[] inactiveLayers) {}
@Override
public void onVad(String mid, Optional<String> sourceId) {}
@Override
public void onViewerCount(int count) {}
}
3. Set up credentials
Get your stream name and stream ID from the dashboard and set them up in the SDK.
Subscriber.Credential creds = this.subscriber.getCredentials();
creds.accountId = "streamName"; // The name of the stream you want to subscribe to
creds.streamName = "ACCOUNT"; // The ID of your Dolby.io Real-time Streaming account
creds.apiUrl = "https://director.millicast.com/api/director/subscribe"; // The subscribe API URL
subscriber.setCredentials(creds);
4. Configure the viewer by setting your preferred options
Configure your stream to receive multi-source content.
Subscriber.Option optionSubscriber = new Subscriber.Option();
optionSubscriber.pinnedSourceId = Optional.of("mainSource"); // The main source that will be received by the default media stream
optionSubscriber.multiplexedAudioTrack = 3; // Enables audio multiplexing and denotes the number of audio tracks to receive as Voice Activity Detection (VAD) multiplexed audio
optionSubscriber.excludedSourceId = new String[]{ "excluded" }; // Audio streams that should not be included in the multiplex, for example your own audio stream
// Set the selected options
subscriber.setOptions(optionSubscriber);
5. Create a WebSocket connection
Authenticate and create a WebSocket connection to connect with the Dolby.io Real-time Streaming server.
subscriber.connect();
If the connection fails, the listener's on_authentication_failed
method is called with the HTTP error code and failure message. If the code is 0, double-check your internet connection or the API URL set in the credentials. If the connection is successful, the SDK calls the on_authenticated
method.
6. Subscribe to the streamed content
subscriber.subscribe();
When the operation is successful, the SDK calls on_subscribed
and sends you an event in the listener with the created audio and video tracks. Otherwise, the SDK calls on_subscribed_error
with an error message.
7. Project media
If publishers use the multi-source feature, you need to project tracks into a specified transceiver using its media ID (mid). By default, if you do not project anything, you receive no media. When you start subscribing, you receive the active
event with the track IDs and the source ID. In order to project a track into a transceiver, you must use the project
method of the viewer. You need to specify the source ID you are targeting and an array of the tracks you want to project.
By default, only one video and audio track is negotiated in the SDP. If there are several publishers sending media in one stream, you can dynamically add more tracks using the add_remote_track
method each time you receive an active event. The method adds a new transceiver and renegotiates locally SDP. When successful, the SDK creates a new track and calls the on_track
callback, so you can get the track and its corresponding mid.
// Get mid either from the `on_track` callback of the listener object or by calling the `get_mid` method with the track ID
// Option 1
public class SubListener implements Subscriber.Listener {
/* ... */
@Override
public void onTrack(VideoTrack videoTrack, Optional<String> mid) {
// Store the mid value somewhere
}
/* ... */
}
// Option 2
String mid = subscriber.getMid(track.getName()).get();
// Project a video track
ArrayList<Subscriber.ProjectionData> projectionDataArray = new ArrayList<>();
Subscriber.ProjectionData projectionData = new Subscriber.ProjectionData();
projectionData.mid = mid; // The media ID of the transceiver you want to project into
projectionData.media = "video"; // The media track type, either video or audio
projectionData.trackId = trackId; // The name of the track on the media server side, which is the track ID you get in the active event
subscriber.project(sourceId, projectionDataArray);
subscriber.addRemoteTrack("video"); // "audio" or "video" depending on the type of track you want to add
To stop projecting the track, call the unproject
method, which requires an array of the media IDs that you want to stop projecting.
8. Select a layer that you want to receive
When a publisher uses Simulcast or the SVC mode when sending a video feed, the media server automatically chooses the right layer to send to the viewer according to the bandwidth estimation. However, you can force the server to send you a specific layer by calling the select
method.
For example, if the sender uses Simulcast, it is possible to receive three different encoding IDs: 'h' for the high resolution, 'm' for the medium one, and 'l' for the low. In order to choose the medium resolution, you have to do the following:
LayerData layerData = new LayerData();
layerData.encoding_id = "m"; // The encoding ID, which is the ID of the Simulcast layer
layerData.temporal_layer_id = 1; // The ID of the temporary layer
layerData.spatial_layer_id = 0; // The ID of the spatial layer
subscriber.select(Optional.of(layerData));
// Empty optional means the server will make an automatic selection
You can retrieve all the available layers with their corresponding IDs by calling the layer
event.
9. Manage broadcast events
When broadcast events occur, the SDK calls the corresponding callback in the listener object. The SDK listens to all events and does not allow disabling them; it offers the following event listeners:
- Publisher event listeners:
on_active
: called when the first viewer starts viewing a streamon_inactive
: called when the last viewer stops viewing a streamon_viewer_count
: called each time the number of viewers changes; all clients connected to the stream are notified about the current number of viewers
- Viewer event listeners:
on_active
: called when a new source starts publishing a stream; it contains the stream ID, the track information, and the source IDon_inactive
: called when a source is no longer published within a stream; it contains the stream ID and the source IDon_stopped
: called when a stream stopson_vad
: called when a source ID is multiplexed into an audio track based on the voice activity level; it contains mid of the track and the source IDon_layers
: called when Simulcast or SVC layers are available; contains arrays of theLayerData
object that you can use in theselect
commandon_viewer_count
: called each time a new viewer enters or leaves a stream; all clients connected to the stream are notified about the current number of viewers
10. Render video
The SDK provides an interface that lets you implement a class responsible for receiving video frames.
<!-- Write a class that inherits _VideoRenderer_ -->
<!-- Add a layout in your fragment to add a view later in Java code -->
<LinearLayout
android:id="@+id/linear_layout_video"
android:layout_width="match_parent"
android:layout_height="0dp"
android:layout_weight="0.6"
android:orientation="horizontal">
</LinearLayout>
After this step, you will get the VideoFrame
whenever it is available. You can render the frame using any graphic library.
To get video data from the VideoFrame
, use the get_buffer
method. Allocate a buffer with the correct size beforehand. You can get the size of the buffer according to its VideoType
using the size
method. Both get_buffer
and size
have a template parameter that lets you choose in which VideoType
you want your video data, either millicast::VideoType::ARGB
or millicast::VideoType::I420
.
Then, create an instance of your renderer and add it to a local or remote video track, select one playback device to be able to play audio, and adjust the volume of remote tracks.
// Create an instance of your renderer and add it to a local or remote video track
linearLayoutVideo = view.findViewById(R.id.linear_layout_video);
VideoRenderer renderer = new VideoRenderer(context); // Create a renderer with the application context
linearLayoutVideo.addView(renderer);
// When you get your video track
videoTrack.setRenderer(renderer);
// Select one playback device to be able to play audio
Media media = Media.getInstance(applicationContext);
ArrayList<AudioPlayback> audioPlayback = media.getAudioPlayback();
audioPlayback.initPlayback();
// Adjust the volume of remote tracks
audioTrack.setVolume(1.0); // The volume should be between 0 and 1
Collecting RTC statistics
You can periodically collect the WebRTC peer connection statistics if you enable them through the getStats method of the viewer or publisher. After enabling the statistics, you will get a report every second through the onStatsReport callback in the listener object. The identifiers and way to browse the stats are following the RTC specification.
Updated 12 days ago