Pre-processing the Local Audio Stream

Suggest Edits

The Dolby.io Communications SDK 3.9 for Android and iOS allows manipulating the local participant's audio stream before sending the stream through an open session. You may use it to build your own voice effects or speech recognition to remove or bleep out curse words from the sent audio. The following steps present a basic implementation of this feature.

Prerequisites

The feature requires using one of the following SDKs:

Android Client SDK 3.9 or later
iOS Client SDK 3.9 or later

Enabling pre-processing

To pre-process your audio, implement a delegate or a callback responsible for processing the local audio stream. On iOS, add an implementation that conforms the LocalInputAudioProcessor protocol to your class and uses the process method. On Android, implement the LocalInputAudioProcessor instance.

class CustomAudioProcessor: LocalInputAudioProcessor {

 func process(frame: AudioFrame) {
   guard let data = frame.data else { return frame }
   let count = Int(frame.dataByteSize) / MemoryLayout<Int16>.size
   let pointer = data.bindMemory(to: Int16.self, capacity: count)
   let buffer = UnsafeMutableBufferPointer(start: pointer, count: count)
   ... // Modify the audio frame buffer here.
 }
}

LocalInputAudioProcessor callback = new LocalInputAudioProcessor() {
    @Override
    public byte[] onSample(LocalInputAudioSamples sample) {
        // The sample provides access to getAudioFormat(), getChannelCount(), getData(), and getSampleRate
        return sample.getData()
    }
};

val audioSampleProcessor = LocalInputAudioProcessor {
    // This provides access to audioFormat, channelCount, data, and sampleRate
    // and return the manipulated samples
    it.data
}

Then, start manipulating the local audio stream by using the audio processor. Only one instance of the processor can be registered at a time. On iOS, set your class as an audio processor for VoxeetSDK.shared.audio.local. On Android, call setLocalInputAudioProcessor using VoxeetSDK.audio().getLocal().

let processor = CustomAudioProcessor() // Your instance
VoxeetSDK.shared.audio.local.processor = processor

LocalInputAudioProcessor processor = // Your instance
VoxeetSDK.audio().getLocal().setLocalInputAudioProcessor(processor);

val processor = LocalInputAudioProcessor { it.data }
VoxeetSDK.audio().local.setLocalInputAudioProcessor(processor)

After this step, the delegate and the callback will be called whenever the local participant's audio is processed in a conference, allowing you to modify audio samples in real time.

Disabling pre-processing

To stop pre-processing audio, set the processor to nil on iOS or call the setLocalInputAudioProcessor with a null value on Android.

VoxeetSDK.shared.audio.local.processor = nil

VoxeetSDK.audio().getLocal().setLocalInputAudioProcessor(null);

VoxeetSDK.audio().getLocal().setLocalInputAudioProcessor(null)

Updated 4 months ago