It is not very intuitive on how to obtain the raw audio data from microphone and put it on a stream. The documentation does not clearly show the way on how to use MLAudioInput for the purpose of raw data manipulation. It seemed to me that MLAudioInput.BufferClip is the class to get the data with the GetData() API.
Here is the sample code snippet (Inspired from the AudioCaptureExample)
private void FixedUpdate()
{
if (_startConvertSignal)
{
ReadBuffer();
}
}
private void StartMicrophone()
{
var captureType = MLAudioInput.MicCaptureType.VoiceCapture;
mlAudioClip = new MLAudioInput.BufferClip(MLAudioInput.MicCaptureType.VoiceCapture, AUDIO_CLIP_LENGTH_SECONDS, MLAudioInput.GetSampleRate(captureType));
}
private int detectionAudioPosition = 0;
// Fixed update 20ms
// Frequency = number of samples per second
// 1000ms => SAMPLE_RATE
// 1ms => SAMPLE_RATE / 1000
// 16ms => SAMPLE_RATE * 20 / 1000
private readonly float[] detectionAudioSamples = new float[SAMPLE_RATE * 20 / 1000];
private void ReadBuffer()
{
int readSampleCount = mlAudioClip.GetData(detectionAudioSamples, detectionAudioPosition, out int nextPosition);
if (readSampleCount > 0)
{
HandleAudioBuffer(detectionAudioSamples, 2);
}
detectionAudioPosition = nextPosition;
}
The HandleAudioBuffer method will pass the data buffer to a stream which plays back on a remote client.
The result is that the remote client receives repetitive non-sense audio, not what I speak to the headset.
Please provide some guidance/code snippet to properly obtain the data stream.
Thank you for letting us know about the issue. Our 1.1.0 update will include a GetData function to AudioInputBufferClip that does not automatically wrap the audio data and instead sends you exactly what is in the audio buffer. The update is slated to go live before November 17th.
Just adding this post to get updates on this topic, while sharing our approach for reference;
We use the Streamingclip which we then play on a audiosource (with mixer volume set to 0), and then we use audioSource.GetOutputData(samples, 1) to get the float data (which we in our case need to convert back to byte[ ]).
So what we really need is a GetData which serves a byte array - (or just access to the microphone through the default unity/android API; Unity - Scripting API: Microphone - which currently doesn't work).
Hi Christian, I am on the latest v1.1.0 update now. Still using the GetData() API is very cumbersome work on. But I found a better approach. Looking at the source code of MLAudioInput.cs, the abstract method protected abstract void OnReceiveSamples(float[] samples) is actually the exact function I need. Is it possible to create a public callback event with the same signature? I could just register to that callback and get the samples at the exact right timing, size and format. Better yet, it is even better that callback is passing the sample in short array instead of float, from CopyMLAudioInputBuffer(), because I would convert the float to short in the raw PCM audio processing.
@rick I submitted your feedback but want to make sure I have all of the details. Do you have the specifics on what difficulty you were having while working with the new GetData() call? The GetData() call should provide the same samples that OnReceiveSamples would receive, so I want to make sure that our team has all the information they need
@kbabilinski I would need to keep track of the position of the read from GetData, because it is driven by the application level timing. I would have to create my own event to extract it from the buffer, which is in very Update() frame. It is not very accurate and I am getting weird results. Programming in this way require some steps to trial and test. If the data is received from a callback, it is much more certain and self contained.
Update on the implementation - I am now extending the class so that the callback is available to me, and this works very well. Code here:
/// <summary>
/// Extending BufferClip class for callback function
/// </summary>
public class ML2BufferClip : MLAudioInput.BufferClip
{
public ML2BufferClip(MLAudioInput.MicCaptureType captureType, int lengthSec, int frequency) : this(captureType, (uint)lengthSec, (uint)frequency, (uint)MLAudioInput.GetChannels(captureType)) { }
public ML2BufferClip(MLAudioInput.MicCaptureType captureType, uint samplesLengthInSeconds, uint sampleRate, uint channels)
: base(captureType, samplesLengthInSeconds, sampleRate, channels) { }
public event Action<float[]> OnReceiveSampleCallback;
protected override void OnReceiveSamples(float[] samples)
{
base.OnReceiveSamples(samples);
if (OnReceiveSampleCallback != null)
{
OnReceiveSampleCallback(samples);
}
}
}
Thank you for those details. You touched on it earlier but note that the callback occurs outside of the main thread.(Incase anyone stubbles on this thread when implementing their own script)
This is quite interesting as we did the exact same approach.
We are adding the array to a buffer that holds the entire audio so in the update so you can do what's needed without any issue of the main thread or anything
using UnityEngine.Events;
namespace UnityEngine.XR.MagicLeap
{
public partial class MLAudioInput
{
public class StreamingClip : Clip
{
public AudioClip UnityAudioClip { get; }
private readonly CircularBuffer<float> circularBuffer;
private readonly object bufferLock = new object();
public UnityEvent<float[]> OnNewData = new UnityEvent<float[]>();
public StreamingClip(MicCaptureType captureType, int lengthSec, int frequency) : this(captureType, (uint)lengthSec, (uint)frequency, (uint)GetChannels(captureType)) { }
public StreamingClip(MicCaptureType captureType, uint samplesLengthInSeconds, uint sampleRate, uint channels) : base(captureType, samplesLengthInSeconds, channels)
{
uint sampleCount = samplesLengthInSeconds * sampleRate;
circularBuffer = new CircularBuffer<float>(sampleCount);
UnityAudioClip = AudioClip.Create("ML Streamed AudioClip", (int)sampleCount, (int)channels, (int)sampleRate, true, OnAudioRead);
}
protected override void OnReceiveSamples(float[] samples)
{
lock (bufferLock)
{
circularBuffer.Enqueue(samples);
OnNewData?.Invoke(samples);
}
}
void OnAudioRead(float[] data)
{
lock (bufferLock)
{
circularBuffer.Dequeue(data);
}
}
}
}
}
I just wanna add that after updating to 1.1.0-dev2 and unity SDK 1.2.0, the latest "AudioCapture" sample still doesn't playback in realtime - there's still about a second delay.