Magic Leap issues with updating voice intents

rp1 · June 22, 2023, 9:58am

Hi,
We have encountered a problem with updating Magic Leap voice command settings.
We only update these voice commands once in a while, i.e. irregularly and not in short succession. With a mixture of new voice intentions and previously configured commands.

For updating the settings we are stopping processing, passing the settings and start processing again:

std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}], "sys_intent_list":{"name":[]}";
settings.app_intent = json.c_str();
...
MLVoiceIntentStopProcessing(voice_handle_);
...
MLVoiceIntentConfigureSettings(voice_handle_, &settings);
...
MLVoiceIntentStartProcessing(voice_handle_);

Starting with ID=0 and including a previously configured voice intent is not working for us.
With the 2nd update, commands are not recognised by Magic Leap (no callback):

1. update 
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"end","value":"end"}, {"id":2,"name":"start","value":"start"}], "sys_intent_list":{"name":[]}";

2. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}, {"id":2,"name":"stop","value":"stop"}], "sys_intent_list":{"name":[]}";

3. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"quit","value":"quit"}, {"id":2,"name":"pause","value":"pause"}], "sys_intent_list":{"name":[]}";

What works is adding new voice intents, not removing commands, using continous IDs, and sending the entire json with voice intentions added, like this:

std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, "sys_intent_list":{"name":[]}";

2. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}], "sys_intent_list":{"name":[]}";

3. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}, {"id":2,"name":"stop","value":"stop"}], "sys_intent_list":{"name":[]}";

4. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}, {"id":2,"name":"stop","value":"stop"},  {"id":3,"name":"back","value":"back"}], "sys_intent_list":{"name":[]}";
....

However, with this the list of intents is growing over time. I'm worried about a upper limit for the number of configured voice intents.

Are my assumptions correct regarding ids for the voice intent json

List of intents start with ID zero
IDs must be continuous, not skipping a number
list of intends must not contain duplicate names or values with different IDs

Thanks,
Richard

ML2 OS version: 1.3.0-dev1(secure)
MLSDK version: 1.2.0-dev2(secure)
Host OS: MacOS

etucker · June 22, 2023, 8:05pm

This is not a requirement.

It may still work but it is not advised.

As far as your issues regarding adding voice intents at runtime. I will reach out to our developers and see what is the best practice when it comes to this.

Thanks,

El

etucker · June 22, 2023, 9:26pm

Hi @rp1,

Currently, changing voice intents at runtime is supported for developers but not for end users. We are working on improving this feature over time.

Best,

El

Edit 6/28: I erroneously stated that changing voice intents at runtime is not supported. Edited to apply a more nuanced answer.

rp1 · June 23, 2023, 7:02am

@etucker
Thanks again,
Richard

etucker · June 23, 2023, 3:03pm

Just to be sure, You are trying to change voice intent configurations at runtime correct? I don't want to steer you wrong by assuming.

rp1 · June 23, 2023, 3:06pm

Yes, exactly. We want to change the voice intent configuration at runtime.

ejurinsky · June 24, 2023, 1:48am

@rp1 please make sure you first call MLVoiceIntentStopProcessing() then reconfigure with new set and then call MLVoiceIntentStartProcessing(). It should work.

ejurinsky · June 24, 2023, 1:51am

@rp1 i see you do exactly what I suggested. Let me double check and get back to you.

ejurinsky · June 24, 2023, 2:07am

@rp1 pls try to re-register callbacks.

      MLVoiceIntentCallbacks voiceCb;
      MLVoiceIntentCallbacksInit(&voiceCb);
      voiceCb.on_voice_event =
        [](const MLVoiceIntentEvent* event, void* data) {
          //.....
        };
      auto result = MLVoiceIntentSetCallbacks(voiceHandle_, &voiceCb, this);

then

      MLVoiceIntentSettings settings;
      MLVoiceIntentSettingsInit(&settings);
      settings.app_intent = json_file_content.c_str();
      auto res = MLVoiceIntentConfigureSettings(voiceHandle_, &settings);

and only then call MLVoiceIntentStartProcessing()

let us know if it does not work. we have an internal test app that we can share if you still experience issues.

rp1 · June 30, 2023, 6:04am

We tested re-registering the callback when updating the voice intent configuration.
But with out success. It results in voice intents not getting recognised (callback not called).

Thanks,
Richard.

ejurinsky · June 30, 2023, 3:57pm

@rp1 please see attached for test app that show how to switch things at runtime. the app reads a number of json files from the assets folder and lets you switch between those.

make sure your system language is set to english.

voice.zip (93.6 KB)
.

rp1 · July 4, 2023, 12:33pm

Thanks for the sample source code!

We can not reproduce the error with your sample app. Call sequence is similar (identical) compared to our implementation.

Although, we encountered some strange behaviour with json parsing. Characters like “+” causing problems. Adding a “+” in the name or value field breaks the voice intent (it looks like). We don’t receive callbacks after that, even if we try to switch to new voice targets with your sample code.

{“app_intents”:[
{“id”:0,“name”:“Home+“,”value”:“Home”}
],“sys_intent_list”:{“name”:[]}}

Another issue occurs when passing empty app_intents list. MLVoiceIntentConfigureSettings() returns MLResult_Ok, but stops working.

{“app_intents”:[],“sys_intent_list”:{“name”:[]}}

If we try to intentionally pass an invalid json string into MLVoiceIntentConfigureSettings(), we get MLResult_Ok as return value, although the call does not work.

We tested it with latest OS and SDK version 1.3.0

Thanks,
Richard

rp1 · July 5, 2023, 1:40pm

Is it on purpose MLVoiceIntentConfigureSettings() always returning MLResult_Ok with the above mentioned configuration issues?

Thanks,
Richard

ejurinsky · July 10, 2023, 3:51pm

@rp1 let me take a look and I will get back to you.

etucker · July 11, 2023, 9:44pm

Hi @rp1,

I would like to start by saying thank you for bringing this to our attention. I am going to ask a few follow-up questions so we can better understand your use case.

What is the workflow that you are trying to create here?
What are the critical voice command features that you would need to have a working application.

Any and all feedback is helpful to ensure that we can make the Magic Leap 2 experience better for developers and users.

Thanks,
El

rp1 · July 12, 2023, 10:48am

Hi @etucker

What is the workflow that you are trying to create here?
We have an application where the user navigates through "workflows".
A workflow consists of several steps. Each step is presented with written instructions for the user and various option buttons that an admin defines.
The buttons represent different choices that the user can select.
The idea is to let the user speak the actions that are currently represented as buttons. These actions are created dynamically and can change with each screen.
What are the critical voice command features that you would need to have a working application.
We need to change the list of voice commands with each new step (screen).
A new list of voice commands can also contain already used commands, e.g. for navigating through workflows (back, cancel, ok,..).

Thanks again,
Richard

etucker · July 12, 2023, 2:53pm

Thank you for your quick response. This information is very helpful for us to understand how to develop moving forward so that we accommodate as many use cases as possible. I'll go ahead and pass this over to the team.

Best,

El

rp1 · August 18, 2023, 6:39am

Hi @etucker,

Are there any updates regarding the voice commands?
Thanks again for the great support!

Best,
Richard

etucker · August 18, 2023, 7:00pm

Hi @rp1,

Your patience is greatly appreciated. I would like to inform you of our Voice Intent Development Kit v1.0 available in the Magic Leap Hub for developing your own app intents.

Let me know if you have any questions.

Best,

El

ejurinsky · September 7, 2023, 6:54pm

@rp1 based on your feedback we are also adding validation for MLVoiceIntentConfigureSettings(). In subsequent releases the call will fail if you pass

{“app_intents”:[
{“id”:0,“name”:“Home+“,”value”:“Home”}
],“sys_intent_list”:{“name”:[]}}

as "+" is invalid symbol. the toolkit @etucker mentioned above has validation capabilities.

Topic		Replies	Views
Voice Intents Not Working with MLSDK Unity Development unity , voice-intents	10	39	October 4, 2024
Runtime Voice Intents Crash Unity Application on Command Recognition Input/Interaction development , unity	5	287	May 31, 2023
Voice Intents not working with App Simulator Unity Development unity , app-sim	2	254	April 4, 2023
MLResult_Unknown on MLVoice.SetupVoiceIntents Unity Development unity	2	220	January 20, 2024
Voice intents bug when multiple Data Slots are used in one Voice Command Unity Development development , unity	6	220	August 4, 2023

Magic Leap issues with updating voice intents

Related topics