Magic Leap issues with updating voice intents

[rp1]

Hi,
We have encountered a problem with updating Magic Leap voice command settings.
We only update these voice commands once in a while, i.e. irregularly and not in short succession. With a mixture of new voice intentions and previously configured commands.

1. For updating the settings we are stopping processing, passing the settings and start processing again:

std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}], "sys_intent_list":{"name":[]}";
settings.app_intent = json.c_str();
...
MLVoiceIntentStopProcessing(voice_handle_);
...
MLVoiceIntentConfigureSettings(voice_handle_, &settings);
...
MLVoiceIntentStartProcessing(voice_handle_);

Starting with ID=0 and including a previously configured voice intent is not working for us.
With the 2nd update, commands are not recognised by Magic Leap (no callback):

1. update 
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"end","value":"end"}, {"id":2,"name":"start","value":"start"}], "sys_intent_list":{"name":[]}";

2. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}, {"id":2,"name":"stop","value":"stop"}], "sys_intent_list":{"name":[]}";

3. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"quit","value":"quit"}, {"id":2,"name":"pause","value":"pause"}], "sys_intent_list":{"name":[]}";

What works is adding new voice intents, not removing commands, using continous IDs, and sending the entire json with voice intentions added, like this:

std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, "sys_intent_list":{"name":[]}";

2. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}], "sys_intent_list":{"name":[]}";

3. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}, {"id":2,"name":"stop","value":"stop"}], "sys_intent_list":{"name":[]}";

4. update
std::string json = "{"app_intents":[{"id":0,"name":"mute","value":"mute"}, {"id":1,"name":"next","value":"next"}, {"id":2,"name":"stop","value":"stop"},  {"id":3,"name":"back","value":"back"}], "sys_intent_list":{"name":[]}";
....

However, with this the list of intents is growing over time. I'm worried about a upper limit for the number of configured voice intents.

Are my assumptions correct regarding ids for the voice intent json

1. List of intents start with ID zero
2. IDs must be continuous, not skipping a number
3. list of intends must not contain duplicate names or values with different IDs

Thanks,
Richard

ML2 OS version: 1.3.0-dev1(secure)
MLSDK version: 1.2.0-dev2(secure)
Host OS: MacOS

[etucker]

@rp1

List of intents start with ID zero

This is not a requirement.

@rp1

IDs must be continuous, not skipping a number

This is not a requirement.

@rp1

list of intends must not contain duplicate names or values with different IDs

It may still work but it is not advised.

As far as your issues regarding adding voice intents at runtime. I will reach out to our developers and see what is the best practice when it comes to this.

Thanks,

El

[etucker]

Hi @rp1,

Currently, changing voice intents at runtime is supported for developers but not for end users. We are working on improving this feature over time.

Best,

El

Edit 6/28: I erroneously stated that changing voice intents at runtime is not supported. Edited to apply a more nuanced answer.

[rp1]

@etucker
Thanks again,
Richard

[etucker]

Just to be sure, You are trying to change voice intent configurations at runtime correct? I don't want to steer you wrong by assuming.

[rp1]

Yes, exactly. We want to change the voice intent configuration at runtime.

[ejurinsky]

@rp1 please make sure you first call MLVoiceIntentStopProcessing() then reconfigure with new set and then call MLVoiceIntentStartProcessing(). It should work.

[ejurinsky]

@rp1 i see you do exactly what I suggested. Let me double check and get back to you.

[ejurinsky]

@rp1 pls try to re-register callbacks.

      MLVoiceIntentCallbacks voiceCb;
      MLVoiceIntentCallbacksInit(&voiceCb);
      voiceCb.on_voice_event =
        [](const MLVoiceIntentEvent* event, void* data) {
          //.....
        };
      auto result = MLVoiceIntentSetCallbacks(voiceHandle_, &voiceCb, this);

then

      MLVoiceIntentSettings settings;
      MLVoiceIntentSettingsInit(&settings);
      settings.app_intent = json_file_content.c_str();
      auto res = MLVoiceIntentConfigureSettings(voiceHandle_, &settings);

and only then call MLVoiceIntentStartProcessing()

let us know if it does not work. we have an internal test app that we can share if you still experience issues.

[rp1]

We tested re-registering the callback when updating the voice intent configuration.
But with out success. It results in voice intents not getting recognised (callback not called).

Thanks,
Richard.

[ejurinsky]

@rp1 please see attached for test app that show how to switch things at runtime. the app reads a number of json files from the assets folder and lets you switch between those.

make sure your system language is set to english.

voice.zip (93.6 KB)
.

[rp1]

Thanks for the sample source code!

We can not reproduce the error with your sample app. Call sequence is similar (identical) compared to our implementation.

Although, we encountered some strange behaviour with json parsing. Characters like “+” causing problems. Adding a “+” in the name or value field breaks the voice intent (it looks like). We don’t receive callbacks after that, even if we try to switch to new voice targets with your sample code.

{“app_intents”:[
{“id”:0,“name”:“Home+“,”value”:“Home”}
],“sys_intent_list”:{“name”:[]}}

Another issue occurs when passing empty app_intents list. MLVoiceIntentConfigureSettings() returns MLResult_Ok, but stops working.

{“app_intents”:[],“sys_intent_list”:{“name”:[]}}

If we try to intentionally pass an invalid json string into MLVoiceIntentConfigureSettings(), we get MLResult_Ok as return value, although the call does not work.

We tested it with latest OS and SDK version 1.3.0

Thanks,
Richard

[rp1]

Is it on purpose MLVoiceIntentConfigureSettings() always returning MLResult_Ok with the above mentioned configuration issues?

Thanks,
Richard

[ejurinsky]

@rp1 let me take a look and I will get back to you.

[etucker]

Hi @rp1,

I would like to start by saying thank you for bringing this to our attention. I am going to ask a few follow-up questions so we can better understand your use case.

1. What is the workflow that you are trying to create here?
2. What are the critical voice command features that you would need to have a working application.

Any and all feedback is helpful to ensure that we can make the Magic Leap 2 experience better for developers and users.

Thanks,
El

[rp1]

Hi @etucker

1. What is the workflow that you are trying to create here?
   We have an application where the user navigates through "workflows".
   A workflow consists of several steps. Each step is presented with written instructions for the user and various option buttons that an admin defines.
   The buttons represent different choices that the user can select.
   The idea is to let the user speak the actions that are currently represented as buttons. These actions are created dynamically and can change with each screen.

2. What are the critical voice command features that you would need to have a working application.
   We need to change the list of voice commands with each new step (screen).
   A new list of voice commands can also contain already used commands, e.g. for navigating through workflows (back, cancel, ok,..).

Thanks again,
Richard

[etucker]

Thank you for your quick response. This information is very helpful for us to understand how to develop moving forward so that we accommodate as many use cases as possible. I'll go ahead and pass this over to the team.

Best,

El

[rp1]

Hi @etucker,

Are there any updates regarding the voice commands?
Thanks again for the great support!

Best,
Richard

[etucker]

Hi @rp1,

Your patience is greatly appreciated. I would like to inform you of our Voice Intent Development Kit v1.0 available in the Magic Leap Hub for developing your own app intents.

Let me know if you have any questions.

Best,

El

[ejurinsky]

@rp1 based on your feedback we are also adding validation for MLVoiceIntentConfigureSettings(). In subsequent releases the call will fail if you pass

{“app_intents”:[
{“id”:0,“name”:“Home+“,”value”:“Home”}
],“sys_intent_list”:{“name”:[]}}

as "+" is invalid symbol. the toolkit @etucker mentioned above has validation capabilities.