Voice Intents Not Working with MLSDK

Unity Development
,
1

Unity Editor version: 2022.3.6f1
ML2 OS version: 1.9.0
Unity SDK version: 2.4.0
Host OS: Windows

I found the Voice Intents feature to not work in my app, using MLSDK features.

Due to the dependency of my app on Vuforia and MLCamera, I have to choose "use MLSDK (deprecated)" when importing the Magic Leap Setup Tool (not using OpenXR as a provider). To generate this post, I disabled all extra components of my app (Vuforia, MLCamera) and only tested the two example scripts provided at (1) Runtime Voice Intents Example | MagicLeap Developer Documentation (2) Simple Example | MagicLeap Developer Documentation and my configuration is exactly like this Runtime Configuration | MagicLeap Developer Documentation
image

It appears to me that there's some issue with the generation of the json file that leads to the problem. Attached is the two log files I got - seems that it (1) at runtime, wasn't able to create the json file at run time:

09-26 17:36:09.066 10117 16222 16243 I Unity   : Voice Intent Example Started
09-26 17:36:09.066 10117 16222 16243 I Unity   : Example:Start()
09-26 17:36:09.066 10117 16222 16243 I Unity   : 
09-26 17:36:09.070 10117 16222 16243 E Unity   : NullReferenceException: Object reference not set to an instance of an object.
09-26 17:36:09.070 10117 16222 16243 E Unity   :   at MLVoiceIntentsConfiguration.AddSlotsToJSON () [0x00000] in <00000000000000000000000000000000>:0 
09-26 17:36:09.070 10117 16222 16243 E Unity   :   at MLVoiceIntentsConfiguration.SetupJSONContainer () [0x00000] in <00000000000000000000000000000000>:0 
09-26 17:36:09.070 10117 16222 16243 E Unity   :   at MLVoiceIntentsConfiguration.GetJSONString () [0x00000] in <00000000000000000000000000000000>:0 
09-26 17:36:09.070 10117 16222 16243 E Unity   :   at UnityEngine.XR.MagicLeap.MLVoice.SetupVoiceIntents (MLVoiceIntentsConfiguration voiceConfiguration) [0x00000] in <00000000000000000000000000000000>:0 
09-26 17:36:09.070 10117 16222 16243 E Unity   :   at Example.Start () [0x00000] in <00000000000000000000000000000000>:0 
09-26 17:36:09.070 10117 16222 16243 E Unity   : 
09-26 17:36:09.163  1000  3323  3323 I MagicFlinger: INF: Setting display power state DISPLAY_POWER_STATE_LED_OFF, reason "Client - Inactive"
09-26 17:36:09.414  1000  3665  4443 D adbnotify-ml: Shutting down listeners

and (2) if pre-configured, cannot read the configuration file correctly

09-26 17:53:57.905  4034  3419  3522 I VoiceService: voice_service.cpp:1376: [InitialValidateIntentJSONFile]---- Currently for language SUPPORTED, in-app sys intents are en_us
09-26 17:53:57.905  4034  3419  3522 I VoiceService: offline_intent_iface.cpp:724: [initialValidateIntentJSONFile]>>> SUCCESS: pass initial validation on rx intent JSON file
09-26 17:53:57.905  4034  3419  3522 I VoiceService: voice_service.cpp:947: [WriteToReqActPreProcMsgQueue] write a msg (service type = 1, session_type = 0, act_type = 4)
09-26 17:53:57.905  4034  3419  3534 I VoiceService: voice_service.cpp:942: [ReadFromReqActPreProcMsgQueue] read a msg (service type = 1, session_type = 0, act_type = 4)
09-26 17:53:57.905  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:1077: [parseJSONValue2AppGramListFiles]----- Parse json file to app_intents.grm ---
09-26 17:53:57.905  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:1116: [parseJSONValue2AppGramListFiles][LAST line] grammar = <s>({UnityApp_VoiceIntent_ID101}|{UnityApp_VoiceIntent_ID102}|{UnityApp_VoiceIntent_ID103})</s>;
09-26 17:53:57.905  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:1121: [parseJSONValue2AppGramListFiles]----- Parse json file to app_intents.list ---
09-26 17:53:57.905  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:1159: [parseJSONValue2SysListFile]----- Parse json file to sys_intent_list.list ---
09-26 17:53:57.905  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:1165: [parseJSONValue2SysListFile]-- Field exists but empty --> NO system intent is enabled
09-26 17:53:57.906  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:755: [finalParseJSONFile2AppGramAndSysListFiles]>>> SUCCESS: pass final validation on rx intent JSON file and ready to deploy!
09-26 17:53:57.906  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:760: [resetToDefaultSysGramFile]----- To use default sys intent grammar file
09-26 17:53:57.906  4034  3419  3534 I VoiceService: voice_service.cpp:1415: Input size is not 0. To apply new app intent set! Update app& sys JSON intnet file w/ this new string
09-26 17:53:57.906  4034  3419  3534 I VoiceService: offline_intent_iface.cpp:542: [updateSysIntentGramFile]----- Totally 0 ML system intents are enabled
09-26 17:53:57.906  4034  3419  3534 I VoiceService: voice_service.cpp:1431: [IntentReqActPreProcHandler]----- To request post-process type 4 for the underlying speech engine

The full log files are attached for your reference. I can also send you the zip file if necessary.
Voice_Example_2.txt (722.8 KB)
Voice_Example_1.txt (730.2 KB)

Have you been seeing this issue after the August update? I wasn't sure but before using OS 1.8.0 this did not happen to me.

Thanks in advance!

3

Hi @zhhqu0305,

Good news! I was able to reproduce this issue on my end. I will go ahead and reach out to our team to see if we can find the cause or a workaround for now. Thanks for bringing this to our attention.

5

I would try out this example:

and add the following lines of code to ensure that the SlotData is not null at runtime.

if (voiceConfiguration.SlotsForVoiceCommands == null) 
    voiceConfiguration.SlotsForVoiceCommands = new List<SlotData>()
6

Thanks for your reply!

Will give it a try shortly, but if this is the case can I use something like a configuration.xml? My app uses voice slots as well, Is there a way for me to create those at runtime?

7

You can generate a JSON file with the Voice Intent Configuration through the Voice Intent Development Toolkit : Voice Intent Development Toolkit | MagicLeap Developer Documentation

Simply pass in the json string into the MLVoice.SetupVoiceIntents() function.

You mention that you would like to create intents at runtime. You may want to try creating a MLVoiceIntentsConfiguration scriptable object in your project and referencing it in your script. Then at runtime. You can modify the Lists for the scriptable object that was already initialized.

8

Thanks for your reply. I'll try with the first option you provided, but for that scriptable object I don't think it can be properly imported, see the second log chunk I provided at the beginning of this thread.

9

The first one seems like it's an issue because some of the prosperities are not initialized on the class. We are working on updating the script on the developer portal. Here is a WIP example:

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.MagicLeap;

public class Example : MonoBehaviour
{
    // The commands that will be added to the voice configuration
    public string[] VoiceCommands = new string[] { "Select", "Change Color", "Undo", "Slot Test for {Color}"};

    // Voice Configuration that will be created at runtime.
    private MLVoiceIntentsConfiguration _voiceConfiguration;

    // Start is called before the first frame update
    void Start()
    {
        // Create a new instance of the Voice Intents Configuration
        _voiceConfiguration = ScriptableObject.CreateInstance<MLVoiceIntentsConfiguration>();

        // Initialize all lists and properties of the configuration
        InitializeVoiceConfiguration();

        // Optionally, add slots for voice commands
        AddSlotsForCommands();

        // Add voice commands to the configuration
        AddVoiceCommands();

        // Configure the voice intents based on the new configuration
        MLResult result = MLVoice.SetupVoiceIntents(_voiceConfiguration);

        if (result.IsOk)
        {
            MLVoice.OnVoiceEvent += OnVoiceEvent;
        }
        else
        {
            Debug.LogError("Failed to Setup Voice Intents with result: " + result);
        }
    }

    // Initialize all lists and properties of the voice configuration
    private void InitializeVoiceConfiguration()
    {
        _voiceConfiguration.VoiceCommandsToAdd = new List<MLVoiceIntentsConfiguration.CustomVoiceIntents>();
        _voiceConfiguration.AllVoiceIntents = new List<MLVoiceIntentsConfiguration.JSONData>();
        _voiceConfiguration.SlotsForVoiceCommands = new List<MLVoiceIntentsConfiguration.SlotData>();
        _voiceConfiguration.SystemCommands = 0; // Initialize with no system commands enabled
        _voiceConfiguration.AutoAllowAllSystemIntents = false; // Default to not auto-allowing all system intents
    }

    private void AddSlotsForCommands(){
        MLVoiceIntentsConfiguration.SlotData slotData = new MLVoiceIntentsConfiguration.SlotData
        {
            name = "Color",
            values = new List<string> { "Red", "Blue", "Green" }
        };
        _voiceConfiguration.SlotsForVoiceCommands.Add(slotData);

    }

    // Add voice commands from the array to the configuration
    private void AddVoiceCommands()
    {
        for (int i = 0; i < VoiceCommands.Length; i++)
        {
            MLVoiceIntentsConfiguration.CustomVoiceIntents newIntent = new MLVoiceIntentsConfiguration.CustomVoiceIntents
            {
                Value = VoiceCommands[i],
                Id = (uint)i
            };

            _voiceConfiguration.VoiceCommandsToAdd.Add(newIntent);
        }
    }

    // Write a message to the log when a command is detected
    private void OnVoiceEvent(in bool wasSuccessful, in MLVoice.IntentEvent voiceEvent)
    {
        Debug.Log("Detected Voice Command : " + voiceEvent.EventName);
    }
}
10

Also make sure that Voice Commands are enabled in the system settings and that the permission is granted

11

So using the json string I was able to setup voice intents as I needed. Thanks for your help! Looking forward to the fix on the example. Would be better if that MLVoiceIntentsConfiguration scriptable object can come back to work :smiley:

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.