Class SpeechlyClient
Create a new Speechly Spoken Language Understading (SLU) Client to process speech and provide the results of automatic speech recogition (ASR) and natural langugage understanding (NLU) using delegates.
Usage
- Create a new SpeechlyClient instance.
- Create an SLU decoder (CloudDecoder) and pass it to
SpeechlyClient
's Initialize(IDecoder). - Attach delegates like OnSegmentChange to listen to and handle the SLU results.
- Feed audio to process with ProcessAudio(Single[], Int32, Int32, Boolean).
- (delegates are firing up as speech is processed)
- When you don't need SLU services any more call Shutdown() to free resources.
You can feed audio continuously, but control when to start and stop process speech with StartContext(String) and StopContext() or let voice activity detection (VAD) handle that automatically by passing EnergyTresholdVAD to SpeechlyClient constructor.
Inheritance
Namespace: Speechly.SLUClient
Assembly: speechly-unity-net-standard-2.0.dll
Syntax
public class SpeechlyClient : object
Constructors
| Improve this Doc View SourceSpeechlyClient(Int32, Int32, Int32, Boolean, String, EnergyTresholdVAD, Boolean)
Create a new SpeechlyClient to process audio and fire delegates to provide SLU results.
Declaration
public SpeechlyClient(int frameMillis = 30, int historyFrames = 5, int inputSampleRate = 16000, bool manualUpdate = false, string saveToFolder = null, EnergyTresholdVAD vad = null, bool debug = false)
Parameters
Type | Name | Description |
---|---|---|
System.Int32 | frameMillis | Size of one audio frame (default: |
System.Int32 | historyFrames | Count of the audio history frames (default: |
System.Int32 | inputSampleRate | Define the sample rate of incoming audio (default: |
System.Boolean | manualUpdate | Setting |
System.String | saveToFolder | Defines a local folder to save utterance files as 16 bit, 16000 hZ mono raw. Null disables saving. (default: |
EnergyTresholdVAD | vad | EnergyTresholdVAD instance to control automatic listening on/off. Null disables VAD. (default: |
System.Boolean | debug | Enable debug prints thru Log delegate. (default: |
Fields
| Improve this Doc View SourceOnEntity
Declaration
public EntityDelegate OnEntity
Field Value
Type | Description |
---|---|
EntityDelegate |
OnIntent
Declaration
public IntentDelegate OnIntent
Field Value
Type | Description |
---|---|
IntentDelegate |
OnSegmentChange
Read the combined results of automatic speech recoginition (ASR) and natural language detection (NLU).
You can control when to start and stop process speech either manually with StartContext(String) and StopContext() or automatically by providing a voice activity detection (VAD) field to SpeechlyClient.
Declaration
public SegmentChangeDelegate OnSegmentChange
Field Value
Type | Description |
---|---|
SegmentChangeDelegate |
OnStartContext
Declaration
public StartContextDelegate OnStartContext
Field Value
Type | Description |
---|---|
StartContextDelegate |
OnStartStream
Declaration
public StartStreamDelegate OnStartStream
Field Value
Type | Description |
---|---|
StartStreamDelegate |
OnStopContext
Declaration
public StopContextDelegate OnStopContext
Field Value
Type | Description |
---|---|
StopContextDelegate |
OnStopStream
Declaration
public StopStreamDelegate OnStopStream
Field Value
Type | Description |
---|---|
StopStreamDelegate |
OnTentativeEntity
Declaration
public TentativeEntityDelegate OnTentativeEntity
Field Value
Type | Description |
---|---|
TentativeEntityDelegate |
OnTentativeIntent
Declaration
public IntentDelegate OnTentativeIntent
Field Value
Type | Description |
---|---|
IntentDelegate |
OnTentativeTranscript
Declaration
public TentativeTranscriptDelegate OnTentativeTranscript
Field Value
Type | Description |
---|---|
TentativeTranscriptDelegate |
OnTranscript
Declaration
public TranscriptDelegate OnTranscript
Field Value
Type | Description |
---|---|
TranscriptDelegate |
Properties
| Improve this Doc View SourceAudioInputStreamIdentifier
Declaration
public string AudioInputStreamIdentifier { get; }
Property Value
Type | Description |
---|---|
System.String |
IsActive
Declaration
public bool IsActive { get; }
Property Value
Type | Description |
---|---|
System.Boolean |
IsAudioStreaming
Declaration
public bool IsAudioStreaming { get; }
Property Value
Type | Description |
---|---|
System.Boolean |
IsReady
Declaration
public bool IsReady { get; }
Property Value
Type | Description |
---|---|
System.Boolean |
SamplesSent
Declaration
public int SamplesSent { get; }
Property Value
Type | Description |
---|---|
System.Int32 |
StreamSamplePos
Declaration
public int StreamSamplePos { get; }
Property Value
Type | Description |
---|---|
System.Int32 |
UtteranceSerial
Declaration
public int UtteranceSerial { get; }
Property Value
Type | Description |
---|---|
System.Int32 |
Vad
Declaration
public EnergyTresholdVAD Vad { get; }
Property Value
Type | Description |
---|---|
EnergyTresholdVAD |
Methods
| Improve this Doc View SourceInitialize(IDecoder)
SLU decoder instance to use like CloudDecoder.
The SLU decoder provides the automatic speech recognition (ASR) and natural language understanding (NLU) capabilities via SLU delegates (OnSegmentChange, OnTranscript...).
Declaration
public async Task Initialize(IDecoder decoder)
Parameters
Type | Name | Description |
---|---|---|
IDecoder | decoder | SLU decoder implementing IDecoder interface like CloudDecoder. |
Returns
Type | Description |
---|---|
Task | Task that completes when the decoder is ready. |
ProcessAudio(Stream)
Declaration
public void ProcessAudio(Stream fileStream)
Parameters
Type | Name | Description |
---|---|---|
Stream | fileStream |
ProcessAudio(Single[], Int32, Int32, Boolean)
Process speech audio samples from a microphone or other audio source.
It's recommended to constantly feed new audio as long as you want to use Speechly's SLU services.
You can control when to start and stop process speech either manually with StartContext(String) and StopContext() or automatically by providing a voice activity detection (VAD) field to SpeechlyClient.
The audio is handled as follows:
- Downsample to 16kHz if needed
- Add to history ringbuffer
- Calculate energy (VAD)
- Automatic Start/StopContext (VAD)
- Send utterance audio to a file
- Send utterance audio to Speechly SLU decoder
Declaration
public void ProcessAudio(float[] floats, int start = 0, int length = null, bool forceSubFrameProcess = false)
Parameters
Type | Name | Description |
---|---|---|
System.Single[] | floats | Array of float containing samples to feed to the audio pipeline. Each sample needs to be in range -1f..1f. |
System.Int32 | start | Start index of audio to process in samples (default: |
System.Int32 | length | Length of audio to process in samples or |
System.Boolean | forceSubFrameProcess | StopStream(Boolean) internally uses this to force processing of last subframe at end of audio stream (default: |
ProcessAudioFile(String)
Declaration
public void ProcessAudioFile(string fileName)
Parameters
Type | Name | Description |
---|---|---|
System.String | fileName |
Shutdown()
Closes any connections (e.g. cloud SLU) and frees resources.
Declaration
public async Task Shutdown()
Returns
Type | Description |
---|---|
Task | Task that completes with the shutdown. |
StartContext(String)
Start listening for user speech and feeding it to the SLU decoder.
OnContextStart
is triggered upon a call to StartContext. It's also triggered by automatic VAD activation.
You don't need to await this call if you don't need the utterance id.
Declaration
public Task<string> StartContext(string appId = null)
Parameters
Type | Name | Description |
---|---|---|
System.String | appId | Cloud decoder only: The Speechly app id to connect to, if not the default. If specified, you must use project id based login with cloud decoder. |
Returns
Type | Description |
---|---|
Task<System.String> | An unique utterance id. |
StartStream(String, Boolean)
StartStream
should be called at start of a continuous audio stream. It resets the stream sample counters and history. For backwards compability, ProcessAudio and StartContext ensure it's been called.
OnStreamStart
delegate is triggered upon a call to StartStream.
Declaration
public void StartStream(string streamIdentifier, bool auto = false)
Parameters
Type | Name | Description |
---|---|---|
System.String | streamIdentifier | |
System.Boolean | auto |
StopContext()
Stop listening for user speech.
OnContextStop
is triggered upon a call to StopContext. It's also triggered by automatic VAD deactivation.
You don't need to await this call if you don't need the utterance id.
Declaration
public Task<string> StopContext()
Returns
Type | Description |
---|---|
Task<System.String> | An unique utterance id. |
StopStream(Boolean)
StopStream
should be called at the end of a continuous audio stream.
OnStreamStop
delegate is triggered upon a call to StopStream.
Declaration
public void StopStream(bool auto = false)
Parameters
Type | Name | Description |
---|---|---|
System.Boolean | auto |
Update()
Call Update in your game loop to fire Speechly delegates manually if you want them to run in main UI/Unity thread.
Declaration
public void Update()