Class SpeechlyClient

Create a new Speechly Spoken Language Understading (SLU) Client to process speech and provide the results of automatic speech recogition (ASR) and natural langugage understanding (NLU) using delegates.

Usage

Create a new SpeechlyClient instance.
Create an SLU decoder (CloudDecoder) and pass it to SpeechlyClient's Initialize(IDecoder).
Attach delegates like OnSegmentChange to listen to and handle the SLU results.
Feed audio to process with ProcessAudio(Single[], Int32, Int32, Boolean).
(delegates are firing up as speech is processed)
When you don't need SLU services any more call Shutdown() to free resources.

You can feed audio continuously, but control when to start and stop process speech with StartContext(String) and StopContext() or let voice activity detection (VAD) handle that automatically by passing EnergyTresholdVAD to SpeechlyClient constructor.

Inheritance

System.Object

SpeechlyClient

Namespace: Speechly.SLUClient

Assembly: speechly-unity-net-standard-2.0.dll

Syntax

public class SpeechlyClient : object

Constructors

| Improve this Doc View Source

SpeechlyClient(Int32, Int32, Int32, Boolean, String, EnergyTresholdVAD, Boolean)

Create a new SpeechlyClient to process audio and fire delegates to provide SLU results.

Declaration

public SpeechlyClient(int frameMillis = 30, int historyFrames = 5, int inputSampleRate = 16000, bool manualUpdate = false, string saveToFolder = null, EnergyTresholdVAD vad = null, bool debug = false)

Parameters

Type	Name	Description
System.Int32	frameMillis	Size of one audio frame (default: `30` ms). Total history duration will be historyFrames*frameSamples. History is sent upon StartContext to capture the start of utterance which especially important with VAD, which activates with a constant delay.
System.Int32	historyFrames	Count of the audio history frames (default: `5`). Total history duration will be historyFrames * frameSamples.
System.Int32	inputSampleRate	Define the sample rate of incoming audio (default: `16000`)
System.Boolean	manualUpdate	Setting `manualUpdate = true` postpones SpeechlyClient's delegates (OnSegmentChange, OnTranscript...) until you manually run Update(). This enables you to call Unity API in SpeechlyClient's delegates, as Unity API should only be used in the main Unity thread. (Default: false)
System.String	saveToFolder	Defines a local folder to save utterance files as 16 bit, 16000 hZ mono raw. Null disables saving. (default: `null`)
EnergyTresholdVAD	vad	EnergyTresholdVAD instance to control automatic listening on/off. Null disables VAD. (default: `null`)
System.Boolean	debug	Enable debug prints thru Log delegate. (default: `false`)

Fields

| Improve this Doc View Source

OnEntity

Declaration

public EntityDelegate OnEntity

Field Value

Type	Description
EntityDelegate

| Improve this Doc View Source

OnIntent

Declaration

public IntentDelegate OnIntent

Field Value

Type	Description
IntentDelegate

| Improve this Doc View Source

OnSegmentChange

Read the combined results of automatic speech recoginition (ASR) and natural language detection (NLU).

You can control when to start and stop process speech either manually with StartContext(String) and StopContext() or automatically by providing a voice activity detection (VAD) field to SpeechlyClient.

Declaration

public SegmentChangeDelegate OnSegmentChange

Field Value

Type	Description
SegmentChangeDelegate

| Improve this Doc View Source

OnStartContext

Declaration

public StartContextDelegate OnStartContext

Field Value

Type	Description
StartContextDelegate

| Improve this Doc View Source

OnStartStream

Declaration

public StartStreamDelegate OnStartStream

Field Value

Type	Description
StartStreamDelegate

| Improve this Doc View Source

OnStopContext

Declaration

public StopContextDelegate OnStopContext

Field Value

Type	Description
StopContextDelegate

| Improve this Doc View Source

OnStopStream

Declaration

public StopStreamDelegate OnStopStream

Field Value

Type	Description
StopStreamDelegate

| Improve this Doc View Source

OnTentativeEntity

Declaration

public TentativeEntityDelegate OnTentativeEntity

Field Value

Type	Description
TentativeEntityDelegate

| Improve this Doc View Source

OnTentativeIntent

Declaration

public IntentDelegate OnTentativeIntent

Field Value

Type	Description
IntentDelegate

| Improve this Doc View Source

OnTentativeTranscript

Declaration

public TentativeTranscriptDelegate OnTentativeTranscript

Field Value

Type	Description
TentativeTranscriptDelegate

| Improve this Doc View Source

OnTranscript

Declaration

public TranscriptDelegate OnTranscript

Field Value

Type	Description
TranscriptDelegate

Properties

| Improve this Doc View Source

AudioInputStreamIdentifier

Declaration

public string AudioInputStreamIdentifier { get; }

Property Value

Type	Description
System.String

| Improve this Doc View Source

IsActive

Declaration

public bool IsActive { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

IsAudioStreaming

Declaration

public bool IsAudioStreaming { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

IsReady

Declaration

public bool IsReady { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

SamplesSent

Declaration

public int SamplesSent { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

StreamSamplePos

Declaration

public int StreamSamplePos { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

UtteranceSerial

Declaration

public int UtteranceSerial { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

Vad

Declaration

public EnergyTresholdVAD Vad { get; }

Property Value

Type	Description
EnergyTresholdVAD

Methods

| Improve this Doc View Source

Initialize(IDecoder)

SLU decoder instance to use like CloudDecoder.

The SLU decoder provides the automatic speech recognition (ASR) and natural language understanding (NLU) capabilities via SLU delegates (OnSegmentChange, OnTranscript...).

Declaration

public async Task Initialize(IDecoder decoder)

Parameters

Type	Name	Description
IDecoder	decoder	SLU decoder implementing IDecoder interface like CloudDecoder.

Returns

Type	Description
Task	Task that completes when the decoder is ready.

| Improve this Doc View Source

ProcessAudio(Stream)

Declaration

public void ProcessAudio(Stream fileStream)

Parameters

Type	Name	Description
Stream	fileStream

| Improve this Doc View Source

ProcessAudio(Single[], Int32, Int32, Boolean)

Process speech audio samples from a microphone or other audio source.

It's recommended to constantly feed new audio as long as you want to use Speechly's SLU services.

The audio is handled as follows:

Downsample to 16kHz if needed
Add to history ringbuffer
Calculate energy (VAD)
Automatic Start/StopContext (VAD)
Send utterance audio to a file
Send utterance audio to Speechly SLU decoder

Declaration

public void ProcessAudio(float[] floats, int start = 0, int length = null, bool forceSubFrameProcess = false)

Parameters

Type	Name	Description
System.Single[]	floats	Array of float containing samples to feed to the audio pipeline. Each sample needs to be in range -1f..1f.
System.Int32	start	Start index of audio to process in samples (default: `0`).
System.Int32	length	Length of audio to process in samples or `-1` to process the whole array (default: `-1`).
System.Boolean	forceSubFrameProcess	StopStream(Boolean) internally uses this to force processing of last subframe at end of audio stream (default: `false`).

| Improve this Doc View Source

ProcessAudioFile(String)

Declaration

public void ProcessAudioFile(string fileName)

Parameters

Type	Name	Description
System.String	fileName

| Improve this Doc View Source

Shutdown()

Closes any connections (e.g. cloud SLU) and frees resources.

Declaration

public async Task Shutdown()

Returns

Type	Description
Task	Task that completes with the shutdown.

| Improve this Doc View Source

StartContext(String)

Start listening for user speech and feeding it to the SLU decoder. OnContextStart is triggered upon a call to StartContext. It's also triggered by automatic VAD activation.

You don't need to await this call if you don't need the utterance id.

Declaration

public Task<string> StartContext(string appId = null)

Parameters

Type	Name	Description
System.String	appId	Cloud decoder only: The Speechly app id to connect to, if not the default. If specified, you must use project id based login with cloud decoder.

Returns

Type	Description
Task<System.String>	An unique utterance id.

| Improve this Doc View Source

StartStream(String, Boolean)

StartStream should be called at start of a continuous audio stream. It resets the stream sample counters and history. For backwards compability, ProcessAudio and StartContext ensure it's been called. OnStreamStart delegate is triggered upon a call to StartStream.

Declaration

public void StartStream(string streamIdentifier, bool auto = false)

Parameters

Type	Name	Description
System.String	streamIdentifier
System.Boolean	auto

| Improve this Doc View Source

StopContext()

Stop listening for user speech. OnContextStop is triggered upon a call to StopContext. It's also triggered by automatic VAD deactivation.

You don't need to await this call if you don't need the utterance id.

Declaration

public Task<string> StopContext()

Returns

Type	Description
Task<System.String>	An unique utterance id.

| Improve this Doc View Source

StopStream(Boolean)

StopStream should be called at the end of a continuous audio stream. OnStreamStop delegate is triggered upon a call to StopStream.

Declaration

public void StopStream(bool auto = false)

Parameters

Type	Name	Description
System.Boolean	auto

| Improve this Doc View Source

Update()

Call Update in your game loop to fire Speechly delegates manually if you want them to run in main UI/Unity thread.

Declaration

public void Update()