Class SpeechlyClient

Create a new Speechly Spoken Language Understading (SLU) Client to process speech and provide the results of automatic speech recogition (ASR) and natural langugage understanding (NLU) using delegates.

Usage

Create a new SpeechlyClient instance.
Create an SLU decoder (CloudDecoder) and pass it to SpeechlyClient's Initialize(IDecoder, AudioProcessorOptions, ContextOptions, Boolean).
Attach delegates like OnSegmentChange to listen to and handle the SLU results.
Feed audio to process with ProcessAudio(Single[], Int32, Int32).
(delegates are firing up as speech is processed)
When you don't need SLU services any more call Shutdown() to free resources.

You can feed audio continuously, but control when to start and stop process speech with Start(String) and Stop() or let voice activity detection (VAD) handle that automatically by passing EnergyThresholdVAD to SpeechlyClient constructor.

Inheritance

System.Object

SpeechlyClient

Namespace: Speechly.SLUClient

Assembly: speechly-dotnet.dll

Syntax

public class SpeechlyClient : object

Constructors

| Improve this Doc View Source

SpeechlyClient(Boolean, String, AudioInfo, Boolean)

Create a new SpeechlyClient to process audio and fire delegates to provide SLU results.

Declaration

public SpeechlyClient(bool manualUpdate = false, string saveToFolder = null, AudioInfo output = null, bool debug = false)

Parameters

Type	Name	Description
System.Boolean	manualUpdate	Setting `manualUpdate = true` postpones SpeechlyClient's delegates (OnSegmentChange, OnTranscript...) until you manually run Update(). This enables you to call Unity API in SpeechlyClient's delegates, as Unity API should only be used in the main Unity thread. (Default: false)
System.String	saveToFolder	Defines a local folder to save utterance files as 16 bit, 16000 Hz mono raw. Null disables saving. (default: `null`)
AudioInfo	output
System.Boolean	debug	Enable debug prints thru Log delegate. (default: `false`)

Fields

| Improve this Doc View Source

debug

Declaration

public bool debug

Field Value

Type	Description
System.Boolean

| Improve this Doc View Source

OnEntity

Declaration

public EntityDelegate OnEntity

Field Value

Type	Description
EntityDelegate

| Improve this Doc View Source

OnIntent

Declaration

public IntentDelegate OnIntent

Field Value

Type	Description
IntentDelegate

| Improve this Doc View Source

OnSegmentChange

Read the combined results of automatic speech recoginition (ASR) and natural language detection (NLU).

You can control when to start and stop process speech either manually with Start(String) and Stop(). Alternatively, you may let SpeechlyClient control this automatically by setting setting AudioProcessorOptions.VADControlsListening to true upon SpeechlyClient.Initialize() or by calling SpeechlyClient.AdjustAudioProcessor(true).

Declaration

public SegmentChangeDelegate OnSegmentChange

Field Value

Type	Description
SegmentChangeDelegate

| Improve this Doc View Source

OnStart

Declaration

public StartDelegate OnStart

Field Value

Type	Description
StartDelegate

| Improve this Doc View Source

OnStartStream

Declaration

public StartStreamDelegate OnStartStream

Field Value

Type	Description
StartStreamDelegate

| Improve this Doc View Source

OnStop

Declaration

public StopDelegate OnStop

Field Value

Type	Description
StopDelegate

| Improve this Doc View Source

OnStopStream

Declaration

public StopStreamDelegate OnStopStream

Field Value

Type	Description
StopStreamDelegate

| Improve this Doc View Source

OnTentativeEntity

Declaration

public TentativeEntityDelegate OnTentativeEntity

Field Value

Type	Description
TentativeEntityDelegate

| Improve this Doc View Source

OnTentativeIntent

Declaration

public IntentDelegate OnTentativeIntent

Field Value

Type	Description
IntentDelegate

| Improve this Doc View Source

OnTentativeTranscript

Declaration

public TentativeTranscriptDelegate OnTentativeTranscript

Field Value

Type	Description
TentativeTranscriptDelegate

| Improve this Doc View Source

OnTranscript

Declaration

public TranscriptDelegate OnTranscript

Field Value

Type	Description
TranscriptDelegate

Properties

| Improve this Doc View Source

AudioInputStreamIdentifier

Declaration

public string AudioInputStreamIdentifier { get; }

Property Value

Type	Description
System.String

| Improve this Doc View Source

IsActive

Declaration

public bool IsActive { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

IsAudioStreaming

Declaration

public bool IsAudioStreaming { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

IsReady

Declaration

public bool IsReady { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

Output

Declaration

public AudioInfo Output { get; }

Property Value

Type	Description
AudioInfo

Methods

| Improve this Doc View Source

AdjustAudioProcessor(Nullable<Boolean>)

Control AudioProcessor parameters.

Declaration

public void AdjustAudioProcessor(bool? vadControlsListening = null)

Parameters

Type	Name	Description
System.Nullable<System.Boolean>	vadControlsListening	`true` enables VAD to control listening. `false` disables VAD feature and stops listening immediately. `null` for no change.

| Improve this Doc View Source

Initialize(IDecoder, AudioProcessorOptions, ContextOptions, Boolean)

SLU decoder instance to use like CloudDecoder.

The SLU decoder provides the automatic speech recognition (ASR) and natural language understanding (NLU) capabilities via SLU delegates (OnSegmentChange, OnTranscript...).

Declaration

public async Task Initialize(IDecoder decoder, AudioProcessorOptions audioProcessorOptions = null, ContextOptions contextOptions = null, bool preferLibSpeechlyAudioProcessor = false)

Parameters

Type	Name	Description
IDecoder	decoder	SLU decoder implementing IDecoder interface like CloudDecoder.
AudioProcessorOptions	audioProcessorOptions
ContextOptions	contextOptions
System.Boolean	preferLibSpeechlyAudioProcessor

Returns

Type	Description
Task	Task that completes when the decoder is ready.

| Improve this Doc View Source

ProcessAudio(Stream)

Declaration

public void ProcessAudio(Stream fileStream)

Parameters

Type	Name	Description
Stream	fileStream

| Improve this Doc View Source

ProcessAudio(Single[], Int32, Int32)

Process speech audio samples from a microphone or other audio source.

It's recommended to feed audio early and often instead of large chunks to benefit from real-time ASR and NLU output.

You can control when to start and stop process speech either manually with Start(String) and Stop(). Alternatively you may use automatic voice activity detection (VAD) with AudioProcessorOptions passed to Initialize(IDecoder, AudioProcessorOptions, ContextOptions, Boolean).

The audio is handled as follows:

Downsample to 16kHz if needed
Add to history ringbuffer
Calculate energy (VAD)
Automatic Start/Stop (VAD)
Send utterance audio to a file
Send utterance audio to Speechly SLU decoder

Declaration

public void ProcessAudio(float[] floats, int start = 0, int length = null)

Parameters

Type	Name	Description
System.Single[]	floats	Array of float containing samples to feed to the audio pipeline. Each sample needs to be in range -1f..1f.
System.Int32	start	Start index of audio to process in samples (default: `0`).
System.Int32	length	Length of audio to process in samples or `-1` to process the whole array (default: `-1`).

| Improve this Doc View Source

ProcessAudioFile(String)

Declaration

public void ProcessAudioFile(string fileName)

Parameters

Type	Name	Description
System.String	fileName

| Improve this Doc View Source

Shutdown()

Closes any connections (e.g. cloud SLU) and frees resources.

Declaration

public async Task Shutdown()

Returns

Type	Description
Task	Task that completes with the shutdown.

| Improve this Doc View Source

Start(String)

Start listening for user speech and feeding it to the SLU decoder. OnContextStart is triggered upon a call to Start. It's also triggered by automatic VAD activation.

You don't need to await this call if you don't need the utterance id.

Declaration

public Task<string> Start(string appId = null)

Parameters

Type	Name	Description
System.String	appId	Cloud decoder only: The Speechly app id to connect to, if not the default. If specified, you must use project id based login with cloud decoder.

Returns

Type	Description
Task<System.String>	An unique utterance id.

| Improve this Doc View Source

StartStream(String, Boolean)

StartStream should be called at start of a continuous audio stream. It resets the stream sample counters and history. For backwards compability, ProcessAudio and Start ensure it's been called. OnStreamStart delegate is triggered upon a call to StartStream.

Declaration

public void StartStream(string streamIdentifier, bool auto = false)

Parameters

Type	Name	Description
System.String	streamIdentifier
System.Boolean	auto

| Improve this Doc View Source

Stop()

Stop listening for user speech. OnContextStop is triggered upon a call to Stop. It's also triggered by automatic VAD deactivation.

You don't need to await this call if you don't need the utterance id.

Declaration

public Task<string> Stop()

Returns

Type	Description
Task<System.String>	An unique utterance id.

| Improve this Doc View Source

StopStream(Boolean)

StopStream should be called at the end of a continuous audio stream. OnStreamStop delegate is triggered upon a call to StopStream.

Declaration

public void StopStream(bool auto = false)

Parameters

Type	Name	Description
System.Boolean	auto

| Improve this Doc View Source

Update()

Call Update in your game loop to fire Speechly delegates manually if you want them to run in main UI/Unity thread.

Declaration

public void Update()