Tutorial — Conversation Client Plugin

Augmented Traveler local tutorial. Back to Unreal — Tutorials · the service: AI Mind — Process · how the figure uses it: Tutorial — Authoring a Q&A Experience with Ambient Idle

How to build ATConversation, a small C++ plugin that connects a figure in Unreal to the studio's conversation service. It does four jobs:

Draft. The code below is a skeleton showing the Unreal parts. Fix the message protocol with the AI engineer first (section 2). Compile and test it on the kiosk PC. Unreal engineer + AI engineer.

1. Why a C++ plugin

2. The protocol (agree it first)

One WebSocket per conversation turn or session, wss:// in production:

Direction Message Payload
→ service start (JSON) figure ID, venue ID, session ID, language, audio format (16 kHz mono PCM16)
→ service audio frames (binary) 20–40 ms of PCM each
→ service end_of_speech (JSON) — (push-to-talk released, or end of speech detected)
← service transcript (JSON) recognised question (shown briefly as a caption)
← service answer_meta (JSON) answer ID, authored-or-generated flag, gesture tag, refusal flag, citation IDs
← service caption (JSON) text chunks with timing
← service audio frames (binary) TTS audio, 24 kHz mono PCM16 (agree the rate)
← service answer_end (JSON) —
← service error (JSON) code; the client falls back (section 6)

If an authored answer matches, the service sends answer_meta with authored: true and a sequence ID. Unreal then plays that pre-animated Level Sequence instead of streaming audio (Tutorial — Sequencer for Monologues).

3. Create the plugin

  1. Edit → Plugins → + Add → Blank (a C++ plugin). Name it ATConversation. It goes in AT_Studio/Plugins/.
  2. In ATConversation.uplugin: module type Runtime, loading phase Default. Also enable the engine plugin AudioCapture.
  3. In ATConversation.Build.cs, add the dependencies:
PublicDependencyModuleNames.AddRange(new string[] {
    "Core", "CoreUObject", "Engine",
    "WebSockets", "HTTP", "Json", "JsonUtilities",
    "AudioCapture", "AudioCaptureCore", "AudioMixer"
});

4. The component (skeleton)

One UATConversationComponent, added to BP_AT_<Figure> (the child Blueprint of the MetaHuman; Tutorial — MetaHumans in Unreal).

// ATConversationComponent.h
#pragma once
#include "Components/ActorComponent.h"
#include "IWebSocket.h"
#include "ATConversationComponent.generated.h"

class USoundWaveProcedural;
class UAudioComponent;

DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnATText, const FString&, Text);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnATAnswerMeta, const FString&, GestureTag, bool, bAuthored);
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnATSignal);

UCLASS(ClassGroup=(AT), meta=(BlueprintSpawnableComponent))
class ATCONVERSATION_API UATConversationComponent : public UActorComponent
{
    GENERATED_BODY()
public:
    UPROPERTY(EditAnywhere, Category="AT") FString ServiceUrl;   // from kiosk config, not content
    UPROPERTY(EditAnywhere, Category="AT") FString FigureId;
    UPROPERTY(EditAnywhere, Category="AT") int32 TtsSampleRate = 24000;

    UPROPERTY(BlueprintAssignable) FOnATText OnTranscript;
    UPROPERTY(BlueprintAssignable) FOnATText OnCaption;
    UPROPERTY(BlueprintAssignable) FOnATAnswerMeta OnAnswerMeta;
    UPROPERTY(BlueprintAssignable) FOnATSignal OnAnswerEnd;
    UPROPERTY(BlueprintAssignable) FOnATSignal OnServiceError;

    UFUNCTION(BlueprintCallable, Category="AT") void BeginQuestion();   // push-to-talk pressed
    UFUNCTION(BlueprintCallable, Category="AT") void EndQuestion();     // push-to-talk released

    UPROPERTY(BlueprintReadOnly) UAudioComponent* VoiceAudio = nullptr; // lip-sync source listens to this
protected:
    virtual void BeginPlay() override;
    virtual void EndPlay(const EEndPlayReason::Type Reason) override;
private:
    TSharedPtr<IWebSocket> Socket;
    UPROPERTY() USoundWaveProcedural* VoiceWave = nullptr;
    void HandleText(const FString& Message);
    void HandleBinary(const void* Data, SIZE_T Size, SIZE_T BytesRemaining);
};

// ATConversationComponent.cpp (key parts)
#include "ATConversationComponent.h"
#include "WebSocketsModule.h"
#include "Sound/SoundWaveProcedural.h"
#include "Components/AudioComponent.h"
#include "Serialization/JsonSerializer.h"
#include "Async/Async.h"

void UATConversationComponent::BeginPlay()
{
    Super::BeginPlay();
    VoiceWave = NewObject<USoundWaveProcedural>(this);
    VoiceWave->SetSampleRate(TtsSampleRate);
    VoiceWave->NumChannels = 1;
    VoiceWave->Duration = INDEFINITELY_LOOPING_DURATION;
    VoiceWave->bLooping = false;
    VoiceAudio = NewObject<UAudioComponent>(GetOwner());
    VoiceAudio->RegisterComponent();
    VoiceAudio->SetSound(VoiceWave);
}

void UATConversationComponent::BeginQuestion()
{
    Socket = FWebSocketsModule::Get().CreateWebSocket(ServiceUrl);
    TWeakObjectPtr<UATConversationComponent> Weak(this);
    Socket->OnConnected().AddLambda([Weak]() {
        if (Weak.IsValid()) { /* send "start" JSON; start mic capture (see section 5) */ }
    });
    Socket->OnMessage().AddLambda([Weak](const FString& Msg) {
        // Callbacks may arrive off the game thread: hop back before touching UObjects.
        AsyncTaskGameThread, [Weak, Msg]() { if (Weak.IsValid()) Weak->HandleText(Msg); };
    });
    Socket->OnRawMessage().AddLambda([Weak](const void* Data, SIZE_T Size, SIZE_T Remaining) {
        TArray<uint8> Copy((const uint8*)Data, Size);
        AsyncTaskGameThread, [Weak, Copy = MoveTemp(Copy), Remaining]( {
            if (Weak.IsValid()) Weak->HandleBinary(Copy.GetData(), Copy.Num(), Remaining);
        });
    });
    Socket->OnConnectionError().AddLambda([Weak](const FString&) {
        AsyncTaskGameThread, [Weak]() { if (Weak.IsValid()) Weak->OnServiceError.Broadcast(); };
    });
    Socket->Connect();
}

void UATConversationComponent::HandleBinary(const void* Data, SIZE_T Size, SIZE_T)
{
    // TTS PCM16 audio: queue it and start playback on the first chunk.
    VoiceWave->QueueAudio(static_cast<const uint8*>(Data), Size);
    if (!VoiceAudio->IsPlaying()) VoiceAudio->Play();
}

void UATConversationComponent::HandleText(const FString& Message)
{
    TSharedPtr<FJsonObject> Obj;
    if Create(Message), Obj) || !Obj.IsValid() return;
    const FString Type = Obj->GetStringField(TEXT("type"));
    if (Type == TEXT("transcript"))       OnTranscript.Broadcast(Obj->GetStringField(TEXT("text")));
    else if (Type == TEXT("caption"))     OnCaption.Broadcast(Obj->GetStringField(TEXT("text")));
    else if (Type == TEXT("answer_meta")) OnAnswerMeta.Broadcast(Obj->GetStringField(TEXT("gesture")), Obj->GetBoolField(TEXT("authored")));
    else if (Type == TEXT("answer_end"))  OnAnswerEnd.Broadcast();
    else if (Type == TEXT("error"))       OnServiceError.Broadcast();
}

EndQuestion() stops the microphone and sends end_of_speech. EndPlay closes the socket and calls VoiceWave->ResetAudio().

Not compiled yet. This is a skeleton. Check the exact signatures against the 5.8 API pages listed in the Sources and fix them in this page after the first build.

5. Microphone capture

6. Latency and failure

7. Wiring to the figure

8. Check before a venue build

Sources