Blog Post

Azure AI Foundry Blog
5 MIN READ

VoiceRAG: An App Pattern for RAG + Voice Using Azure AI Search and the GPT-4o Realtime API for Audio

pablocastro's avatar
pablocastro
Icon for Microsoft rankMicrosoft
Oct 01, 2024
The new Azure OpenAI gpt-4o-realtime-preview model opens the door for even more natural application user interfaces with its speech-to-speech capability.   This new voice-based interface...
Updated Oct 02, 2024
Version 4.0
azure ai search
azure openai service
ronalgonzalez's avatar
ronalgonzalez
Copper Contributor
Nov 01, 2024

Super, I love this post, I have cloned and followed the following steps but I have encountered a problem, I don't know if you could help me and I would be grateful.

 

I have been doing all the necessary steps referenced in the github and this is exactly the data I'm getting from azure AI Studio Deployment 

https://github.com/Azure-Samples/aisearch-openai-rag-audio

 

and I'm getting the error I'm attaching at the end

 

 

 

 

Starting backend

INFO:voicerag:Running in development mode, loading from .env file
INFO:voicerag:Using DefaultAzureCredential
INFO:azure.identity._credentials.environment:No environment configuration found.
INFO:azure.identity._credentials.managed_identity:ManagedIdentityCredential will use IMDS
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED'
Request method: 'GET'
Request headers:
    'User-Agent': 'azsdk-python-identity/1.18.0 Python/3.11.9 (Windows-10-10.0.22631-SP0)'
No body was attached to the request
INFO:azure.identity._credentials.chained:DefaultAzureCredential acquired a token from AzureCliCredential
======== Running on http://localhost:8765 ========
(Press CTRL+C to quit)
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:18 -0600] "GET /audio-processor-worklet.js HTTP/1.1" 304 177 "http://localhost:8765/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:18 -0600] "GET /audio-playback-worklet.js HTTP/1.1" 304 177 "http://localhost:8765/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:31 -0600] "GET / HTTP/1.1" 200 235 "-" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:31 -0600] "GET /assets/index-MbNbqoGC.css HTTP/1.1" 200 237 "http://localhost:8765/" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:31 -0600] "GET /assets/index-BrCwpeIS.js HTTP/1.1" 200 253 "http://localhost:8765/" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:31 -0600] "GET /assets/index-BrCwpeIS.js HTTP/1.1" 200 253 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:31 -0600] "GET /favicon.ico HTTP/1.1" 304 178 "http://localhost:8765/" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 Edg/130.0.0.0"
INFO:aiohttp.access:::1 [01/Nov/2024:10:09:31 -0600] "GET /assets/index-BrCwpeIS.js.map HTTP/1.1" 200 243 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0"
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request