Blog Post

AI - Azure AI services Blog
5 MIN READ

VoiceRAG: An App Pattern for RAG + Voice Using Azure AI Search and the GPT-4o Realtime API for Audio

pablocastro's avatar
pablocastro
Icon for Microsoft rankMicrosoft
Oct 01, 2024
The new Azure OpenAI gpt-4o-realtime-preview model opens the door for even more natural application user interfaces with its speech-to-speech capability.   This new voice-based interface...
Updated Oct 02, 2024
Version 4.0
azure ai search
azure openai service
ronalgonzalez's avatar
ronalgonzalez
Copper Contributor
Nov 01, 2024

I have recently downloaded this project and I have installed everything just as you say in your github, but I have a doubt when I do the execution it shows me this error

 

Starting backend

INFO:voicerag:Running in development mode, loading from .env file
======== Running on http://localhost:8765 ========
(Press CTRL+C to quit)
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler    
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages     
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler    
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages     
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler    
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages     
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler    
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages     
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler    
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages     
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 208, in _websocket_handler    
    await self._forward_messages(ws)
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\app\backend\rtmt.py", line 180, in _forward_messages     
    async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "C:\Users\RonalGonzalez\Documents\projects\web-development\nodejs\TrackingwithSocketsio\aisearch-openai-rag-audio\.venv\Lib\site-packages\aiohttp\client.py", line 848, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url=URL('wss://aihubrealtimem6781383663.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')

 

What can I do to solve it?