Raw Media Access (Audio and Video) is available in Public Preview

Public preview of audio media access is now complemented by the availability of video media access in public preview. Developers can now access the full raw media stream for incoming and outgoing audio and video media content during a call.

 

The video media access API provides support for developers to get real-time access to video streams to capture, analyze, and process video content during active calls. Developers can access the incoming call video stream directly on the call object and send custom outgoing video stream during the call. This sets the foreground services to support a variety of video and audio manipulation. Outgoing video access can be captured and implemented with screen sharing, background blur, and video filters before being published to the recipient, allowing viewers to build privacy into their calling experience. In more complex scenarios, video access can be fitted with a virtual environment to support augmented reality. Spatial audio can be injected into remote incoming audio to add music to enhance a waiting room lobby.

 

In one example, like a contact center scenario, these streams can be used to run custom AI models for analysis such as your homegrown natural language processing (NLP) for conversation analysis or provide real-time insights and suggestions to boost agent productivity. In virtual appointments, media streams can be used to analyze sentiment when providing virtual care for patients or provide remote assistance during video calls leveraging Mixed Reality capabilities. This also opens a path for developers to leverage newer innovations with endless possibilities to enhance interaction experiences.

 

Learn more in the Raw media access Overview and QuickStart guide.