The Future of Real-time Communication in Agentic AI

The real-time communications (RTC) landscape is undergoing a transformation, driven by advances in artificial intelligence, codec development, and transport protocols. This evolution is reshaping how we think about and implement real-time communication solutions. This post is based on the thoughts from a talk from a few weeks ago at IIT Chicago’s yearly Real-time Communication Conference

Keynote - From Codecs to Conversations: AI-Driven WebRTC Unleashed

TL;DR:
• AI integration is enabling contextual, intelligent communication experiences
• Neural audio codecs from Microsoft (Satin) and Google (Lyra, SoundStream) are setting new standards
• Protocol evolution shows a clear progression toward more flexible, efficient solutions
• Voice AI and real-time AI agents are becoming fundamental to modern RTC applications

The Rise of Voice AI in RTC

Voice AI has evolved from basic speech recognition into sophisticated real-time interaction systems. Modern Voice AI systems can now understand context, manage conversations, and execute complex tasks - all while maintaining natural dialogue flow. This transformation has enabled use cases from intelligent meeting facilitation to real-time language translation.

At Daily.co, we’ve embraced this evolution through our daily-python SDK, which creates a seamless bridge between WebRTC media streams and Python-based AI processing. This integration enables developers to capture high-quality audio/video streams and process them through sophisticated AI pipelines. The SDK’s integration with our WebRTC infrastructure allows for real-time AI processing while maintaining low latency and high quality.

Protocol Evolution: From WebSockets to WebTransport

The evolution of real-time communication protocols reflects our deepening understanding of latency, reliability, and scalability requirements in modern applications. Each advancement has brought us closer to optimal real-time communication, though with different trade-offs and capabilities.

WebSockets introduced bi-directional communication over TCP, marking a significant improvement over HTTP polling. However, its TCP foundation presents inherent limitations for real-time applications. TCP’s reliability mechanisms, particularly head-of-line blocking, can introduce significant latency as a single lost packet blocks all subsequent data until retransmission. The protocol’s strict ordering and inherited TCP congestion control mechanisms, while excellent for reliable data transfer, aren’t optimized for real-time media delivery where some packet loss might be acceptable in favor of lower latency.

WebRTC emerged as a purpose-built solution for real-time communications, introducing UDP-based communication with specialized protocols. It incorporates DTLS for security and NAT traversal for connectivity, along with built-in congestion control algorithms designed specifically for real-time media. While these features made WebRTC ideal for communicating with users in any network.

WebTransport represents the latest evolution, combining the strengths of its predecessors while addressing their limitations. Built on QUIC, it enables multiple streams over a single connection, eliminating the connection overhead of WebSockets. Its modern architecture prevents head-of-line blocking through independent stream multiplexing and offers flexible reliability options - developers can choose between reliable ordered, reliable unordered, or unreliable delivery based on their specific needs. The protocol includes modern congestion control and built-in encryption, achieving lower latency than WebSockets while maintaining simpler implementation requirements than WebRTC.

Codec Innovation: The Next Generation

The latest generation of audio codecs represents a significant leap forward in compression efficiency and quality. Microsoft’s Project Satin leverages AI to achieve remarkable quality at extremely low bitrates, while maintaining computational efficiency crucial for mobile devices. Google’s Lyra codec demonstrates how neural networks can revolutionize audio compression, particularly in challenging network conditions.

These innovations are particularly significant for emerging markets and areas with limited bandwidth infrastructure, where traditional codecs struggle to deliver acceptable quality.

AI Integration: Beyond Voice Recognition

Modern AI integration in RTC has fundamentally transformed how we approach real-time communications. Real-time emotion analysis has evolved from simple facial recognition to sophisticated systems that can analyze vocal tone, facial micro-expressions, and body language to provide meaningful insights about participant engagement and meeting dynamics.

Advanced noise suppression has made significant strides through neural network approaches. Unlike traditional statistical methods, AI-powered noise suppression can now understand the context of different sounds, distinguishing between a dog barking in the background versus someone speaking off-camera, or separating multiple speakers in a crowded environment.

AI-powered bandwidth optimization represents another breakthrough, using predictive algorithms to anticipate network conditions and adjust video and audio parameters preemptively. This results in smoother transitions during network fluctuations and better overall quality compared to traditional reactive approaches.

Codec Evolution: The Neural Audio Revolution

The landscape of audio and video codecs has undergone a dramatic transformation, marked by both traditional improvements and neural network-based approaches.

In video compression, the evolution from VP8 to VP9 and now AV1 represents a significant advancement in open-source video coding. AV1, developed by the Alliance for Open Media (AOMedia), achieves approximately 30% better compression than VP9 and 50% better than H.264, while remaining royalty-free. This makes it particularly attractive for WebRTC applications where licensing costs can be prohibitive. Major streaming platforms including YouTube and Netflix have already adopted AV1, demonstrating its effectiveness at scale.

The VP family of codecs, particularly VP9, continues to serve as a crucial bridge technology. VP9 offers significant bandwidth savings over its predecessor VP8, while maintaining broad hardware support across devices. This makes it particularly valuable for real-time communications where hardware acceleration is essential for battery life and performance.

On the neural codec front, Microsoft’s Project Satin represents a significant advancement in audio codec technology, utilizing AI to achieve compression ratios previously thought impossible while maintaining voice quality. Satin’s neural networks can reconstruct high-quality audio from extremely compressed data streams, making it particularly effective for mobile devices and low-bandwidth scenarios.

Google’s Lyra codec approaches the challenge differently, using machine learning models to synthesize speech in the frequency domain. This allows for incredibly efficient compression while maintaining natural-sounding voice reproduction. Lyra can operate at bitrates as low as 3kbps while maintaining intelligibility, making it transformative for areas with limited connectivity.

In the emerging space of neural video codecs, companies like NVIDIA are developing AI-powered video compression that can reduce bandwidth requirements by up to 50% compared to traditional H.264/H.265 codecs. These neural video codecs analyze frame content to predict and reconstruct video elements more efficiently than traditional block-based approaches.

The evolution of these codecs represents more than just incremental improvements in compression efficiency. They fundamentally change how we think about media compression, moving from mathematical transforms to neural network-based understanding and reconstruction of audio and video content. This shift enables new possibilities in low-bandwidth scenarios and opens doors for more sophisticated real-time communication applications.

The future of codec development likely lies in hybrid approaches that combine the reliability and hardware support of traditional codecs with the efficiency gains possible through neural network-based compression. This combination could provide the best of both worlds: consistent performance across devices with the advanced compression capabilities of AI-powered solutions.

Looking Ahead

The future of RTC technology presents exciting possibilities as these technologies converge. I am particularly excited about the potential of combining these advances with our python framework, which enables sophisticated voice and multimodal AI agents in real-time communications.

For developers looking to leverage these emerging technologies, it’s crucial to partner with platforms that not only embrace innovation but also provide stable, production-ready implementations. As a community, we remain committed to providing developers with cutting-edge tools while maintaining the reliability and performance essential for business-critical applications.

Links to relevant sections of the talk

• AI-driven codecs and network optimizations - Learn how Opus codec enhancements, Microsoft’s Satin, and Google’s Lyra and Sound Stream are improving quality while reducing bitrates
• Object detection and tracking capabilities - Explore advanced video processing features including background manipulation and content moderation
• Audio transcription as a fundamental feature - See how transcription enables noise cancellation, multi-language captioning, and real-time translation
• LLMs as REST API alternatives - Understanding how LLMs simplify WebRTC application development
• Real-time background enhancement with AI - Learn about using generative AI to improve video quality with limited bandwidth
• AI-powered content moderation - Implementing real-time moderation using machine learning models
• Custom LLM training for specific domains - How retraining improves accuracy for specialized use cases
• On real-time background manipulation: "I can actually talk to an LLM and say something like clean up my background and generate it in real-time while it is processing my video feed"
• On audio transcription evolution: "Audio transcriptions are becoming part and parcel of the game, being able to detect and track speakers speech"
• On clinical applications: "We’ve exposed an API which takes the transcript uh takes the recording or in real time so the transcript can be real time or can be based on recordings can be processed by llm it will create a clinical notes draft"