Why Voice AI Does Not Need SFUs

TL;DR: Voice AI applications should bypass SFUs and connect directly to voice bots via WebRTC, reducing jitter buffer delays and simplifying infrastructure whilst leveraging WebRTC’s battle-tested last-mile optimisations.

Recently in conversation with Emil Ivov, we discussed if SFUs are needed. He builds and operates one at Jitsi, while at Daily.co, we do the same. We recently open-sourced Pipecat.ai, which made this conversation more releavant. Foreshadowing, we largely agreed that architecture changes are afoot.

The rise of Voice AI has brought new attention to an old debate in real-time communications: when do we actually need a Selective Forwarding Unit (SFU)? As someone who’s spent years optimising WebRTC infrastructure, I’ve watched the pendulum swing from peer-to-peer (P2P) to SFU and now I think, back to P2P. Now, with Voice AI reshaping how we think about real-time communication, it’s time to reconsider whether SFUs are truly necessary for every use case.

The Traditional Role of SFUs

Selective Forwarding Units emerged with group video calling. In a typical scenario, each participant sends multiple video streams at different quality levels, what we call simulcast. Each stream varies in target bitrate because they have different frame rates and resolutions, simplifying congestion control for the sender, and allowing the SFU to forward the appropriate quality to each receiver based on their bandwidth capacity. For a five-person video call, this architecture makes perfect sense: the SFU acts as an intelligent traffic router, ensuring everyone gets the best possible experience without overwhelming anyone’s connection.

But here’s the thing: not all calls involve five people. In fact, the vast majority of WebRTC sessions are between just two participants. This is where the story gets interesting.

WebRTC P2P4121

Many communication platforms developed what Emil Ivov (from Jitsi) called the WebRTC P2P4121 feature, peer-to-peer connections for one-to-one communication. The logic was sound: why route traffic through a server when two endpoints can communicate directly? The server’s bandwidth savings alone made this attractive, not to mention the potential latency improvements.

Yet P2P had stumbling blocks, corporate firewalls and mobile networks threw up barriers that often required TURN servers to relay traffic anyway. If you’re already maintaining TURN infrastructure to punch through NATs and firewalls, the argument went, why not consolidate everything through SFUs? You’d have one infrastructure to scale and maintain instead of two.

This consolidation made sense in the pre-Voice AI era. But the landscape shifted dramatically during the pandemic (covid) era, wherein, last mile issues became more pronounced and routing through SFUs even for two person calls became more of the norm.

Why Voice AI Changes Everything

Voice AI presents a fundamentally different communication pattern. When a human speaks to an AI agent, we’re not dealing with a symmetric conversation between two endpoints behind unpredictable NATs. Instead, we have: an AI agent running on a server with a public IP address and no immediate need for multi-party capabilities in most use cases. We still have last-mile issues with the human participant, ergo, we should use WebRTC, but prefer P2P connections.

Think about it: your Voice AI agent like Pipecat is running on infrastructure you control, hence no real issues with firewalls. When discussing P2P versus SFU routing, there is hop-by-hop latency but oftentimes SFUs and AI agents co-locate, they often sit in the same region within the same cloud provider. Daily.co for example, supports P2P connections and is available in 40+ regions across two cloud providers. However, the additional latency from routing through an SFU might only be a few milliseconds. My concern is not so much the latency through the extra hop, but the additional jitter buffer that each hop may add in the worst-case scenario.

Every server in your media path maintains its own jitter buffer to handle out-of-order packets. When a packet with a higher sequence number arrives, the server will wait to determine if the missing packets are lost or not. This will slightly delay the packet. At the next hop, the same process repeats, but in this case, the endpoint may either drop the packet because it past its playout time or correctly playout the packet. In the worst case, the jitter buffers may interact poorly, each adding delay as they attempt to smooth out network inconsistencies. Thus, just having one jitter buffer at the endpoints is better.

Consider a scenario where network conditions cause packets to arrive slightly out of order. The SFU’s jitter buffer holds packets for, say, 40 milliseconds to reorder them. Then those packets travel to the endpoints, where network jitter causes another reordering delay. Suddenly, you’ve added 40-60 milliseconds to your end-to-end latency—not from transmission time, but from buffering. The actual buffering delay depends on the NetEq implementations, typically, there is a high- and low-watermarks to control the amount of buffering. The high watermark is the maximum amount of buffering allowed for error-resilience (retx, fec), while the low watermark is the minimum amount of buffering allowed to ensure smooth playout (below this, the buffer underruns and you’ve no audio to playback).

A direct P2P connection eliminates this redundancy. The AI agent’s and the human participant’s WebRTC stack handles all the jitter compensation in one place, making decisions with full visibility into the end-to-end connection quality.

The good news is that most modern WebRTC platforms are beginning to recognise this nuance. Daily’s WebRTC transport, for instance, supports starting calls as P2P connections and seamlessly upgrading to SFU routing when a third participant joins. This hybrid approach gives you the best of both worlds: optimal performance for two-party conversations and the scalability of SFUs when needed (third party joins, recording, etc).

This seamless transition is crucial for Voice AI applications. Imagine a customer service scenario where an AI agent handles initial queries via P2P, then smoothly brings in a human supervisor when needed. The infrastructure adapts to the use case rather than forcing all conversations through the same architectural pattern.

Implementation Considerations

When implementing P2P connections for Voice AI, consider these factors:

Direct WebRTC connections to AI agents require proper signalling infrastructure. Your AI agent needs to handle WebRTC negotiation directly, which platforms like Pipecat already support. This isn’t significantly more complex than SFU integration, but it does require thinking about your architecture differently.

Monitor your connection success rates carefully. While AI agents on public IPs should have high P2P success rates, some client networks might still pose challenges. Have a fallback strategy, whether that’s TURN servers or SFU routing, for the small percentage of connections that can’t establish P2P.

Design your system to handle transitions gracefully. If you start with P2P and need to add participants later, ensure your application can migrate to SFU routing without disrupting the user experience.

Looking Forward

The key insight for Voice AI developers is this: use WebRTC for what it does best—handling last-mile networking challenges—without automatically adopting the full SFU-centric architecture that evolved for different use cases.

WebRTC gives you a congestion control algorithm implemented in each endpoint, echo cancellation and noise suppression, and NAT traversal capabilities when needed

You don’t need to reinvent these wheels. But you also don’t need to route every packet through an SFU just because that’s become the default architecture for video conferencing.

There’s also the elephant in the room that I’ve deliberately avoided until now: SIP. The resurrection of SIP in modern communications infrastructure adds another dimension to this discussion. But that’s a topic that deserves its own deep dive—perhaps in a future post.