Upgrading Fiber To The Home to terabit speeds (posted 2024-04-09)
Last week, Jaap van Till asked me if BGP would be capable of supporting the terabit class interconnectivity that he foresees we’ll need in the future, possibly due to the rise of artificial intelligence. He explains his reasoning in the blog post What Link speeds will we need for AI, where he quotes VAN TILL’s CONJECTURE:
The network connection Wide Area access speed will grow in time until it matches the internal device BUS speed of the more and more complex processors and datastores.
And then concludes that 14 Tbps external links will be required in 2039. Today I can get 4 Gbps where I live. So that means a 70% speed increase per year.
Let’s first get that BGP question out of the way: I see no problems. 25 years ago I ran BGP over 64 and 128 kbps links without trouble. Six orders of magnitude later, BGP is still fine, and there is no reason to believe that even faster speeds will be a problem, just as long as the packet loss rates remain minimal.
But what would terabit class network connectivity at home look like?
Actually, I think we have all the parts to build this today. With Wavelength Devision Multiplexing (WDM), it’s possible to transmit multiple data streams through a single fiber by using slightly different wavelengths/frequencies of infrared laser light. Coarse WDM (CWDM) is relatively cheap and appropriate over shorter distances, with 18 wavelengths standardized over high performance fiber. (Fewer over most existing fiber.) For long distances, dense WDM (DWDM) can use as many as 160 wavelengths over a single fiber pair.
Bandwidth per wavelength is now 100 or 200 Gbps, and expected to increase in the future. So anything between say 10 x 100 Gbps = 1 Tbps and the 20 Tbps used by modern seacables should be possible. The catch is of course the cost.
The difficulty is with the transmitting side, as this requires a tuned laser per wavelength. On the receiving side, the wavelengths can be split using a prism and hit a set of wideband receivers. As someone who is definitely not in the business of building this equipment, it seems to me that a system with one or a small number of transmitters, a passive optical bus, and a large(r) set of receivers is definitely something that could enjoy radical performance vs cost improvements over time. And it fits perfectly with the most efficient / high speed way to connect homes to the internet that we have today: PON (passive optical network). So just add additional wavelengths to existing PON installations to gain more bandwidth in the downstream direction.
However, now we have a new challenge: TCP/IP is not a good fit for sending the massive data streams that would make good use of such a network. The problem is that TCP tries to adjust its end-to-end data transmission rate to available bandwidth. This means it needs to wait for acknowledgments from the receiving side to know it can increase its transmission rate, maintain it, or slow the transmission rate down. Downloading 100 MB over a 1 Tbps link takes less than a millisecond. But even over PON, the round-trip-time is a millisecond or two. This means that the bottleneck is the number of round trip TCP requires to reach that full terabit speed. Even if that’s an extremely unrealistic 10 RTTs, that means the total transmission time is now 11 ms, effectively only using a tenth of the available bandwidth.
So we need to overhaul TCP/IP for the super high speed stuff and instead use something more like circuit switching / time division multiplexing / token passing. Yes, everything old is new again! So for instance reserve ten 100 μs timeslots and transmit ten 10 MB “megapackets”.
So I think all of this is highly doable!
Well, there is the slight challenge of how to pipe all that bandwidth into your laptop without connecting/disconnecting that fiber all the time. Maybe use eight Thunderbolt 5 interfaces in parallel to reach 960 Gbps?