The mostly negative NAP experience led to calls to “disintermediate” IXPs. In technical terms, that meant keeping TCP/IP layer 2 operations fully separate from layers 3 and 4. In governance terms, the rallying call was ensuring IXP operations were not only open and transparent but also proactively prevented potential conflicts of interest among providers and participants. The models developed by non-profit IXPs in the US and Europe, mentioned in the previous post, provided a comprehensive blueprint. And commercial models such as those in Amsterdam and Frankfurt offered a for-profit version.
At the same time, large telco providers, both old and new, learned a key lesson. They could connect their networks directly without going through the local or regional IXP. Let us not forget that IXPs are usually located in colocation data centers, which provide them with the full IP stack. Large providers can operate in the same data centers. So running a wire between two or more of them, bypassing the IXP, is not a major undertaking. And that is, in fact, what occurred: a practice that emerged from the NAP experiment at a time when competition among telcos and IP providers was rapidly intensifying.
A process of concentration and centralization followed in the sector. Early providers such as UUNET (AlterNet), PSINet, ANS and MCI, among others, quickly joined the ranks of Tier 1 providers, an exclusive club of about 15 members who used settlement-free peering across their networks. Members also included traditional telcos such as Deutsche Telekom, Orange, Telecom Italia, NTT (Japan), Telstra (Australia), Tata Communications (India) and Singtel (Singapore). Many of the latter were public telco monopolies privatized in the 1990s. UUNET and company no longer exist, as they were either merged, acquired by larger providers or killed by sheer competition. In any event, many of these private providers also jumped into the submarine fiber-optic business to protect their market shares and consolidate dominance. That was certainly the case until 2015 or so, when Big Tech started laying its own international fiber-optic network, bypassing traditional operators.
In principle, setting up a local IXP does not require investing millions of dollars, unlike the endlessly growing data centers always hungry for massive amounts of data. For starters, IXPs do not provide IP routing, nor are they intended to connect clients to the Internet. For those reasons, they must locate where third parties are already implementing those functions. IXPs, therefore, depend heavily on data centers, big and small. From an ISP’s perspective, the IXP offers an Ethernet switch port where it can connect to other ISPs connected to that same switch. This provides a one-to-many connection, reducing networking costs; otherwise, the ISP would have to deploy several one-to-one connections to every other ISP. From the IXP’s perspective, it offers many-to-one and many-to-many connections, providing economies of scale for potential clients.
In the early days of the Internet, when developing countries connected via developed-country networks, setting up a local IXP could have been challenging and extremely costly. First, traffic was still negligible while ISPs were mostly missing in action. Obviously, data centers had yet to materialize almost everywhere. Consequently, the best bet for an IXP was to find a relatively well-provisioned academic computing center. Recall that in those days, academic institutions in the Global South led the Internet charge. Connecting other local universities and educational entities in this way was feasible. An IXP at the main academic computer center could then serve them, keeping local traffic local and preventing it from traveling around the world before reaching its destination. Of course, once they emerged, local ISPs could add the same functionality.
A basic IXP instance in a developing country can nowadays be set up as follows. First, install two second-hand 24-port Ethernet switches in a local colocation data center. Also deploy a Linux server with the BIRD internet routing daemon properly configured. Obtain an IP subnet with 256 addresses (254 of which can be used for IXP business). Ensure the IXP is listed on PeeringDB and enlist at least one engineer or technical expert who can run and manage the overall operation. That certainly sounds like a relatively affordable investment in countries where both financial and human resources might be scarce. It also sounds like a very low-profit or mostly non-profit type of endeavor.
However, that alone will not make the operation sustainable. For that, a clear governance mechanism should be formally in place so that those joining understand how the IXP is intended to operate. Demand for IXP services should exist to ensure clients are eager to join and pay connection fees. Furthermore, adequate technical capacity should be in place, with detailed succession plans if needed engineers leave. If a local telco owns the data center, ensure the provider guarantees access to fiber and doesn’t limit it. The latter is not necessarily a financial or technical issue. It is essentially a political economy issue, so it requires proper regulation at the local level, which more often than not does not occur.
What about Big Tech? Could small IXPs work with the behemoths? The answer is yes, with some qualifications. First, companies such as Google have essentially created their own IXPs, which are called Private Peering Exchanges or Private Interconnection Fabrics. While Big Tech might use IXPs in developing countries, once those markets grow and become massive, it bypasses local IXPs and builds its own private exchanges. IXPs’ low port speeds, connection costs, profitability, and congestion and latency can all trigger this process. In some developing countries, Big Tech already owns large data centers along with the fiber and the internal networking architecture. Suppose a Tier 1 ISP wants to connect to, say, Google. In that case, they will deploy a fiber cable directly to the Google data center, thus bypassing the local IXP. Now, Google is the local IXP of sorts, albeit entirely private and with no links whatsoever to the local IXPs.
In some other countries where markets are smaller, Big Tech uses a different strategy, since they don’t own a local data center and launching one is not feasible for profitability reasons. In such cases, Google first connects to the local IXP. It then assesses which ISPs control most of the traffic. They then rent space in a neutral data center owned by, say, Equinix, Digital Realty or even a local data center provider. It then approaches the dominant ISP and suggests running a fiber cable to its switches in the rented data center space. This creates a private peering exchange, while Google continues to serve the other non-dominant ISP via the local IXP. That is partly why local IXPs should have a Big Tech strategy to ensure their own sustainability. Here, Big Tech is poaching clients by offering better overall connectivity and networking, both of which are hard for cash-strapped IXPs in developing countries to scale.
Raul
