Introducing Data Transfer Limits to NDT

Consumer Internet speeds increasing to 1Gbps and beyond pose a scaling challenge to servers engaged in measuring global Internet quality, as every bit represents physical infrastructure and energy costs to measure that information. Maintainers of Measurement Lab are investigating options to limit the amount of data transferred by an NDT test that will reduce resource strain while maintaining quality of Internet measurement and sustain user experience.

Motivation

Consumer Internet speeds increasing to 1Gbps and beyond pose a scaling challenge to servers engaged in measuring global Internet quality such as those deployed by M-Lab. For example, an NDT test will transfer 1.25 GB in 10 seconds for clients with 1 Gbps network capacity, which causes a financial strain as every bit represents both information and the physical infrastructure and energy costs to measure that information. Further, mobile speed tests may be constrained by monthly data transfer caps and could benefit from less aggressive speed test methodologies to avoid exhausting finite transfer limits; especially when their goal is to verify the availability of a certain amount of bandwidth vs. measuring the maximum throughput of a connection.

These observations have led maintainers of Measurement Lab to investigate options to limit the amount of data transferred by an NDT test. Specifically, we propose a client-side parameter that specifies the maximum object size to be transferred by the test. This parameter will be optional for NDT integrations. Initially, a download cap of 250MB will be applied to tests from the Internet Speed Test Google search’s NDT integration which comprises 85% of our tests and 80% of data transferred. Below we summarize the analysis that led us to choose the cap of 250 MB. We hope that this analysis will help inform others that may want to apply a cap to their own NDT integrations.

Evaluating download caps

Figure 1: Visualization of a speed test for clients with 200 Mbps, 100 Mbps and 50 Mbps.

Figure 1 illustrates speed test transfers of a 125 MB object for clients with speed of 200 Mbps, 100 Mbps and 50 Mbps, respectively. The 100 Mbps client will transfer the object in exactly 10 seconds whereas the 50 Mbps client will take 20 seconds to transfer the same object. If the test is ended after 10 seconds, as is done by NDT, the 50 Mbps client will have transferred only 62.5 MB during the test. This example gives some intuition about how we may want to choose a maximum data transfer for the NDT test. Specifically, clients with speed that is lower than the speed needed to transfer the object in 10 seconds i.e. the “target rate” are not impacted by the cap because they transfer less than the cap amount in 10 seconds. However, for clients that are faster than the target rate, the test will end early e.g., after 5 seconds for a client with 200 Mbps available bandwidth.

The table below summarizes the maximum object sizes we consider along with their corresponding target rates. The specific set of object sizes was chosen to model a standard set of Internet speed tiers.

Data savings vs. download cap

Figure 2. Percentage data savings vs. download cap. Percentage savings is computed over all tests in Q1 2023.

Figure 2 shows the percent of data transferred by NDT tests that would be saved if all tests were capped at a given value. Data savings start at 91% for a cap of 10 MB per test and rapidly drop off with a savings of only 5% for a 625 MB cap. We note that 100-250 MB seem to provide a decent middle ground with data savings of 25-50% for these caps.

Accuracy vs. download cap

We next study the accuracy of NDT tests with different download caps. For this analysis, we analyze traces of existing NDT tests and emulate the bandwidth estimation process if the test had stopped after a fixed amount of data was transferred. We compare this estimated throughput with the throughput that was returned by the full test to study the accuracy of NDT with different sized download caps.

Figure 3. Percentage error vs. download cap showing the median-95th percentile error for all tests with a given download cap.

Figure 3 shows the accuracy of NDT tests for the different size caps. Overall accuracy is quite good, even for smaller object sizes with the median error near zero with an object size of 100 MB and the 75th percentile reaching zero at 125MB. The 90th and 95th percentile errors take longer to drop off with an object size of 250 MB having a 95th percentile error of 5% across all tests. 2

This result may seem to imply that we can use a very small object size (e.g., 100 MB) while maintaining relatively good accuracy, especially if we focus on the median and 75th percentile error. However, as we noted above, accuracy is dependent on the client speed as well as the object size.

Figure 4. 95th percentile error vs. download cap for different client speeds.

Figure 4 illustrates the relationship between the download cap and 95th percentile error for different client speeds. While the 95th percentile accuracy is fairly good for client speeds up to 250 Mbps with less than 5% error, the error increases with 95th percentile error of 18-30% for client speeds above 250 Mbps.

Figure 5. Distribution of client speeds for NDT tests.

As we’ve seen, our overall accuracy is high even for small object sizes, but the accuracy is very dependent on the client’s speed. Figure 5 shows the breakdown of client speeds for NDT tests. We find that 91% of the clients have speeds of <=250Mbps. From this we can conclude that the overall accuracy of the shorter tests that we observe is related to the NDT client population.

Choosing a cap

Based on our analysis there are a few options for a download cap. We could pick a small value like 100 MB which gives 50% data savings and a median accuracy of close to 0 or we can go with a larger value that gives higher accuracy.

In the end, we have decided on a cap of 250 MB for the Internet speed test. This corresponds to a client speed of 200 Mbps if the object is transferred in exactly 10 seconds. 200 Mbps is also substantially above the currently defined definition for broadband by the FCC of 25 Mbps for downloads and 3 Mbps for uploads. This cap will yield 25% data savings with an overall 95th percentile error of 5%.

Community call

Measurement Lab held a call with community members on June 1st, 2023 to announce these changes to NDT. For more information, the recording can be found here and the slides here.

Anticipating concerns

While Measurement Lab’s intention is to assess the throughput of a connection, we recognize that most contributors running a test are keen to understand whether the Internet service they have paid for is meeting its advertised benefits. We recognize that many companies and households pay for speeds beyond 250 Mbps, reaching upwards of 1Gig and higher, and that shorter NDT Tests may be inadequate for the 9% of contributors seeking to test their bandwidth. We note that the object size limit is strictly opt-in at the client side and higher bandwidth integrations can still leverage an unlimited NDT test. We do stress that the need of the majority of Internet users maxes out well below 250 Mbps and are curious about the perception that more and faster is better, and whether we can improve our understanding of quality Internet over quantity.

Call to action

While we have focused on limiting data transferred by the speed test as one way to allow our tests to scale in the face of higher consumer broadband speeds, there is an opportunity for more research in this space. We are open to collaborating with those exploring novel research for speed tests that are able to measure higher speeds, with high accuracy while conserving resources. Please reach out if you are working toward improving this space!