web100 - NDT (Network Diagnostic Tool)

web100 was the NDT protocol used by M-Lab until November 2019. As a part of M-Lab’s platform upgrade, ndt-server replaced the now deprecated web100-based NDT server. The new server provides the ndt5 protocol for backward compatibility with current clients, and will add the ndt7 protocol for future client use.

NDT data using the web100 protocol was collected using the Web100 Linux kernel patch to provide access to a rich set of TCP information for each NDT test.

web100 NDT data can be found in the NDT web100 dataset.

Web100 Field Types and BigQuery Equivalents

Web100 fields were defined by the original working group in the file tcp-kis.txt. This file defines each Web100 variable with a specific SNMP type. The table below shows how to map each SNMP type to a BigQuery type.

Blacklist Flags Field

The field blacklist_flags was used to mark test affected by the “switch discard issue” identified in 2015-2016. M-Lab NDT data from 2010-01-01 to 2015-10-02 was marked using this field. M-Lab may use the field for other use cases in the future for tests potentially impacted by site configuration issues, or otherwise communicate potentially relevant information about the state of the platform at the time of the test.

Currently, the following values are present in NDT data from 2010-01-01 to 2015-10-02 in these fields:

Querying Historical NDT web100 Data

M-Lab recommends querying NDT web100 data using our “unified” views:

• measurement-lab.ndt.unified_downloads
• measurement-lab.ndt.unified_uploads

These views contain a field called node_instruments, which will contain web100 if collected using the now deprecated web100 ndt server.

However, some researchers may wish to query this data directly. If this is desired, please refer to the schema and subsequent sections below, querying the table:

• measurement-lab.ndt.web100

web100 NDT BigQuery Schema

Calculating Common Metrics for NDT Web100 Data

If you are not using our unified_downloads or unified_uploads views, but instead are queryiing measurement-lab.ndt.web100, commonly reported metrics such as upload or download speed need to be calculated from data fields saved by the web100 NDT protocol.

The most commonly reported metrics are: Download Throughput (Mbit/s), Upload Throughput (bit/s), and Round Trip Time (ms). Recommended queries for these metrics are outlined below, limiting results to only valid tests. For the purposes of research, we consider valid NDT web100 tests to be those that:

• At least 8 KB of data was transferred
• Test duration was between 9 and 60 seconds
• Congestion was detected
• Tests with NULL results excluded
• Tests from M-Lab Operations and Management infrastructure excluded
• Traffic queuing on the switch was not present at the time of the test

Units of Measure, Converting to megabits per second (Mbit/s)

In the formulas and examples below, it is important to note the units of measure that NDT uses for fields that represent data sent and received, as well as for fields representing durations.

• Data sent/recevied fields are a count of bytes
• Duration fields are saved in microseconds

The formula below shows how we convert our final values to Mbit/s when calculating Download or Upload throughput, using 8 as a multiplier in our BigQuery clauses:

1 byte/microsecond * 8 bits/byte =
8 bits/microsecond * (1 / 1,000,000) megabits/bit =
8 / 1,000,000 megabits/microsecond * 1,000,000 microseconds/second =
8 megabits/second => 8 Mbit/s

Download throughput

Download throughput is computed for every server-to-client test as the ratio of the data transmitted during the test and the duration of the test. Results of tests that ended during slow start are excluded.

Download throughput is calculated using this formula within the query:

8 *
(web100_log_entry.snap.HCThruOctetsAcked /
 (web100_log_entry.snap.SndLimTimeRwin +
 web100_log_entry.snap.SndLimTimeCwnd +
 web100_log_entry.snap.SndLimTimeSnd))

The complete BigQuery example is:

#standardSQL
SELECT
  8 * (web100_log_entry.snap.HCThruOctetsAcked /
    (web100_log_entry.snap.SndLimTimeRwin +
    web100_log_entry.snap.SndLimTimeCwnd +
    web100_log_entry.snap.SndLimTimeSnd)) AS download_Mbps
FROM
  `measurement-lab.ndt.web100`
WHERE
  partition_date BETWEEN '2017-01-01' AND '2017-08-28'
  -- Data_direction specifies upload (0) or download (1) test
  AND connection_spec.data_direction = 1
  -- Traffic queuing on the switch was not present at the time of the test
  (blacklist_flags = 0 OR
    (blacklist_flags IS NULL AND anomalies.blacklist_flags IS NULL))
  -- Exclude tests from M-Lab Operations and Management infrastructure
  AND web100_log_entry.connection_spec.local_ip IS NOT NULL
  AND web100_log_entry.connection_spec.remote_ip IS NOT NULL
  AND web100_log_entry.connection_spec.remote_ip NOT IN("45.56.98.222", "35.192.37.249", "35.225.75.192", "2600:3c03::f03c:91ff:fe33:819", "23.228.128.99", "2605:a601:f1ff:fffe::99")
  -- At least 8 KB of data was transferred
  AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
  -- Test duration was between 9 and 60 seconds
  AND (web100_log_entry.snap.SndLimTimeRwin +
    web100_log_entry.snap.SndLimTimeCwnd +
    web100_log_entry.snap.SndLimTimeSnd) >= 9000000
  AND (web100_log_entry.snap.SndLimTimeRwin +
    web100_log_entry.snap.SndLimTimeCwnd +
    web100_log_entry.snap.SndLimTimeSnd) < 60000000
  -- Congestion was detected
  AND web100_log_entry.snap.CongSignals > 0
  -- Sensible TCP end state
  AND web100_log_entry.snap.State IN (1,5,6,7,8,9,10,11)
  LIMIT 100

Upload throughput

Upload throughput is computed for every client-to-server test as the ratio of the data transmitted during the test and the duration of the test.

It is not possible to exclude results of tests that ended during slow start, because the web100 variable web100_log_entry.snap.CongSignals is not updated during client-to-server tests.

Upload throughput is calculated using this formula within the query:

8 * (web100_log_entry.snap.HCThruOctetsReceived/web100_log_entry.snap.Duration) AS upload_Mbps

The complete BigQuery example is similar to the download query above. Substitute the upload calculation for the one used above to calculate download, and change connection_spec.data_direction to 0.

Round Trip Time (RTT)

Server-to-client RTT is affected by TCP congestion. As a consequence, there are (at least) 2 ways to estimate the RTT using web100 data. These 2 ways provide different, non-equivalent information about the user connection.

Server-client (time) distance

• Estimated using the minimum RTT measured during the test, which most likely happened before the test reached congestion.
• This value is reported by the web100 variable web100_log_entry.snap.MinRTT
• However, using this variable has the drawback that it might underestimate the connection RTT, because it might be measured in the SYC ACK exchange or some other tiny transaction which, for low speed links, does not represent the typical RTT for the full data segment.
• Note that using PreCongSumRTT/PreCongCountRTT does not provide a more accurate estimate, because both PreCongSumRTT and PreCongCountRTT are recorded right before the first congestion signal, which, in the worst case, occurs when the receiver queue is already full, which affects the RTT.

Server-client latency during data transfers (with congestion)

• Estimated using the average RTT, uniformly averaged over an entire test.
• This value can be computed as web100_log_entry.snap.SumRTT/web100_log_entry.snap.CountRTT
• In this case, it makes sense to exclude results of tests with 10 or fewer round trip time samples, because there are not enough samples to accurately estimate the RTT. This condition is expressed in BigQuery with:web100_log_entry.snap.CountRTT > 10

Given that the NDT server updates the web100 variables web100_log_entry.snap.MinRTT and web100_log_entry.snap.CountRTT only when it receives an acknowledgement and given that, during client-to-server tests the NDT server receives an ack only during the 3-way-handshake, RTT values are computed only for server-to-client tests in the web100 datatype.

The complete BigQuery example is:

#standardSQL
SELECT
  web100_log_entry.snap.MinRTT AS min_rtt
FROM
  `measurement-lab.ndt.web100`
WHERE
  connection_spec.client_geolocation.country_code = 'US'
  AND partition_date BETWEEN '2017-01-01' AND '2017-01-02'
  AND connection_spec.data_direction = 1
  AND web100_log_entry.snap.HCThruOctetsAcked >= 8192
  AND (web100_log_entry.snap.SndLimTimeRwin +
       web100_log_entry.snap.SndLimTimeCwnd +
       web100_log_entry.snap.SndLimTimeSnd) >= 9000000
  AND (web100_log_entry.snap.SndLimTimeRwin +
       web100_log_entry.snap.SndLimTimeCwnd +
       web100_log_entry.snap.SndLimTimeSnd) < 600000000
  AND web100_log_entry.snap.CountRTT > 10
  AND (web100_log_entry.snap.State = 1
       OR (web100_log_entry.snap.State >= 5
       AND web100_log_entry.snap.State <= 11))
  LIMIT 100