SideStream
SideStream collected statistics about the TCP connections used by the measurement tools running on the M-Lab platform when it used the web100 kernel extension. SideStream ran as a “side” addition to other tools, taking advantage of TCP connections generated by incoming tests to M-Lab.
Source code
Citing the M-Lab SideStream Dataset
Please cite this data set as follows: The M-Lab SideStream Data Set, <date range used>. https://measurementlab.net/tests/sidestream
SideStream Data in Raw Format
Data collected by SideStream is available in raw format in Google Cloud Storage: https://console.developers.google.com/storage/browser/archive-measurement-lab/sidestream/.
SideStream Data in BigQuery
M-Lab parsed all SideStream data into BigQuery tables and views, and makes query access available for free by subscription to a Google Group. Find out more about how to get access on our BigQuery QuickStart page.
BigQuery Tables/Views for SideStream Data:
Sidestream Schema
The table below describes the schema for the table: measurement-lab.sidestream.web100.
| Field name | Type | Description |
|---|---|---|
partition_date | date | The day (in UTC) in which the data was loaded. |
test_id | string | ID of the test. It represents the filename of the log that contains the data generated during the test (e.g. 20090819T02:01:04.507508000Z_189.6.232.77:3859.c2s_snaplog.gz). |
task_filename | string | The raw data file in Google Cloud Storage from which the test row was parsed. |
parse_time | timestamp | The timestamp of when test data was parsed into BigQuery from Google Cloud Storage. |
parser_version | string | A link to the tagged version of the M-Lab ETL parser which processed the test row. |
project | integer | |
log_time | timestamp | |
type | integer | |
anomalies | record | |
anomalies.exclusion_level | integer | |
web100_log_entry | record | |
web100_log_entry.log_time | integer | |
web100_log_entry.version | string | |
web100_log_entry.group_name | string | |
web100_log_entry.connection_spec.local_af | integer | IPv4 = 0IPv6 = 1 |
web100_log_entry.connection_spec.local_ip | string | |
web100_log_entry.connection_spec.local_port | integer | |
web100_log_entry.connection_spec.remote_ip | string | |
web100_log_entry.connection_spec.remote_port | integer | |
web100_log_entry.connection_spec.local_geolocation | record | |
web100_log_entry.connection_spec.local_geolocation.area_code | integer | |
web100_log_entry.connection_spec.local_geolocation.city | string | |
web100_log_entry.connection_spec.local_geolocation.continent_code | string | |
web100_log_entry.connection_spec.local_geolocation.country_code | string | |
web100_log_entry.connection_spec.local_geolocation.country_code3 | string | |
web100_log_entry.connection_spec.local_geolocation.country_name | string | |
web100_log_entry.connection_spec.local_geolocation.latitude | float | |
web100_log_entry.connection_spec.local_geolocation.longitude | float | |
web100_log_entry.connection_spec.local_geolocation.metro_code | integer | |
web100_log_entry.connection_spec.local_geolocation.postal_code | string | |
web100_log_entry.connection_spec.local_geolocation.region | string | |
web100_log_entry.connection_spec.local_geolocation.radius | integer | |
web100_log_entry.connection_spec.remote_geolocation.area_code | integer | |
web100_log_entry.connection_spec.remote_geolocation | record | |
web100_log_entry.connection_spec.remote_geolocation.area_code | integer | |
web100_log_entry.connection_spec.remote_geolocation.city | string | |
web100_log_entry.connection_spec.remote_geolocation.continent_code | string | |
web100_log_entry.connection_spec.remote_geolocation.country_code | string | |
web100_log_entry.connection_spec.remote_geolocation.country_code3 | string | |
web100_log_entry.connection_spec.remote_geolocation.country_name | string | |
web100_log_entry.connection_spec.remote_geolocation.latitude | float | |
web100_log_entry.connection_spec.remote_geolocation.longitude | float | |
web100_log_entry.connection_spec.remote_geolocation.metro_code | integer | |
web100_log_entry.connection_spec.remote_geolocation.postal_code | string | |
web100_log_entry.connection_spec.remote_geolocation.region | string | |
web100_log_entry.connection_spec.remote_geolocation.radius | integer | |
web100_log_entry.snap | record | |
web100_log_entry.snap.AbruptTimeouts | integer | |
web100_log_entry.snap.ActiveOpen | integer | |
web100_log_entry.snap.CERcvd | integer | |
web100_log_entry.snap.CongAvoid | integer | |
web100_log_entry.snap.CongOverCount | integer | |
web100_log_entry.snap.CongSignals | integer | |
web100_log_entry.snap.CountRTT | integer | |
web100_log_entry.snap.CurAppRQueue | integer | |
web100_log_entry.snap.CurAppWQueue | integer | |
web100_log_entry.snap.CurCwnd | integer | |
web100_log_entry.snap.CurMSS | integer | |
web100_log_entry.snap.CurRTO | integer | |
web100_log_entry.snap.CurReasmQueue | integer | |
web100_log_entry.snap.CurRetxQueue | integer | |
web100_log_entry.snap.CurRwinRcvd | integer | |
web100_log_entry.snap.CurRwinSent | integer | |
web100_log_entry.snap.CurSsthresh | integer | |
web100_log_entry.snap.CurTimeoutCount | integer | |
web100_log_entry.snap.DSACKDups | integer | |
web100_log_entry.snap.DataOctetsIn | integer | |
web100_log_entry.snap.DataOctetsOut | integer | |
web100_log_entry.snap.DataSegsIn | integer | |
web100_log_entry.snap.DataSegsOut | integer | |
web100_log_entry.snap.DupAckEpisodes | integer | |
web100_log_entry.snap.DupAcksIn | integer | |
web100_log_entry.snap.DupAcksOut | integer | |
web100_log_entry.snap.Duration | integer | |
web100_log_entry.snap.ECESent | integer | |
web100_log_entry.snap.ECN | integer | |
web100_log_entry.snap.ECNNonceRcvd | integer | |
web100_log_entry.snap.ECNsignals | integer | |
web100_log_entry.snap.ElapsedMicroSecs | integer | |
web100_log_entry.snap.ElapsedSecs | integer | |
web100_log_entry.snap.FastRetran | integer | |
web100_log_entry.snap.HCDataOctetsIn | integer | |
web100_log_entry.snap.HCDataOctetsOut | integer | |
web100_log_entry.snap.HCSumRTT | integer | |
web100_log_entry.snap.HCThruOctetsAcked | integer | |
web100_log_entry.snap.HCThruOctetsReceived | integer | |
web100_log_entry.snap.InRecovery | integer | |
web100_log_entry.snap.IpTosIn | integer | |
web100_log_entry.snap.IpTosOut | integer | |
web100_log_entry.snap.IpTtl | integer | |
web100_log_entry.snap.LimCwnd | integer | |
web100_log_entry.snap.LimMSS | integer | |
web100_log_entry.snap.LimRwin | integer | |
web100_log_entry.snap.LimSsthresh | integer | |
web100_log_entry.snap.LocalAddress | string | |
web100_log_entry.snap.LocalAddressType | integer | |
web100_log_entry.snap.LocalPort | integer | |
web100_log_entry.snap.MSSRcvd | integer | |
web100_log_entry.snap.MSSSent | integer | |
web100_log_entry.snap.MaxAppRQueue | integer | |
web100_log_entry.snap.MaxAppWQueue | integer | |
web100_log_entry.snap.MaxCaCwnd | integer | |
web100_log_entry.snap.MaxMSS | integer | |
web100_log_entry.snap.MaxPipeSize | integer | |
web100_log_entry.snap.MaxRTO | integer | |
web100_log_entry.snap.MaxRTT | integer | |
web100_log_entry.snap.MaxReasmQueue | integer | |
web100_log_entry.snap.MaxRetxQueue | integer | |
web100_log_entry.snap.MaxRwinRcvd | integer | |
web100_log_entry.snap.MaxRwinSent | integer | |
web100_log_entry.snap.MaxSsCwnd | integer | |
web100_log_entry.snap.MaxSsthresh | integer | |
web100_log_entry.snap.MinMSS | integer | |
web100_log_entry.snap.MinRTO | integer | |
web100_log_entry.snap.MinRTT | integer | |
web100_log_entry.snap.MinRwinRcvd | integer | |
web100_log_entry.snap.MinRwinSent | integer | |
web100_log_entry.snap.MinSsthresh | integer | |
web100_log_entry.snap.Nagle | integer | |
web100_log_entry.snap.NonRecovDA | integer | |
web100_log_entry.snap.NonRecovDAEpisodes | integer | |
web100_log_entry.snap.OctetsRetrans | integer | |
web100_log_entry.snap.OtherReductions | integer | |
web100_log_entry.snap.PipeSize | integer | |
web100_log_entry.snap.PostCongCountRTT | integer | |
web100_log_entry.snap.PostCongSumRTT | integer | |
web100_log_entry.snap.PreCongSumCwnd | integer | |
web100_log_entry.snap.PreCongSumRTT | integer | |
web100_log_entry.snap.QuenchRcvd | integer | |
web100_log_entry.snap.RTTVar | integer | |
web100_log_entry.snap.RcvNxt | integer | |
web100_log_entry.snap.RcvRTT | integer | |
web100_log_entry.snap.RcvWindScale | integer | |
web100_log_entry.snap.RecInitial | integer | |
web100_log_entry.snap.RemAddress | integer | |
web100_log_entry.snap.RemPort | integer | |
web100_log_entry.snap.RetranThresh | integer | |
web100_log_entry.snap.SACK | integer | |
web100_log_entry.snap.SACKBlocksRcvd | integer | |
web100_log_entry.snap.SACKsRcvd | integer | |
web100_log_entry.snap.SampleRTT | integer | |
web100_log_entry.snap.SegsIn | integer | |
web100_log_entry.snap.SegsOut | integer | |
web100_log_entry.snap.SegsRetrans | integer | |
web100_log_entry.snap.SendStall | integer | |
web100_log_entry.snap.SlowStart | integer | |
web100_log_entry.snap.SmoothedRTT | integer | |
web100_log_entry.snap.SndInitial | integer | |
web100_log_entry.snap.SndLimBytesCwnd | integer | |
web100_log_entry.snap.SndLimBytesRwin | integer | |
web100_log_entry.snap.SndLimBytesSender | integer | |
web100_log_entry.snap.SndLimTimeCwnd | integer | |
web100_log_entry.snap.SndLimTimeRwin | integer | |
web100_log_entry.snap.SndLimTimeSnd | integer | |
web100_log_entry.snap.SndLimTransCwnd | integer | |
web100_log_entry.snap.SndLimTransRwin | integer | |
web100_log_entry.snap.SndLimTransSnd | integer | |
web100_log_entry.snap.SndMax | integer | |
web100_log_entry.snap.SndNxt | integer | |
web100_log_entry.snap.SndUna | integer | |
web100_log_entry.snap.SndWindScale | integer | |
web100_log_entry.snap.SoftErrorReason | integer | |
web100_log_entry.snap.SoftErrors | integer | |
web100_log_entry.snap.SpuriousFrDetected | integer | |
web100_log_entry.snap.SpuriousRtoDetected | integer | |
web100_log_entry.snap.StartTimeStamp | integer | |
web100_log_entry.snap.State | integer | |
web100_log_entry.snap.SubsequentTimeouts | integer | |
web100_log_entry.snap.SumOctetsReordered | integer | |
web100_log_entry.snap.SumRTT | integer | |
web100_log_entry.snap.ThruOctetsAcked | integer | |
web100_log_entry.snap.ThruOctetsReceived | integer | |
web100_log_entry.snap.TimeStampRcvd | integer | |
web100_log_entry.snap.TimeStampSent | integer | |
web100_log_entry.snap.TimeStamps | integer | |
web100_log_entry.snap.Timeouts | integer | |
web100_log_entry.snap.WAD_CwndAdjust | integer | |
web100_log_entry.snap.WAD_IFQ | integer | |
web100_log_entry.snap.WAD_MaxBurst | integer | |
web100_log_entry.snap.WAD_MaxSsthresh | integer | |
web100_log_entry.snap.WAD_NoAI | integer | |
web100_log_entry.snap.WillSendSACK | integer | |
web100_log_entry.snap.WillUseSACK | integer | |
web100_log_entry.snap.WinScaleRcvd | integer | |
web100_log_entry.snap.WinScaleSent | integer | |
web100_log_entry.snap.X_OtherReductionsCM | integer | |
web100_log_entry.snap.X_OtherReductionsCV | integer | |
web100_log_entry.snap.X_Rcvbuf | integer | |
web100_log_entry.snap.X_Sndbuf | integer | |
web100_log_entry.snap.X_dbg1 | integer | |
web100_log_entry.snap.X_dbg2 | integer | |
web100_log_entry.snap.X_dbg3 | integer | |
web100_log_entry.snap.X_dbg4 | integer | |
web100_log_entry.snap.X_rcv_ssthresh | integer | |
web100_log_entry.snap.X_wnd_clamp | integer | |
web100_log_entry.snap.ZeroRwinRcvd | integer | |
web100_log_entry.snap.ZeroRwinSent | integer |