Utilization Dataset
Since June 2016, M-Lab has collected high resolution switch telemetry for each M-Lab server and site uplink and published it as the utilization dataset, with one datatype table: switch.
Utilization Data in Raw Format
Utilization data in raw format can be found in Google Cloud Storage: https://console.developers.google.com/storage/browser/archive-measurement-lab/utilization/.
Utilization Data in BigQuery
M-Lab parses all switch utilization data into BigQuery, and makes query access available for free by subscription to a Google Group. Find out more about how to get access on our BigQuery QuickStart page.
BigQuery Tables/Views for Switch Data:
Switch BigQuery Schema
| Field name | Type | Description |
|---|---|---|
partition_date | date | The day (in UTC) in which the data was loaded. |
test_id | string | ID of the test. It represents the Google Cloud Storage path, M-Lab server, and file name of log that contains the data generated during the test (e.g. 2019/02/09/mlab1.yul02.measurement-lab.org/20190209T23:00:00-to-20190210T00:00:00-switch.json.gz). |
task_filename | string | The raw data file in Google Cloud Storage from which the test row was parsed. |
parse_time | timestamp | Timestamp of when test data was parsed into BigQuery from Google Cloud Storage. |
parser_version | string | A link to the tagged version of the M-Lab ETL parser which processed the test row. |
log_time | timestamp | Never set for the switch data. The sample.timestamp should be used instead for the sample collection time. |
sample | record | A repeated record with the value and timestamp of each 10 second observation. Typically, there will be 360 samples per hour. Due to system maintenance, or machine restarts, some intervals may contain more or less samples. |
sample.timestamp | timestamp | Timestamp of the beginning of the 10 second time bin. |
sample.value | float | Delta value of the metric during this 10 second time bin. |
metric | string | The canonical metric name for samples, e.g. switch.discards.uplink.tx |
hostname | string | The fully qualified domain name of the machine that collected the data, e.g. mlab2.abc01.measurement-lab.org. |
experiment | string | The fully qualified domain name of the switch that produced the data, e.g. s1.abc01.measurement-lab.org. |
Source Code
The switch dataset is produced by the M-Lab Collectd monitoring tool, which can be found on Github: https://github.com/m-lab/collectd-mlab.