M-Lab 2.0 Platform Migration Update
Last year, we outlined our plans to Modernize the M-Lab Platform. This year, we’re bringing them to life. Here’s a summary of why the platform update is so valuable and what you can expect throughout the year.
Advantages
Developing new tools for the modern M-Lab platform is easier than ever. If the tool runs in a Docker container (with or without --net=host), then it can run on M-Lab. GitHub repos connected to Dockerhub.com build fresh containers on new tags. And, with approval from M-Lab staff, kubernetes will safely automate the rollout.
Faster, automated deployments tightly couple the measurement tool source code to the binary docker image, to the version running in production, to the data collected, and to the rows in BigQuery. The only way to get code to run on the platform is to build from an open source repository, ensuring that all code on the M-Lab platform is open source. We can also archive every single binary image that has been run on the platform, aiding in the reproduction and verification of past results. This new level of openness, transparency, and reproducibility is, to our knowledge, without precedent in the Internet measurement space.
Longitudinal Analysis
From 2009-2019, M-Lab NDT and Sidestream measurements have used the web100 data type. From 2019 forward, the modern platform will collect the tcp-info data type for all TCP connections.
We will preserve the ability to perform longitudinal analysis on our 10+ years of network measurements. During the beginning of our production deployment milestone, we will collect and compare measurements from the legacy and new platforms. For slower clients, their performance should not change. We expect a large population of clients to have the same behavior on the old and new platform. For the subset of clients that perform better on the new platform, we will provide a way of calibrating past data to be comparable to the new.
Upcoming Milestones
Canary deployment begins (in progress)
Right now, we’re bringing together the last 9 months of work starting with a quarter of the physical platform. Historically, a fourth of the platform served as a “spare” machine at every site (~120 machines). Starting with the Kubernetes platform, a small fraction of clients that opt-in to using the M-Lab location service will be directed to these “canary” machines. With a small fraction of real traffic, we will validate that everything works as intended end to end.
Production deployment begins (target end of Q2)
Once the canary deployment is stable and represents most of the functionality we need, we will begin deployments to a third of the production platform (~120 machines). At this point, we will be running a “hybrid platform” consisting of both legacy and modern software platform. During this time, we will collect A/B data for cross validation. As well, all experiments that are ready to deploy to the new platform can be deployed as well.
Production deployment completes (target mid-Q3)
Once all operational challenges are resolved, all A/B data collected, and all measurement tools that we will continue supporting on the new platform have containerized versions available, we will complete the update to the Kubernetes-based platform.