π
2014-01-06 01:01
in Linux
LCA 2014 was the last event I gave my ProdNG talk at, the summary of 3.5 years of work at Google and the craziest upgrade I've done in my life. I also wrote a more detailled ProdNG LISA/Usenix Paper if you'd like more details.
ProdNG.odp
ProdNG.pdf
html (full screen)
More importantly, you can get the ProdNG LISA Paper which has many more details on how the upgrade went and the implementation.
>
This talk will look at how we upgraded our ancient linux distribution on all the Google production servers to a more modern one based on debian stripped down and built from source.The video:
We do live upgrades on running machines using an rsync-like rollout mechanism, and we slowly switched distributions on the fly without rebooting or downtime unless required by separately upgraded kernel, but doing a 10 year jump of linux userspace without breaking users was no piece of cake.
The talk will explain how something so crazy made sense, was even possible, executed, and worked out, as well as recommendations for maintaining linux big servers and dealing with upstream upgrades.
>
See more images for My Live Upgrading Many Thousands of Servers ProdNG talk at Linux.conf.au 2014