Last week, the globe experienced a world without Facebook. Some 2.89 Billion users were unable to access their ccounts, and 10 million advertisers were similarly locked out. Other companies under the Facebook umbrella (Instagram, WhatsApp, and Oculus) were also down. This left engineers scrambling around data centers in California, trying to get to the bottom of just what went wrong. They were able to get things back up and running about 6:00PM
What made matters worse is that Facebook employees couldn’t access their own accounts to communicate within the company. Because they access basic tools like Google Docs and Zoom through their Facebook accounts, many employees had to resort to using apps outside of the network, just to communicate. Furthermore, employees were locked out of buildings and unable to access physical sites, adding more garbage to the quagmire.
Facebook’s VP of Infrastructure spoke on the subject:
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Janardhan wrote. “This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”
BGP: A Primer (Very Simplified)
Border Gateway Protocol, or BGP, is a kind of Waze map of the internet. Visualize the internet like a giant map. The internet service providers and large organizations (corporations, governments and universities) are ‘cities’. There are roads between them that allow data packets to move so that you can arrive at different sites and information. The BGP is the guide that helps to find the path of least resistance to get between these ‘cities’ (known as autonomous systems).
During a routine BGP update, Facebook made an error somewhere. This lead to it’s own internal autonomous mapping system to break down. As a result, the road maps between servers within Facebook had no guides for getting information around. Think of a bunch of people stuck in traffic with no GPS and no destination. That’s what happened allegorically.
Some are calling this the digital apocalypse, which can be a good reminder to check back in to the real world. The Facebook shutdown reminds us that we are not dependent on an app for life. Regardless of the status of the internet, the world keeps spinning, the birds keep singing, and we keep on dancing.
