The OVH Strasbourg fire : Customs Bridge's adaptability

Over the night between Tuesday 9th and Wednesday 10th of march 2021, OVH Strasbourg’s SGB2 data center caught fire. 3,6 millions websites ended up offline, and among them, some of Customs Bridge’s servers. Fortunately, thanks to our IT department’s skills and reactivity, we were ready to quickly switch to an other solution. Indeed, since early march, our IT department has been working on setting up a new software engineering approach called Continuous Integration/Continuous Deployment (CI/CD). This practice, providing the means to frequently and automatically update software, has enabled our IT team to restore our services before noon.

Continuous Integration is a software engineering practice that aims to prevent conflicts between developers working on the same software. It aims to ensure that those updates are releasable, and if so, to deploy them quickly. Writing an automated test suite as well as configuring the buildautomation is a serious workload. But an effort that had to be made, given how much Customs Bridge has grown. At the same time, we must keep on frequently delivering new features and upgrades, while keeping a high quality of service. This new system has undergone it’s baptism by fire during the morning on the 10th of march. After evaluating the consequences of that incident, the services that went offline were automaticallyredeployed on servers in Gravelines. After a simple change of URL, we were back online only 2 hours after we took notice of the incident. Therefore, unless a nuclear accident happen, we hope that OVH will not have any trouble with it’s Gravelines Datacenter … we knock on wood … hopefully it’s fireproof !

This kind of event reminds us that it falls under everyone’s responsability to overcome problems and to organize plans to constinuousely keep our services online. Anticipating repetitive backups and services in different places, and if possible, with different service providers, brings a much greater resilience when facing such events. It’s impossible to foresee every possibility, but building our infrastructure while keeping in mind the various risks enables us to gain in reactivity. This is what we strive to do at Customs Bridge, and with this incident, we demonstrated our responsiveness.