We’ve recently completed an NTP server refresh and a management and OSS upgrade, two pieces of the infrastructure our network and exchanges quietly depend on. Below, we look at what went into each and why they matter.
Time Server Hardware Refresh
NTP (Network Time Protocol) is how devices across a network keep their clocks in sync, distributing time from a small number of accurate sources. Those sources are time servers, and we run our own rather than relying on someone else’s – with a lot of key mechanisms all depending on it such as: log correlation during incidents, certificate validity, scheduled jobs and much more.
We have operated an NTP service since 1999, and the current hardware that has been operational since 2007 is now reaching end of support, which prompted this refresh. As a result we sought an ideal replacement and worked with the same engineer that supplied the original solution, Meinberg.
When first launched our servers were atomic clocks, however our system now utilises a highly accurate Oven Controlled Crystal Oscillator (OCXO) that is extremely power efficient and uses about a tenth of power an atomic clock would. We operate three NTP servers, located in two different data centres. Two of these track the German time signal from Mainflingen (DCF77) and the other tracks the UK time signal from Anthorn in Cumbria (MSF) – GPS satellite time also runs on all three servers, as this approach helps protect against outages and spoofed signals.
Our NTP servers are also available to members if they want an additional time source.
Upgrading our Management and Operational Support System (OSS)
Our management and OSS runs on Juniper kit and that enables us to reach our entire estate, covering all LINX infrastructure, our exchanges and our managed exchanges.
As with any software and hardware, it naturally reaches end of life, and as this was approaching for us we’ve spent the past year upgrading that kit. This involved software upgrades across our internal, management and out-of-bound devices, alongside hardware upgrades at remote sites such as Wales and Manchester, as well as London.
This now means we have an additional two years before any mass upgrades are needed again, but equally important, it gave us a chance to test our internal networks pre and post maintenance validation checks which came about and were strengthened because of this work.
We pride ourselves on automation and get it in wherever we can, those validation checks run on a series of Python scripts driven by NAPALM, though the upgrades themselves we handle manually. When we come to do this again, we’d look to introduce more pre-maintenance testing alongside additional hardware in our lab.
In total, across a year we upgraded the following kit:
- London – 22 external and 4 internal (the four internal devices sit on the OSS network for our core infrastructure)
- Manchester – 14 external facing
- Scotland – 4 external and 2 internal
- Wales – 2 external
- North Virginia – 7
- Nairobi – 4
- Mombasa – 2
- Accra – 3
- KSA – 4