A resilient IP-based data communications network is helping Electricity North West in the United Kingdom to improve the service provided by its electricity distribution network. Electricity North West’s Paul Gornall, Network and Field Operations Manager, and Lee Wakefield, Telecoms Transmission Engineer, explain how advanced networking technologies from Westermo enabled on time installation and reduce operational costs.

Reliable data communications are essential to support improved quality of service for Electricity North West’s 2.4 million customers. To implement new services such as NMS (Network Management Service) and CLASS (Customer Load Active System Services) and better support existing operations, the company installed a resilient IP-based data communications network. Over the last two years this has supported the company’s goals of improving reliability of supply, reduce the load and create operational savings that can be passed on to customers.

The IP-based data network connects well over 500 primary substations across a vast area of the North West, enabling monitoring and controlling of equipment which reroutes power should problems arise. Previously, a VF Serial-based network was in place which used over 11,000km of ageing copper cabling and offered limited bandwidth, no redundancy and insufficient resilience. A lack of network reliability created high operational costs and resulted in a fault occurring almost every day or night. A fault on the communications network would typically require two people for each call-out and would take at least three hours to rectify, depending upon the type of fault and its location. In addition, resolving issues at night could cause planned works the following day to be postponed or cancelled. Such planned works would normally be capital projects supporting the Quality of Service programmes, and when these are not completed as quickly as they should be, it impacts on our customers.

Solution

Replacing the copper cabling with fibre optic infrastructure was unfeasible in terms of budget and timescale. This led us to Westermo and its Wolverine Ethernet Line Extenders. These devices are based on SHDSL communications and allow Ethernet networks to be created over long distances using existing cabling. They also enable data rates of up to 15.3 Mbit/s to give us the required bandwidth. As we needed to connect over 500 primary substations spread across a vast area, we needed to be very efficient in terms of installing the new data communications technology and configuring and testing the network. It was essential to use in-house resources and the telecoms team had to be very efficient. We also needed to be very smart with the use of technology. The use of Westermo’s Wolverine Ethernet Line Extenders played a major role in the success of the project. Eliminating the need to replace the existing copper cable network made the project feasible by significantly reducing costs. The use of Westermo’s WeConfig network configuration tool was also a key factor, helping the telecoms team to configure, test and implement network security. Having installed the network, the technology has continued to support efficiency savings and plays a vital role in keeping the network secure.

Installation 

With both time and engineering resources limited, the telecoms team needed to be very efficient when installing the IP network. With almost 1000 devices to be installed at primary substations, it was vital to limit the number of times we visited each site, and the use of WeConfig supported this aim. In general, one visit was required to physically install the Wolverine devices and perform configuration, and another to perform testing. However, if parts of the network were not yet in place, that would have required us to make multiple visits. If there were network errors or perhaps a mistake made configuring the device, that would necessitate further visits and more testing. During testing we would need to ensure that the Wolverine devices all had the same software level, configure the unit, harden the device, and then test the connection to another site and perform a further test at the other end of the connection. By performing the configuration, diagnostics and testing tasks using WeConfig at a central location, this removed the need for additional site visits. As a result, we estimate that in excess of 2000 engineering hours (300 work days) were saved. This efficiency improvement was critical in enabling us to complete the project on time and to budget.

Results

The combination of implementing a reliable IP network and the ability to identify cable issues has reduced the time spent on call-outs by approximately 2100 working hours a year. This has not only reduced operating costs, but also enables us to maximise the use of our engineers’ time, allowing them to focus on capital investment improvement projects that contribute towards better quality of service for our customers.

Network reliability

It is important to have visibility of the network. The SolarWinds network management tool is used to monitor the network. When the software identifies an issue, it raises an alert. We then use WeConfig to diagnose the cause of that alert in order to resolve it more efficiently. The diagnostic functionality of WeConfig is very powerful and provides us with detail about the problem. We can immediately determine whether the connection is up and then the likely location of the fault. We can quickly see the quality of the link. We can also run diagnostics to understand the signal to noise ratio over a connection. We use this information to help guide the engineer when it comes to rectifying the issue.

Previously we have not been able to monitor the performance of network connections. Without the monitoring function, faults and loss of communications due to the cabling would have just occurred unexpectedly. This would require a site visit to determine the cause and try to rectify/repair the communications connection. The Westermo technology has provided us with that capability and we have a very good indication of the performance of the connections to all the primary substations over time. The Wolverine devices enable us to monitor the copper cabling, and data is collected and made available in the WeConfig software. We can identify potential issues at an earlier stage. That enables us to better schedule cable maintenance to prevent unexpected network downtime. Better planning reduces the cost of doing the work and enables better use of our internal resources.

Using the performance data, we identify areas that are problematic and make a business case for investment in new cabling. We use this information within strategic planning related to HV overlays. If there are significant issues with a communications link and an HV cable connecting the same two points is to be overlaid in the future, that work may be brought forward to enable a new communications cable to be installed sooner. The result has been an improved communications network, which in turn helps to improve the overall power delivery service.

Using the Westermo Wolverine devices and WeConfig software we have been able to test the copper cabling to see if it is suitable to be used in the first place. We have also used the tool to monitor the impact of other devices sharing the copper infrastructure. This has allowed us to make fundamental changes in the way we share the copper cables and to gain the maximum benefit from the network.

Network security

Network security is extremely important to Electricity North West. Although this network is `closed’ and not accessible externally, we are conscious of potential threats coming from within the organisation. Cybersecurity is about having the right processes in place and then performing them consistently. Human error can undermine this philosophy and the larger the network, the greater the possibility of an error or something being missed.

One vulnerability is unused ports on networking devices being left open. This practice can allow an unauthorised contractor to access the network and inadvertently expose it to a malicious threat. To manually perform the steps needed to block open ports on all devices would be challenging and extremely time-consuming for a large network. It also increases the possibility of an error or step being missed. The first device may be perfectly secure, but the tenth or hundredth device may not.

We use the Westermo WeConfig tool to simplify this task, ensuring a uniform approach, with consistent steps for every Wolverine device that can be performed in one go. Security is hardened by `locking down’ the configuration settings, changing the factory passwords and blocking all unused ports. Critically, using the tool we remove the possibility of human error and can be assured that the task has been performed correctly for every single device. It would have taken approximately half an hour per device to perform the steps manually. With nearly 1000 installed devices, that means we saved around 500 working hours. Without WeConfig it may not have been feasible to have performed this task, thus without question it has enhanced our network security.

Having the ability to automatically secure all the devices very quickly and easily provides great confidence. Having implemented steps to enhance network security, it is essential to maintain security levels. When new devices are added to the network or we access devices to change the configuration or perform a firmware upgrade, we use WeConfig to identify if all the security steps have been performed. This is a very simple procedure, which encourages us to use it on a regular basis. That further enhances network security.

Firmware upgrades

Maintaining network security is an ongoing process. We have performed several firmware upgrades for the networking devices since we began installing them. These have fixed vulnerabilities and provided greater functionality, such as user authentication, that further increases network security. All devices now run a recent version of the WeOS operating system, which provides the functionality we require. We may consider an upgrade in the future, but currently if we install a new Westermo device that has a new version of WeOS, we upload the older version of WeOS using WeConfig. 

WeConfig has been an extremely valuable tool when performing firmware upgrades and has produced significant time savings. Normally each device would have to be updated individually in turn. Using WeConfig it is possible to perform firmware updates over the network for groups of installed devices, without the need to visit the substations. The task is performed from the office and when the firmware is being uploaded the engineer is free to do other tasks. By automating the process, we have not only increased worker efficiency considerably, but also minimised the possibility of human error. With such a large number of devices forming the network, making changes to each one becomes very laborious and this is when mistakes happen. This possibility has now been removed.

For each firmware upgrade we have saved about 35 working hours. We have performed four firmware upgrades, which means another 140 working hours have been saved by WeConfig. Some substations have an Operational Restriction status which doesn’t permit access by telemetry engineers. The status of a specific substation is not always visible, so this could previously have resulted in a wasted site visit and prevented an upgrade to the installed devices. However, performing the task remotely means we no longer worry about gaining access to all the sites.