Over the past five years, ransomware has emerged as a vexing menace that has shut down factories, hospitals, and local municipalities and school districts around the world. In recent months, researchers have caught ransomware doing something that’s potentially more sinister: intentionally tampering with industrial control systems that dams, electric grids, and gas refineries rely on to keep equipment running safely.
As an expert in the field of critical infrastructure cybersecurity, I always expect this response to any discussion surrounding this topic. It’s by far the most salient example of “easier said than done.” There are many technical and business reasons contributing to the connectedness of critical infrastructure.
I wasn’t sure anyone cared if I listed some out, so I kept my initial post short.
The main power company I’m housed at owns thousands of electrical substations across seven states. To maintain the same level of operational agility and performance you’d have to staff each site with several techs and engineers (pricey salaries). This translates to millions of dollars per year and is the biggest reason.
Regulations (NERC/CIP in N. America) require generation and distribution companies to report hourly/daily grid information to coordinating centers to (attempt to) balance load and demand across hundreds of operators in a grid. This data has to get out of the distributed grid elements one way or the other.
Meter reading (from residential to commercial to substation meters) used to require contracting a fleet of people, vehicles, and related equipment. Now it happens automatically and mostly wirelessly, saving tons of money and removing lots of risk (personnel, safety, cyber risks around patching extraneous mobile devices, etc.).
Site connectivity allows you to embrace newer, better tech like “smart grid” type things. I won’t expand into that (yet), that’s a whole other polarized area of discussion.
Engineers are always needing to configure equipment or acquire at remote sites. Remote connectivity is a huge convenience.
Power companies don’t accomplish 100% of things themselves; they outsource a lot of things. The primary company I’m at outsources the maintenance and administration of renewable energy equipment (wind farm turbines and similar), which requires remote connectivity, vendor DMZs, etc. Less holistic things like transmission line quality analytics typically requires dropping a vendor’s sensor on-site so they can house their big compute pieces in AWS.
You see the biggest reasons stem from massive cost savings. You might argue, “this is important - suck it up and spend the cash,” but alas these power companies are beholden to their rate payers. You can only recover costs by raising rates which has to be done in a formal rate case to state entities and gets highly scrutinized by local politicians, activists, watchdog groups, etc. Nobody wants to pay more for power. If we raise rates in an area by a penny we get massive backlash; “Why are you raising our rates to pay for your new fleet of Windows 10 laptops? XP was just fine, and my power kept running at my house…” Now imagine we take all critical infrastructure offline and try to increase your power bill 100-fold.
Additionally, even if a company wanted to suck it up and pay… this is a private company we’re talking about. All the big ones are (Duke Energy, NRG, Southern Company, etc.). Their CEO would get fired in a heartbeat.
There is a difference between being Internet-connected and network-connected. We’re not dropping some residential modem at these sites; we lease a variety of WAN options depending on availability (nobody will run fiber lines to a mountain-top, so sometimes you can only do things like microwave repeaters) where we can. Not every company does this.
Remote connectivity often means serial instead of IP connectivity, though more likely it means serial protocols encapsulated over IP nowadays.
99% of attacks against energy companies affect their headquarters IT network. These get blown up in news reports because journalists don’t understand the difference, and any energy company worth their electrons will properly firewall + DMZ between IT and OT networks.
I keep thinking of more caveats. You might retort to the outsourcing part, “Why not obtain the talent yourself and insource?” to which I’d reply that most (jeez… all?) renewable energy is taken advantage of whenever there’s some huge government incentive. To take full advantage of grants, you need to be able to execute on contracts very quickly in areas where your staff lacks expertise. You just can’t do that with traditional staffing.
Anyway, like anything else it all comes down to $$$ money $$$.
oh yeah over here - nationally - we have a system in place where you can send the index by phone (voice or sms) or through the data line with an account … but i mean the internal-infrastructure itself could be decentralized using p2p/TOR/VPN (not too happy about sending everything through voice/sms … )
and they have a fleet that comes bi-annually or once a year to take the measurements ON-site …
I understand that someone operating an electricity distribution network will want remote access and yes it will probably cost more money to tie something together without using the internet than using the internet. I understand also that someone might make one decision for end-user meters and a different decision for other parts of the network. (At this stage I am unsure whether the end-user meter at my place represents more of a security risk to me than the other way round. )
At the same time, some parts of the organisation may be on the internet, hence the need for and inconvenience of an air gap.
If the business reasons are overwhelming to put critical infrastructure on the internet, I suppose the question would then arise: what do companies need to do in order to make that acceptably unsafe given the clear and present danger?
In some countries the government is going to take the decision out of your hands. There is clearly a national security element to this discussion and that may overrule cost considerations. Countries that prioritise cost may be at a disadvantage were hostilities to break out.
However obviously you have the real sector experience, so feel free to expand on anything.
There must be of course a distinction between OT and IT network and in f.i. in some countries it is regulatory requirement to have ISO-certified, audited, ringfenced OT network. However last-mile (consumer voltage) network is not frequently considered as Operational hence is not regulated (unlike the grid).
I am also in cyber-security, but my recent expertise is more along the lines of classified, air-gapped enclave networks. However, I used to be into distributed flight test telemetry, from the aircraft in test, relayed across the country (or across the sea), to the realtime displays in the mission control room. Our technicians remotely operate, maintain, troubleshoot and upgrade each site (most of which are un-manned). All of these links are tunneled using dedicated hardware encryptors, and the transmission media vary from microwave broadcast, laser over the air, dedicated fiber, commercial fiber or copper, depending on the latency and throughput requirements.
I understand that cost and scale are issues, but I’ve always wondered why most of your problem wouldn’t be solved by keeping the critical infrastructure part of your business behind dedicated hardware encryptors. The ones we use are not cheap, and the key management effort is substantial, but it seems like small potatoes in the context of possible attack resulting in mass casualties and a prolonged return to the Paleolithic.