the question is inspired by the Stuxnet attack on the Iranian factory equipment.
To me the moral of the story seems to be that if you are doing things that may be disapproved of by the German government or by other people who can order around a major German machine tools company, you better keep an air gap between your machines and the outside world and prevent traitors with flash drives from getting anywhere near your electronics.
Still, this raises the natural question - are there any normal procedures in the lifecycle of machine tools in the factory that could be used for the introduction of trojans if the manufacturer were so inclined? E.g. is there an equivalent of “Windows updates” for the machines’ software?
Or is the rigorous air gap and gallows-for-flash-drives policy sufficient for countering any trojan threat?
I can’t (well…won’t) speak for my current employer… But at my previous company, a major US cell phone manufacturer, we ran Windows XP on our stations that calibrated and tested the devices. There were defined maintenance windows to handle updates.
I’m not sure what operating system was used for the SMT part of the line.
-D/a
Not all factory computer equipment gets updated. In one of my jobs they had equipment run by computer software that was truly ancient. It was custom-written for the platform, and was probably pre-DOS. If the computers ever went down, I doubt if they’d be able to find anyone to get it back up and running.
The company itself has since gone under, so the point’s moot.
Where I used to work, production employees didn’t have admin rights on their computers and were limited in the programs they were allowed to run. That didn’t stop regular Microsoft updates from occasionally breaking production software. If I was the CIA or Mossad, I’d have someone inside Microsoft to slip their payload into regular updates releases. You would only need Stuxnet for computers that aren’t hooked to a network or aren’t updated by IT.
The hard part was dealing with equipment in the service department that might involve some custom built ISA bus card or hardware from some company that no longer exists or doesn’t support it anymore or components that aren’t manufactured anymore. Sometimes you end up with some old POS computer sitting on the shelf, so you can pull it down to run an engraving machine with software for Windows 3.1 or a parallel port driver for Windows 95 for an EEPROM burner.
why would people want to control a mission-critical CNC machine from a Windows computer? Wouldn’t a minimalistic Linux distro be more stable for this purpose?
Of course, Stuxnet was in fact a Windows worm, so apparently the Iranians thought it a fine thing to use Windows machines.
I work in IT for a pharmaceutical manufacturing and distribution center. We run a mix of different systems but most are just Windows XP workstations. The level of checks that new code goes through to make it into production is insane so that when something is working properly, it is mostly locked down from updates until someone wants to go through the paperwork and hassle of getting new software approved. We don’t really use the newest software at all even for new projects. We lag about 3 - 5 years behind the newest software just because reliability is so much more important than new features. We still get the occasional patch pushed down from a vendor like Microsoft that breaks something big so there are reasons to be very careful.
However, anyone in the building could cause problems if they really wanted to and knew what they were doing (most do not). That is a difficult problem to avoid completely. An IT environment that is completely locked down loses a whole lot of functionality and it creates a real issue when dealing with unexpected issues. Banning all flash drives is good from a security standpoint but very bad for many different types of users. You would also have to take away floppy disks and optical media if you were that concerned about rogue files and you can’t get rid of e-mail attachments completely.
another question would be, if we use a Windows machine to control mission critical equipment, why risk doing updates in the first place? Assuming that the purpose of updates is to protect against online threats (regardless of the motivation of bad people who create them) why not just get everything to work correctly, impose the air gap and not update the machines any further?
Windows machines don’t control the machines - PLC’s control the machines. PLC’s can have their programs downloaded from any kind of device programmed to download to them. Most people use Windows to do this.
The actual Stuxnet payload was a revised version of the PLC program, not a Windows program.
if flash drives are useful for normal operation of the facility, could you just prohibit the bringing in and bringing out of flash drives and put up a scanner at the entrance to pick them up? Presumably this would make sense in a place where there is no internet connection to the outside or else the incoming traffic is highly restricted (e.g. text messages allowed but binary not allowed).
Most workstations aren’t single purpose and most need some sort of an internet connection. Once you have an internet connection, you need to keep Windows and all your other software updated to protect against threats. Even if you didn’t have an internet connection, you would still need anti-virus updates because they can come from lots of different sources. You need to work at a place like that to understand the problem as a whole. You can make workstations as single-purpose as possible and lock them down as much as possible but it is a balancing act. Industrial workstations are usually multi-purpose just because it is more space and work-flow efficient.
It doesn’t take that much computing power to run a typical industrial machine so it also used to access other applications and maybe even e-mail. Many large companies have web based industrial apps that require a browser and internet connection to work. You can isolate a workstation from the rest of the world very easily and that is sometimes done but that also takes away many of the advantages of modern computing. Making a perfectly secure computer is easy. You just never connect it to anything else and hope that you don’t need driver or other software updates for the equipment you are running. Making it both practical, secure,and centrally accessible for other IT personnel is a lot a more difficult problem that involves not just computers but also physical security and enforceable work practices.
I am familiar with control equipment used in power stations. In the locations I am familiar with, there is a complete ban on any data connection between the control equipment and the outside world. This is not a software firewall, it is a hardware gap. The only data that is allowed to cross this gap is analog 4-20 mA signals or similarly dumb protocols.
Updates do not take the form of ad hoc downloads of software patches, etc., as happens with Windows machines. Instead, a massive upgrade of the whole station control system (hardware and software) is carried out at a cost of many millions of euro. This happens infrequently, at intervals of 10 years or more. Between such upgrades, the station basically lives with what they have.
In my company, “Office Automation” is completely locked down and under a strict regimen of what can and can’t occur. Plant floor stuff, though, is almost completely unregulated. And it sucks. Because the plant floor stuff is on its own private network, it can’t receive security updates automatically. Or virus definition updates. Because there’s a lot of sneaker net, viruses run rampant. The biggest downside to a virus is that it will infect our office PC’s (because we take months to push updates; I have admin rights on my laptop, and so run my own anti-virus).
For us, a virus has never compromised production, because we don’t use Windows machines for production. They’re only MMI’s (man-machine interfaces). Actually, HMI’s (the more modern human-machine interface). Actual controls are propriety hardware; no software PLC’s in my company. Someone could write a virus to put junk parameters into our PLC’s, conveyors, welders, etc., but no one bothers. If the HMI is down, plug in your laptop, and you’re okay (viruses cannot live on the production systems and re-infect a Windows laptop).
As an engineer who works for a major japanese machine tool manufacturer which partnered with a major German machine tool manufacturer I can tell you that the fact that our machine tools’ interfaces run on Windows Embedded is a huge source of concern, especially lately that more and more networking features are included with them. In my opinion, once hackers learn a lesson from Stuxnet, and realize how easily they could do huge amounts of damage… I don’t know, but all hell is going to break loose. just the other month one of our simulators got infected with a virus (this is in building full of engineers, with a dedicated IT staff). For now these stuff is mostly innocuous, but if these viruses start getting nastier and nastier, the average mom-n-pop tool shop doesn’t stand a chance.
Flash drives are so small, they could easily be concealed on a person (or even inside!). I’ve seen them built into fancy wristwatches, or as jewelery.
How about CD’s? Many workers carry music CD’s to listen to at work, and it would be easy to also record some computer code onto one. Or cellphones. You could easily record a virus onto your cellphone, and carry that into work.
I guess it depends on how restrictive you want to be, before all your good workers quit. You could ban all flash drives, CD’s, wristwatches, cellphones, prohibit wearing jewelery, require strip-searches & body-cavity searches, etc. But it would cost a lot, and many of your best workers would soon go elsewhere.
so why should any machine that generates programs for PLC, microcontrollers or anything else related to running the equipment run on Windows? Why can’t this entire toolchain be switched to a Linux distro of the more locked down variety?
Is it a matter of certain key development IDEs and tools being only available for Windows? Or the lack of that easy enough to deal with for the typical shop IT distro? Or why?
ETA: there is also such a thing as Windows emulator on Linux. If this emulator can be made virus resistant, e.g. only running the Windows-based IDE process and nothing else, perhaps that would be the silver bullet solution?
The real answer is “momentum.” The end user (not the engineer, but the plant floor tech) expects things to run in Windows, and have a Windows GUI interface. Sure, you could try to force Linux on the customer, but the customer has 10 other systems to program, and they all run on Windows. If you require Linux, then you can’t click on an icon on the same desktop to program one device versus the other.
Stuxnet (so far) is a special case. Viruses (or rather, their programmers) want something. Look at the proliferation of Mac viruses. Look at the proliferation of Macs. How many Macs exist compared to, say, PLC5 systems? Unless you’re an enemy combatant, there’s not much use in writing a Rockwell-specific virus.
the “enemy combatant” bit is a good point, especially now that at least one successful use of the technique has been demonstrated.
So has Pentagon begun trying to enforce a more secure toolchain at least in the military contractors? I mean, traitors is one thing, but the basic vulnerability to viruses from China looks kinda dumb.
Or is the mechanism of PLC updates such that only a few just-updated machines would get damaged and then everybody else will read the newspapers and start reinstalling Windows?
Does PLC let’s say have access to “system’s time” so that first lots of machines could get infected and then attack would begin at a specified time in many places at once? If that is a threat, would it make sense to desynch the PLC system’s clock from the official time by a random offset?
Yes, they are working on this. My nephew works for DISA (Defense Intelligence Security Agency) and spends much of his time on this.
When he was home last fall, I noticed that his laptop had a fingerprint verification sensor built into it, and that he had to use this to logon. He mentioned that this is or was going to be standard on all military laptop computers.
You may not know this, but a flash drive is useless without admin rights to install the drivers. We also had restrictions on what applications the production people could run.
I see, interesting. Nevertheless, this particular security measure is not directly related to the thread topic. It’s one thing for military to try protect the military powerpoint presentations on their own laptops. It is quite another for them to consider the implications of a Stuxnet variant damaging jet fighter production or other parts of the military industrial infrastructure.