The cause of the east coast black out

Factual Questions
1

I picked up the latest copy of my favorite computer magazine today, and was treated to an explaination of the cause of the big black out.

CT is blaming it on the Lovsan worm. The claim is that the “Control Area Operators” - which manage the load distribution for the powerplants - use OLE for Process Control (OPC) under Windows. OPC uses the same IP port as DCOM, and Lovsan makes use of DCOM for its operation. They leave it rather open as to whether the control room PCs got hit by the worm or if network operators blocking the OPC/DCOM port did the actual damage.

The magazine quotes some German powerplant experts who comment that the CAOs use Windows in the control room and the internet to connect with the powerplants, and that the same situation could never occur in Germany because the powerplants have their own network infrastructure that connects neither with the internet nor the normal office networks in use by the powerplants.

Has anyone else heard that the worm was the cause of the blackout, and where could I find out more about it?

2

The power barons say that there was no sign of sabotage, and they wouldn’t lie to us. The power barons say it surely means they need to build more powerplants and transmission lines, and we have no reason to doubt them. The power barons say it’s preposterous to suggest they crashed the grid intentionally to make us glad to relax pollution regs; they are surely correct, right? The east coast blackout was definitely not rigged like the 2000 California blackout was. I’m pretty sure of that, aren’t you?

:rolleyes:

3

Al Qaeda hid my egg salad sandwich.

4

Jeez, folks. I asked for a little information, maybe a little speculation on whether the systems shouldn’t have been on windows or at least on on the internet - and I get two answers pushing conspiracy theory level crap at me.

No where did I imply that Lovsan was aimed at the powerplants. Just that they got hit with it like a great many people did who use improperly secured Windows systems.

5

AskNott, you know better than to spout nonsensical conspira-rants in GQ.

It’s clear that the root cause of the blackout was the failing of two large transmission lines on Ohio, which led to the eventual overloading of the Lake Erie Loop. The exact sequence of events after that point is still in dispute. It’s clear that some safety systems did not work; the East Coast should have isolated itself from Lake Erie in time, but it didn’t. NYC and Westchester could have probably isolated themselves from upper New York in time, but didn’t. Most of the research to date cites inadequate communication between regional controllers as one of the main factors, though I don’t doubt that a worm could contribute, if a power company was stupid enough to put essential equipment on the public Internet.

The one theme that’s common in all major engineering disasters is that a single failure is usually not a problem. Transmission lines go out all the time and nobody notices because there’s redundancy. For a major catastrophe to occur, you must have several disparate failures that occur at the same time, which is what appears to have happened here.

6

Thanks, friedo.

That was the point that CT was making - the worm screwed up th communications and slowed or stopped the systems that should have isolated the problem. Whether that was a direct effect (control centers infected with the worm) or indirect (routers on the internet blocking port 135 to try to stop the worm) was kind of open.

The thing is, I still don’t know if PCs are used as CT states and if that was enough to topple the system.

Do you think Anthracite has any experience in that direction?

7

Here is a British journalist’s view of some details about the blackout. (Warning: political)

The most interesting part was the news that these same companies have had problems before! I didn’t see that mentioned in any of the TV news coverage.

Here’s the first part of the article:

8

The operators DO communicate with each other over internet connections and do have Windows based sytems to do that. They ALSO communicate via telephone, power line carrier communications, tower to tower microwave and dedicated telephone and fiber optic lines.

On top of that they “listen” in on the lines themselves. System operators all over the US “saw” those disturbances as they happened via system frequency monitoring.

So, while a worm could cause a little confusion, it can not cause communications between operators to cease.

They haven’t stated the cause yet but my guess (25 years in the business so its an educated guess) is that the automatic regional protection systems (called relaying systems) worked like they were supposed to but they failed to coordinate properly with the relay systems of other regions. That caused the other systems to react improperly and actually feed the problem instead of isolate it.

So I think both AskNott and the power barons will be dissappointed when they announce that improved coordination of regional relaying sytems is required; not new power plants.

Bubba

9

Probably should have told you:

My company is further west and was not blacked out.
I don’t care to defend First Energy or any others.
Don’t own any stock in the blackout companes.

Just the opinion of a DUG (dumb utlity guy) engineer
Bubba

10

From my experience working on these systems they are usually
on a seprate sub-net from the internet. Having a control system on the actual internet would be a major security risk.

Majority of controls to breakers go via Radio, Fiberoptics or Telephone lines, that are independent of the Internet.

The other part of this from all the SCADA (control systems) that exist most do not run on windows machines. Most actually run on UNIX machines since the requirments for the servers are very high on up time and until recently windows didn’t meet these.

So I don’t think the worm would have affected the controls too much.

11

I have. I even saw it mentioned on CNN.

12

After the brickbats and overripe tomatoes landed on my porch, I feel I should reply in some way.

  1. The power companies told the press within an hour of the blackout that there was no sign of sabotage. Awfully quick work, no? The media accepted that assertion without question.

  2. Even before the grid was fully restored, power companies were making their case for more transmission lines and more power plants. In fact, former Halliburton CEO Dick Cheney had already declared the need for more power plants before the blackout. Hey, it could be true. What the heck do we know?

  3. If you asked a power company executive if the crash was intentional, don’t you think she’d say it was preposterous? Of course she would. Just yesterday, the EPA announced that regulations for electric utilities would be relaxed. Connected? Of course not, but what a lucky coincidence.

  4. The Justice Department of the state of California has begun proceedings to prove their contention that the energy crisis of 2000 was artificially created by manipulating refinery production and shipping away petroleum that California needed. They further contend that the power companies coordinated their “maintenance downtime” to overload critical transmission lines, causing blackouts. Electricity, they contend, was sold to other states and sold back at higher prices. If this is nonsensical, friedo, I didn’t make it up. California says there was never an energy shortage. The fact that it made California voters angry, just in time for the election was just a coincidence.

The east coast blackout was a different phenomenon, as I said. We wouldn’t fall for that twice. I’m pretty sure of that, aren’t you?

13

Al Qaeda burnt my grilled cheese sandwich.

14

Al Qaeda burnt my grilled cheese sandwich.
Al Qaeda hid my slippers.
Al Qaeda moved my lawnmower.
Al Qaeda sat in my chair and didn’t put the pillow back.

15

Hey AskNott,

I just gotta respond to your comments:

1 - Within an hour the power companies said they could find no evidence of sabatoge. That means within an hour they found no bombs, no breakins, no EVIDENCE of anything other than their protection systems operated locally for legit reasons. They made this bold statement because it was gasp true. Most system operators knew within seconds that their systems crashed because of frequency problems. They have the indicators right there in front of them.

2 - Power companies have been making the case for improved transmission systems for about 30 years now. They sell power. They want to get the power to market. Transmission systems take the power to the market. I don’t believe that to be a conspiracy.

3 - Preposterous? That’s a strong word. I would thing that a power company spokesperson would say that is is highly improbable. Remember those barons you spoke of? I don’t disagree with you that there are some evil filthy top dogs in this business. But the operators aren’t part of the rich elite. They drive minivans and pickups, not limos. And THEY don’t particularly like their own CEOs. You’re suggesting a conspiracy which is anchored by a coordinated effert involving a lot of people (operators) who would,in reality, drop a dime on their fat cat supervisors in a heartbeat.

4 - The California market system was designed by idiots and exploited by marketers. IT is a perfect example of how NOT to run a power market. Whatever happend on the East coast was definately not patterned after California. What you said about California though, I’m not going to argue with.

You want to hate power companies? Go ahead. Its easy to see why a lot of people do. But I still doubt that they intentionally did this. I think that they just plain screwed up.

16

The “power companies” did? The officials I spoke with at the “power companies” on that day said no such thing to me, and it wasn’t any secret communicade either. They said they didn’t know and they meant it. I heard the media report that the Office of Homeland Security said there was no sign of sabotage.

And given that there was no sign of sabotage, I’d say that that was pretty accurate.

This case has been made for more than a decade, ever since excess capacity at most utility systems has been down near the NERC “danger level”. Utilities have been avoiding capital projects for a long time due to liability risk and fighting environmental groups, as well as focusing on stock value during the mid-1990’s and being confused about how deregulation is going to pan out. The only reason it seems “sudden” to people not in the industry is that the reliability of power generation is something people sorely take for granted, and news about complaints lines and corridors are being overloaded gets read in the papers far less than the latest sports scores.

You’re making completely unfounded accusations, and you’re being a jerk about it. First off, it wasn’t “just yesterday”, so don’t lie about that - the comment period started last year. Second, read my 20,000 word post I made in the Pit yesterday about this very subject, complete with numerous cites from the EPA and the 11th Circuit Court (not that you’ll go and read it). All you are doing here is spouting off wild theories with no factual basis for it. Why, exactly?

Odd, since so little power is produced by petroleum in the US. In fact, it was nearly 3% thus far this year. How exactly is the petroleum production related to electric power production in any significant manner?

“They” were in a konspiracy to overload key lines? Cite? Proof?

This happens nearly every day, in nearly every State. It also happened because California-based utilities were allowed to do so by their own government (think LA).

17

BubbaDog disagrees with me on some points, and I’m comfortable with that. He seems to feel that I meant to malign ground-level utility workers. That was not my intent.

Anthracite also disagrees with me on some points, which is OK, but also calls me a liar and a jerk. Allow me to sweep off my porch, again.

quote:
You’re making completely unfounded accusations, and you’re being a jerk about it. First off, it wasn’t “just yesterday”, so don’t lie about that - the comment period started last year. Second, read my 20,000 word post I made in the Pit yesterday about this very subject, complete with numerous cites from the EPA and the 11th Circuit Court (not that you’ll go and read it). All you are doing here is spouting off wild theories with no factual basis for it. Why, exactly?

You say it wasn’t “just yesterday,” and accuse me of lying about it. Here’s a cite. http://www.npr.org/rundowns/segment.php?wfId=1413910 Perhaps you heard about it long ago, but until August 27, it had not come to my attention. The story gives the impression that the easing of the clean air rules happened that day.
I rarely read The Pit, and if you think I’m gonna go there to read a 20,000 word rant, you must think I’m really starved for entertainment.

quote: (first me, then Anthracite)
*4. The Justice Department of the state of California has begun proceedings to prove their contention that the energy crisis of 2000 was artificially created by manipulating refinery production and shipping away petroleum that California needed.

Odd, since so little power is produced by petroleum in the US. In fact, it was nearly 3% thus far this year. How exactly is the petroleum production related to electric power production in any significant manner?*

The California energy scam was not limited to electricity. Gasoline prices, California contends, were artificially jacked up by exporting supplies until an apparent shortage occurred.

quote:
“They” were in a konspiracy to overload key lines? Cite? Proof?

I’ll leave the proof to the State of California. I don’t work for them, and I’m not in a position to see their evidence. You seem to be extremely well read in these matters. Surely you are aware of the case. If you have time to post 20,000 words in The Pit, you don’t need to play dumb about the California case, and you don’t need to backhand me by spelling “conspiracy” with a “k.” The proof, or lack of it, will come out in the courts.

18

First off, it’s 20,000 characters. I mis-spoke. I only know that because I ran into the posting limit.

Second off, it’s not a “rant”, it’s a factual post where I said I had many cites. If you want to put your hands over your eyes and not read it, your loss.

Third, if you’d read my post, you’d see the truth behind the alleged easing of the “clean air rules”.

Fourth, I can certainly believe you didn’t hear about it until that day, but your context was that it was a “lucky coincidence”, and thus timing is important. You claimed the EPA announced it yesterday, when in fact it has been open for comments for 120 days. If what you meant was that you “first heard of it yesterday” when they arrived at the final ruling, then I apologize honestly for saying that you were lying. That’s my bad, if that’s your intent, and I’m wrong. But it’s still no “lucky coincidence”, no matter how one slices it.

Fifth, instead of reading what I wrote, you’ve taken the tack of mocking me (see also your comment below) for trying to write something at length to explain the situation and fight ignorance? That’s really something.

Forgive me, but your post specifically referred to plants and their maintenance downtimes, overloading of transmission lines, and the “energy crisis”, which was almost entirely the electricity-based one. So why is petroleum mixed up in there? You need to separate the topics more cleanly if you want some honest answers. I can’t talk on the vehicle petroleum issue, because I don’t have enough information and experience depth.

Of course I’m aware of the case, which is why you get called out for saying the over-the-top things you did.

So in other words, you have no proof, no cite, and were posting this to elicit a response from other Members here? “I don’t work for X, so I don’t have to provide any cites?” That’s not good form. Do you think people should be allowed to do that in factual discussions?

19

I believe Anthracite and I have lobbed enough stones at each other. Each of us has a clear opinion of the other. If we continued trading fusillades, no clear purpose would be served, other than testing the limits of the beleaguered hamsters. Be well, Anthracite.