NASA screws up again...

[robby]

From this CNN article.

Spirit operated beyond the scientists’ dreams until its 18th day on Mars, when it suddenly stopped sending anything more than tones back to Earth letting scientists know it was still there.

…Trosper said the problem appeared to be that the rover’s flash memory couldn’t handle the number of files it was storing.

…She pointed out that the scientists had thoroughly tested the rover’s systems on Earth, but that the longest trial for the file system was nine days, half of the 18 days Spirit operated before running into the problem.

You have got to be kidding. NASA considers a nine day trial to be a “thorough test” for a $400 million mission projected to last at least 90 days?! :rolleyes:

It seems like it wouldn’t take too many additional resources to run the trials to last at least as long as the mission is projected to last.

[Grey]

Not to rain on your parade but I’d guess that testing the flash memory was one of those bottlenecks in the verification cycle. Lengthen the time to test and you wind up delaying dependant test cases thereby extending the testing time frame a lot more than just 9 days. Besides VxWorks isn’t new technology and neither is using flash storage.

I should probably take at look at what they did do though.

[Kinthalis]

@robby

From this It seems like it wouldn’t take too many additional resources to run the trials to last at least as long as the mission is projected to last.

And you would know this how?

There are a multitude of consideration that had to be taken by NASA, including their budget, and the time window for launch. Waiting 3 months more might have delayed the launch for 2 years, which would have scrapped the mission (yes, even just holding the project ready for launch for two years would cost too much).

[UncleBeer]

And now that Spirit is responding again, it’s my understanding that NASA expects to complete all planned tasks anyway. In fact, according to my local paper yesterday, they expect to be able to operate the rover for a period much longer than the original ninety days.

So, what’s the problem again?

[Liberal]

So, was Trosper just talking out her ass or what?

[robby]

@Kinthalis

And you would know this how?

I don’t. Hence the qualifier of “seems.”

@Kinthalis

…Waiting 3 months more might have delayed the launch for 2 years, which would have scrapped the mission (yes, even just holding the project ready for launch for two years would cost too much).

I realize they have launch windows. When setting up the mission planning, though, I would expect a longer testing phase to have been inserted into the timeline. A nine-day test seems to me to be insufficiently short for a mission with an expected lifetime that is ten times longer.

My point is that this potentially serious failure appears to have been eminently avoidable. The failure has already cost the mission several days of on-planet research, and for a few days there, looked like it was going to shut down the mission entirely. What gets me is that the memory problems were not due to anything unexpected. They have apparently arisen simply from normal operations.

[Weirddave]

@robby

I don’t. Hence the qualifier of “seems.”
I realize they have launch windows. When setting up the mission planning, though, I would expect a longer testing phase to have been inserted into the timeline. A nine-day test seems to me to be insufficiently short for a mission with an expected lifetime that is ten times longer.

My point is that this potentially serious failure appears to have been eminently avoidable. The failure has already cost the mission several days of on-planet research, and for a few days there, looked like it was going to shut down the mission entirely. What gets me is that the memory problems were not due to anything unexpected. They have apparently arisen simply from normal operations.

“seems”, “appears to have been”, “apparently”. Did you wake up today with a sudden urge to rail against NASA? It appears to me that you have no knowledge of the subject, and are simply blowing smoke out of your ass for the hell of it. You should run for Congress, you’d fit right in.

[Grey]

Ok so you have 128 MB of memory that needs to be dumped. The rover can transmit 60MB directly to the orbiter once a day and can transmit for 3 hrs to Earth 130 MB. That means that the flash memory could be filled and dumped a grand total of 9 times over 9 days. From the MarsRovers site

The data rate direct-to-Earth varies from about 12,000 bits per second to 3,500 bits per second (roughly a third as fast as a standard home modem). The data rate to the orbiters is a constant 128,000 bits per second (4 times faster than a home modem). An orbiter passes over the rover and is in the vicinity of the sky to communicate with the rovers for about eight minutes at a time, per sol. In that time, about 60 megabits of data (about 1/100 of a CD) can be transmitted to an orbiter. That same 60 megabits would take between 1.5 and 5 hours to transmit direct to Earth. The rovers can only transmit direct-to-Earth for at most three hours a day due to power and thermal limitations, even though Earth may be in view much longer.
Mars is rotating on its own axis so Mars often “turns its back” to Earth, taking the rover with it. The rover is turned out of the field of view of Earth and goes “dark”, just like nighttime on Earth, when the sun goes out of the field of view of Earth at a certain location when the Earth turns its “back” to the sun. The orbiters can see Earth for about 2/3 of each orbit, or about 16 hours a day. They can send much more data direct-to-Earth than the rovers, not only because they can see Earth longer, but because they can operate their radio for much longer since their solar panels get light most of the time, and they have bigger antennas than the rovers.
On-board memory includes 128 MB of DRAM with error detection and correction and 3 MB of EEPROM. That´s roughly the equivalent memory of a standard home computer. This onboard memory is roughly 1000 more than the Sojourner rover from the Pathfinder mission had.

Now how hard do you think it is to put 128 MB on flash and then dump it? Even if we used the slower rover to earth rates (12,000 bps) it would take 3 hours. Now if you test every 3 hours for 18 hours a day over 9 days you would have run a Mars equivalent of 54 days. Script it to run over 24 hours and you 72 effective Mars days of usage.
Test it with the rover to orbiter rates of 128,000 kbps and it would take 17 minutes. Round that to half an hour and over 9 days you could test the equivalent of 432 Martian days.

[Musicat]

*from http://abcnews.go.com/wire/US/ap20040127_1171.html *
Spirit was crippled after its random-access memory proved inadequate to manage the rover’s flash memory, packed with unnecessary files that had piled up since the spacecraft’s launch.

They must be running Windows, with its “advanced” :rolleyes: garbage collection scheme.

@robby

I realize they have launch windows.

Yep, guess so.

[Revtim]

I was a professional software tester for ten years, and believe me, it’s very easy to devise testing procedures that would find a bug after the bug occurs in the field. Hindsight being 20/20, and all that.

What’s tricky is devising test procedure that will find the most likely problems in the time/budget you have alloted. And it’s all but impossible to find every single bug.

[Grey]

@Musicat

They must be running Windows, with its “advanced” :rolleyes: garbage collection scheme.
Yep, guess so.

I think they run on Wind River’s VxWorks. There is a version with memory management and a version without it as far as I know. I have no idea which they use.

Revtim, you said it perfectly.

[Zebra]

Oh that nutty Spirit.

His system is loaded up with spyware from all those Mars porn sites it was visiting to kill time on the trip to Mars. They need to download Adaware to take care of it.
How do I know that Spirit was doing that?

Well, he is a bit of a rover.

(you may now, stone me to death)

[robby]

@Weirddave

…Did you wake up today with a sudden urge to rail against NASA?

No, I did not “wake up today with a sudden urge to rail against NASA;” the urge to rail did not arise until I read the comments made by NASA’s mission manager. And Trosper’s comment that “the scientists had thoroughly tested the rover’s systems on Earth, but that the longest trial for the file system was nine days” set off my B.S. meter.

@Weirddave

“seems”, “appears to have been”, “apparently”…It appears to me that you have no knowledge of the subject, and are simply blowing smoke out of your ass for the hell of it. You should run for Congress, you’d fit right in.

I never claimed to be an expert on the subject, but I am an engineer. Not being an expert, I do feel compelled to throw in a bunch of qualifiers. Nevertheless, I believe that my OP was more than mindless ranting. I’m sorry you disagree.

[Grey]

Well robby, when you lead with your chin you’re bound to get hit. 9 days of testing does not mean 9 operational days. It means you tested various aspects for 9 days. As I showed you can do numerous effective days by simply avoiding the downtime required to wait for a suitable target to beam the data to.

[Ferrous]

Buncha maroons. Sure, they designed and built the most capable and advanced robot ever, sent it millions of miles through space, then landed and deployed it on Mars. Twice. But then they had a minor software glitch that took them like days to fix! Idjits.

[robby]

@Revtim

I was a professional software tester for ten years, and believe me, it’s very easy to devise testing procedures that would find a bug after the bug occurs in the field. Hindsight being 20/20, and all that.

What’s tricky is devising test procedure that will find the most likely problems in the time/budget you have alloted. And it’s all but impossible to find every single bug.

If Trosper had said that something like this was to blame for the memory problem, I would not have complained.

However, her clear implication was that running the trials for only nine days was why the problem was not encountered during testing, and that longer testing would have exposed the memory problem. If true, that seems like an inadequate testing process to me.

@Grey

…9 days of testing does not mean 9 operational days. It means you tested various aspects for 9 days. As I showed you can do numerous effective days by simply avoiding the downtime required to wait for a suitable target to beam the data to.

In her statement, Trosper directly compared the nine days of testing with the 18 days of operation on the surface. Sure sounds like she’s talking about nine operational days of testing to me.

[MaxTheVool]

NASA has really been on a roll lately. Spirit and Opportunity both landed on Mars, which is close to impossibly difficult. Spirit is hopefully fixable.

Before that, the Stardust probe grabbed particles out of a comet, and before that, the mighty Spitzer Space Telescope was succesfully launched, although no one seems to care. Nothing’s blown up or crashed in quite a while.

[Grey]

Actually the article says

She pointed out that the scientists had thoroughly tested the rover’s systems on Earth, but that the longest trial for the file system was nine days, half of the 18 days Spirit operated before running into the problem.

Which is not a statement by the lady, but her statement paraphrased by a reporter with likely little exposure to engineering verification cycles/testing.

All I’m saying robby, is my money is on the JPL testing scheme rather than your guesses. You’re pissed of that they didn’t do what you would’ve like despite the fact you don’t know what they did and why they did it. Not a strong position, IMHO.

[Musicat]

Does Mars have an Internet connection? If so, I’m sure Spirit’s problem is the MyDoom virus.

No doubt about it.

[robby]

@Grey

…Which is not a statement by the lady, but her statement paraphrased by a reporter with likely little exposure to engineering verification cycles/testing.

So you’re now reduced to questioning the news source? :rolleyes: Who’s guessing here?

@Grey

All I’m saying robby, is my money is on the JPL testing scheme rather than your guesses. You’re pissed of that they didn’t do what you would’ve like despite the fact you don’t know what they did and why they did it. Not a strong position, IMHO.

Oh, give me a break. I am simply questioning the adequacy of their testing program, based on a public statement by the mission manager.

I haven’t been able to corroborate Trosper’s statement via another news source, but did come across this story, which states:

Spirit can operate without using the flash memory, an avenue called the “cripple mode.” That mode tells the flight software upon booting up that it should operate with its file directory out of the robot’s random access memory, rather than the flash memory. “That would avoid any issues that we might have with either the flash memory itself or the flight software that was used to write to it,” Theisinger added.

…Prior to its troubles on Mars, Spirit was being put through its paces – doing things that were not tested with as much vigor here at JPL before being launched. Spirit does not have a huge track record of testing, a source said, for fear of damaging the robot and not meeting an unforgiving launch window.

Elsewhere in the article it states that they are concerned that the memory problems may show up in the other rover as well.