This question has been tickling at me for my entire life. Personal computers started becoming readily available as I was growing up - walking and typing were concurrently learned skills for me. Yet, for as long as I can remember, conventional computer wisdom has always told me: "shut down your computer properly, always. It’s bad not to."
Unfortunately, as is common with these kinds of generalizations, there’s no indication as to why. My thoughts are that the reasons are completely software based, in that manually turning my computer off (with the power switch) may cause programs to lose unsaved data or terminate in the middle of delicate processes. I can’t see any real different in the hardware as to the particular method of current cessation - off is off, right? Today I generally try to shut down my PC when possible, but I feel no great remorse when it locks up and I’m forced to do it the old-fashioned way.
So, what’s the straight dope? What does “shutting down” my computer really do? And why should I bother?
It’s mostly software-based. If the computer is unresponsive, then a hard reboot is more or less the only option.
In addition to losing unsaved data in open programs, you risk damaging the filesystem tables on some systems. This is why Windows will run Scandisk when the system was not shutdown properly. Almost always, the tables are recoverable, but there are always exceptions.
The most important thing that it does is properly save all data to disk. Even if you’ve hit the save button, the operating system won’t necessarily put your data on disk right away. For performance reasons, the OS can keep the data in memory for a time. I’ve seen all kinds of data loss, including one case where I completely trashed the file system, from pulling the plug on a system in our test lab at work(fortunately the lab is designed to completely re-install the operating system in a fresh state in 20 minutes or so whenever you ask it to do so, so trashing file systems isn’t a big deal).
To take an extreme example, I had an Ubuntu Linux setup (haven’t used Linux extensively since moving to Japan as I like the IME in Windows better) a few years ago that would only write data to USB-based disks when they were unmounted/ejected. So if I saved some important document (squirrelnuts.txt) onto my USB flash disk, then either killed the power or carelessly yanked out the device, I would lose that ‘saved’ data.
I don’t know if Ubuntu still handles USB devices that way by default or not.
Even if the OS and all applications were specifically designed to survive a fatal abort (which can’t be done 100% reliably but I’m not sure how many even try), Murphy’s Law says that the one with the bug will be the one active when you power down, and will be executing the line of code with the bug.
IIRC thats write behind caching. The file is sent to be saved but is not saved yet but allows the user to move on while the OS deals with it over the next few seconds.
Slaming down the power on things like servers in large multi user environments where there may be a few seconds lag between any given transaction and it being saved to disk, this can be a huge mess.
Most Enterprise File Storage systems claim 100% write reliability - once data is written to the controller by the OS, the manufacturer guarantees integrity. This is a requirement for many sensitive applications, and is achieved using battery backup cache to store writes even if the disks stop spinning due to a powerdown. Many systems claim to retain data on the controller for up to a year. For this reason, write behind caching should not be used on this sort of high level kit.
Back in the day, the company I worked for sold Compaq kit. We would try to sell our customers the advanced battery-backed-up Compaq raid controller, but the smaller ones would go for the cheapest non-backed-up one. What this meant was that in the future, they could not expand their disk array with more disks on the fly. So when this happened (as it invariably did), we would take down an advanced controller, swap it in for the cheap one (all the Compaq Smart Array controllers were driver and disk configuration compatible), extend the array with the new disks, wait for the rebuild to complete, then reinstall the cheap non-expandable controller. Job done. As I recall, HP raid controllers were nowhere near as capable.
In a somewhat related question, is using the /f parameter on shutdown bad? When I shutdown I hate waiting three minutes for my computer to do what I explicitly told it to do when I use the normal shutdown… so I go into the command line and just type “shutdown /f /p”, since /f /p force closes applications with no time-out/warning does it post a similar danger, or is it just telling the computer “finish what you’re doing but be quick about it!”?
The way it works is that when you activate the shutdown command, Windows sends a message to all running applications saying prepare to shutdown. Applications then have the opportunity to flush data from memory, request saves etc. This is why you can get dialog boxes that allow you to cancel the shutdown. Once all applications have responded to this message, the shutdown proper starts, and applications get a FOAD message (which should skip all the prep for shutdown stuff).
Shutdown /f skips this preparatory message, so applications don’t have the chance to do things they may wish to do. This may not be an issue, but data may not be written to disk, open files may not get closed correctly, and data could be corrupted. This may not happen, but it can. It’s not the ideal approach.
My Windows 98se with FAT32 does that. My Windows XP with NTFS does not. I believe NTFS works in such a way that it can always recover because it makes a kist of what is to be done, then does it and then checks the list to confirm. If it is cut off in the middle of doing it then it can always go back to the list and complete the job.
NTFS implements File System Journalling for metadata only. This means that (in general) directory, filename and file extents are protected (ie the structure of the disk can be recovered, no more lost files/directories), the data that should be within the file will not be. And NTFS file systems may be scanned on boot, it is generally less common, but there is an unobtrusive quick file system cleanup done on startup.
So yes, NTFS is better than FAT in that regard, but it is not a fully journalling file system, and any file system objects (at some level) to the power going down while things are happening.
si_blakely is right about NTFS lacking full journalling, but I wonder now if FAT32 had something to do with my old Ubuntu quirk… All of my hard disks on my Ubuntu machine were formatted as ext3 (save for when I experimented with ReiserFS and XFS) but my flash drive, which I often used with Windows and MacOS machines, was formatted with FAT32 (NTFS support for Linux is [or at least was?] flaky at best). If not a quirk of FAT32 itself, (the same FAT32-formatted drive had no problem in WinXP or MacOS X) perhaps it is/was a quirk with the FAT32 support in the Linux kernel…
Now I want to install Ubuntu on my laptop and give this theory a test… but I’ll probably just wait until I can purchase a new computer… Or do a Google search when I’m feeling more Googley.
It was more likely to do with the fact that it was a Flash drive. Flash is relatively slow, and uses wear-levelling to prevent too many writes to a hardware sector. There may have been an option to use aggressive write-behind caching so that multiple file system operations get optimised, and the FAT table does not get rewritten too often.
I’ve seen this behaviour too, actually. I put it down to a Flash/FAT quirk, rather than Linux itself. I think I’ve seen XP do it as well, but I could not swear to it.
Oh, good. That makes much more sense to me now that I think about it and, in fact, was along the lines of what I suspected before I got myself wrapped up in the fs aspect of the ordeal. Thanks.
The fact that the effect is much more noticeable in Linux than WinXP does suggest that there must be at least something about how Linux does things, and it most likely is along the lines of what you suggested with aggressive write-behind caching. I’ve never personally seen experienced an instance where yanking out the flash drive without an unmount caused data loss in WinXP, but in Linux (at least, wrt my installation) you could count on it happening.
I suppose, in the end, the thing we should all take out of this is: Always soft shutdown/reboot properly when possible and unmount filesystems/disengage warp factor 5 before disconnecting storage devices.
To be honest, though, the number of times I have tried to mount a NTFS volume in Linux from a clean shutdown only to have NTFS 3G complain “The File System is marked DIRTY - do you want me to force mount?”
But, NTFG 3G did allow me to read the data off a dying hard disk better than BartPE (Windows XP on a CDROM). There seems to be no way to mount a NTFS volume in windows read-only.
The flash drive issue might have been Linux trying to avoid wearing out the flash. It’s not as big a deal today, but there was a time when the limited number of writes that flash could take was actually a problem.