Erasing disks

I recently upgraded my system from Windows 8 to Windows 10. It worked OK, but after I shutdown my computer and then restarted it the next day, my ramdrive was already running, with all the files in it. That meant that the system had to have saved everything to the hard drive. That included all my userids and passwords. I kept them in an encrypted file on my hard drive and only decrypted them into RAM. So if the system had them in RAM without my putting them there, that meant they had to have been written to a hard disk. At that point I realized two things:
1. Windows isn't remotely secure.
2. I really needed a disk erase utility.

The problem with deleting files from a disk drive is that it doesn't really erase them. That's why file recovery programs work. Sometimes you can just do an undelete. When you delete a file, the operating system makes an entry in the file system saying that the disk space is now available. If nothing has been written to that space, then the file can be recovered.

The real problem is that even if the space has been overwritten, it is sometimes possible to recover the information.

Think of it as writing on a piece of paper and then erasing it. If you look closely, you can see faint marks on the paper. If you overwrite it, and then erase that, you probably wouldn't be able to see the original writing. But if you used a magnifying glass, you might still see it. If you then overwrote it again, and then erased it, you probably wouldn't be able to see it even with a magnifying glass. But if you used a microscope...

The traces that are left on the disk are due to hysteresis. That is the tendency of magnetized material to keep the magnetic charge after the magnetic field that caused it has been removed. Suppose you wrap a wire around a piece of iron and then pass an electric current though the wire. You would have to insulate the wire to keep the electric current from passing thought the iron. This will make the iron become magnetized. If you reverse the current in the wires, the iron magnet will be reversed: north will become south and south will become north. This is how electro magnets work. Electric motors rely on this technique.

A more elaborate explanation of hysteresis is at Georgia State University.

On a computer disk drive, data bits are encoded as magnetic patterns on the drive. These patterns can be changed by applying a different magnetic field to that point. But the strength of the resulting magnetic pattern depends on what it's previous value was. If the result is a '1' bit instead of a '0' bit, it will be stronger if it had been a '1' before and weaker if it had been a '0' before. This is due the the magnetic medium resisting change: i.e. hysteresis. So, by looking at whether the '0's and '1's are a little stronger or weaker than they should be, we can figure out what their previous values were. Overwriting it multiple times will diminish but not necessarily completely eliminate the information.

If hysteresis is causing the problem, why not just get rid of it? Without hysteresis, the value would return to zero after you used it. The problem is that the whole point of a disk drive is to retain information when the magnetic field is no longer being applied. That's what makes it a storage device. To do that, the disk drive has to have strong hysteresis.

A convenient way of looking at this is to think in terms of levels. The top level (level 1) is actual data that can be read by the operating system. If the file is deleted, but not overwritten, it is at level 2. It can't be read by the operating system through the file system (because the link is no longer there), but it can be read by software that bypasses the file system. File recovery software works like that. If the data is overwritten, it drops to level 3. At that level, it can no longer be read even by software that reads the disk directly. However, tools that measure magnetic fields directly, especially sensitive ones such as magnetic force microscopes could get analog data that in turn could be used to derive the original digital data.

There is a big dispute over how practical that would be. Daniel Feenberg at the National Bureau of Economic Research wrote that it would be impractical. On the other hand, a commenter (Frank) at Raymond.cc wrote that the local Sheriff’s Office Cyber Crime unit was able to retrieve data from an overwritten disk, but that multiple overwrites made it unrecoverable.

The only point of agreement was that high density disks are harder to recover data from and therefore are easier to erase.

There is another big dispute over how many and what type of overwritings are needed. Clean Disk Security offers a choice of a single pass, 7 passes, or 35 passes - which it calls Gutmann, named after Peter Gutmann. The 35 passes (using a set of patterns) are based on Gutmann's suggestions. The problem is that Gutmann himself has written that the full set of patterns would not be relevant to any type of disk drive. For modern disk drives a random pattern would be best. If you run a single pass with random data each day, then by the end of a month you would have the equivalent of the full Gutmann!

Another problem is data that has been overwritten by long lasting data. It is at level 3 (it has other data on top of it). The problem is that it will never drop below that, even with a disk erase utility, because that utility only affects free space. To deal with that, we would need a utility that copies data to RAM (or another place on the disk), overwrites the spot with random data, then rewrites the original data back to where it came from. No disk erase utility does that. However, there is a different utility that does: Spinrite. The purpose of Spinrite is to correct problems in a disk drive caused by slightly moved tracks. It does that by reading then rewriting the data at each sector. In the process, it tests the reliability of the sector by repeatedly writing and rereading various types of data there. Although the purpose is to test reliability, the effect is to erase the level 3 data.

A different problem is sectors that have been marked as bad. Anything there will probably stay there forever. Disk erase utilities won't see it because they go through the operating system. Spinrite will read it, but will just put it back the way it was. This is a problem not just with sensitive data, but also with viruses. If a virus can put itself into a sector (or sectors) and have them marked as bad, then anti-virus programs might not even look there. Maybe Spinrite could add a feature to erase anything in sectors marked as bad.

A separate issue with Spinrite is that if a bad sector can be recovered, Spinrite will change its status to good. That might not be a good idea, because if it went bad once, the odds are it will again. An option to disable this would be a good idea.

Another potential problem is magnetic traces between tracks. The physical location of a track can move slightly due to things like a change in temperature or just continued use. This usually is not a major problem, because the read/write head will find it. The problem is that if a sector is rewritten in a slightly different location than the previous data in that sector, then part of the magnetic data on one side won't be completely overwritten. For example, if heat causes the disk platter to expand, then when a sector gets overwritten, it will be written to a place slightly closer to the center of the platter than the previous time that data was written there. That would mean that the part of the previous data that was farther from the center of the platter wouldn't be overwritten. In principle, that could be detected and exploited. Spinrite might help with that. To do that, it would have to bypass the firmware on the disk drive to make the read/write head move to a place not exactly where the track now is. I'm not sure that is possible.

In my case, I found that the RAM data was being kept in a hidden file (Hiberfil.sys) which was a pain to erase (the erase has to be run with administrator privileges). I ran Disk Investigator to look for my main userid. It found 70 instances. That's because a lot of applications will save your userid. Then I ran Disk Investigator to look for my password for my bank account. It found 4 copies. Then I ran Clean Disk Security, followed by Disk Investigator again to look for the userid. It found 63 instances. Finally I ran Disk Investigator again to look for my password. It didn't find any. So at least the disk eraser works.

So, my conclusion is to run a disk eraser with a single pass with random data maybe once a day. That should work, as long as it doesn't wear out your disk drive. Now, I just have to migrate everything to Linux. Maybe WINE will help.

I asked Peter Gutmann for his opinion. Here is his reply:

Hi,

Thanks for contacting me first, unfortunately a lot of people quote a 20-year-
old paper as if it was still current today, when it really applied to
technology that was in use at the time but is completely extinct today. So
your real concern, as you point out, is things like sector sparing where your
data is invisibly remapped to somewhere else on the drive, leaving the
original untouched by attempts to overwrite it. That's not at the OS level,
that hasn't been done for years, but by the drive firmware, and no software-
level overwrite mechanism can even see it's going on, let alone deal with it.

The background erase process sounds like a reasonable compromise, although on
a large drive you're going to spend a huge amount of time on unnecessary
writes. A better option is probably to just transparently encrypt the entire
drive. Peter.

Posted 2018/August/09