How to delete file(s) in secure manner?

  • Is there a way to make sure that a deleted file can not be recovered?

    As you know, deleting a file by pressing shift-del or using trash doesn't mean that file is gone forever. It resides somewhere in the computer.

    In day to day life, law enforcement or thieves use recovery programs to reveal our private data or something else.

    They can recover all data that resides on hard disk, in RAM, and even USB.

    To protect us from these people, what should one do? Which program should one use?

    Note: I don't want an answer like first use a recovery program then erase that file with shred. With just one program, I want to erase all unused/deleted files on my computer.

    Unfortunately bounty was awarded automatically even though no answer seems to apply to the age of SSD very tightly. With SSD bytes aren't really overwritten (unless you go for entire drive procedures) unless a drive manufacturer specific API call is made, if at all it can. See the article quoted in one of the answers.

  • Takkat

    Takkat Correct answer

    9 years ago

    Shred

    This command line tool is already installed from the core utilities in Ubuntu to securely erase and overwrite single files using the Gutman method.

    Fast shredding

    shred -vzn 0 /dev/sdc1
    

    erases whole partitions by overwriting everything with 0s in a single iteration. If no legal aspects require another procedure, doing so is most probably safe to securely delete your private data Craig Wright Lecture Notes in Computer Science, 2008, 5352, 243-257.

    Secure shredding

    shred -vzn 3 /dev/sdc1
    

    erases the whole partition using 3 iterations with random numbers. In addition (option -z) this writes zeros to hide the shredding process at the end. This will take 4 times longer than the fast method.

    NOTE: By shredding a partition we will overwrite this partition with 0 or random numbers. It therefore efficiently deletes everything including file system caches on that partition forever. This can also be used to remove unwanted remnants of deleted files. Files we want to keep will have to be backed up before shredding.


    Wipe Install wipe

    More options, and the possibility of erasing directories in addition to single files, are offered by this command line utility.

    wipe filename
    wipe -r dirname
    

    Additional notes on journaling file systems and SSDs:

    • Please read the notes in the linked manpages on security issues arising from still recoverable backups in journaling file systems when erasing single files. Overwriting whole partitions rather than single files will effectively erase all data even when using a journaling file system.

    • Erasing data on a solid state disk (SSD) can if at all only be done by overwriting the whole drive (not only single partitions) with several iterations. Some SSDs may have an inbuilt feature to erase data but this may not always be efficient (see this link from comment). At present there is no general recommendation on the wiping processes or number of erase iterations needed to securely remove all data remnants on all SSDs available.

    These options can be added in the context menu of Nautilus and Thunar.

    • In Thunar, open "Edit" then "Configure Custom Actions"

    Add (the plus sign)

    Name "Shred File"

    Description whatever you like

    Action "shred -u %f"

    Similarly for wipe.

    Select "Appearance Conditions" and select "Other Files"

    It is probably also worth noting that if you're using a solid state drive, it also acts as a simple log structured file system and may not overwrite the data.

    "we are able to erase whole partitions", Can you explain more this sentence. I understand it delete all data reside in partitions. If it is true, so how one can protect its useful data which is not deleted, and part of the filesystem. In other words, Is it equal formatting a partion? or Is it only erase deleted files ?

    @shred, "about 20 minutes to erase a 1.44MB". About whole life to erase 640 gb :) . For wipe, "... we cannot guarantee that wipe will actually erase data, or that wiped data is not recoverable by advanced means." In other words, wipe has no real/full power on memory.

    @fatai: I tested shred for speed: here it was approx. 3 min for a 1 GB partition with 1 iteration.

    Note that, while you can use shred on individual files, as well, if you use a modern journaling filesystem, there's no guarantee that shred will result in unrecoverable deletion. You'd have to blow away the entire partition. I think the US government standard is 7 passes with random data.

    One comment regarding SSD: Overwriting the **whole** drive will not really help. In short: You cannot reliably remove data from flash memory. Always store confidential files encrypted. (cf. Wei et al., Reliably Erasing Data From Flash-Based Solid State Drives: http://www.usenix.org/events/fast11/tech/full_papers/Wei.pdf)

    Thank you @freddyb for this very interesting paper! I edited my answer in respect to this.

    @Takkat how can one use shred with folder(s) ?

    @fatai: shred is for single files or partitions only. If you want to delete folders (=directories) use wipe or another tool. Mind that this may not be secure enough.

    Still, @Takkat, I wouldn't even state that deletion on SSD is possible at all. Please suggest a cryptographic solution instead! Some SSD drives come with a erase software switch, that should probably work. Internals are encrypting all content transparently to allow deletion by changing the encryption key.

    @freddyb: I am aware of the fact that at present it is not clear how to securely erase data from a SSD. Encryption is always better - this includes conventional drives as well! The question however was on how to securely **erase** data (implying they may not have been encrypted before).

    Adding a Custom Action for `shred` in Thunar is nice, but I'm loath do so because it never requires confirmation (and I could mistakenly use it to shred pretty much any file). Can a confirmation be configured here?

    @landroni: good point... this was included to my answer by an edit. You should indeed embed the `shred` commmand in a script asking for confirmation (e.g. with a Zenity dialog). Or call it via `pkexec` for sudo permissions after password confirmation.

    Thanks for the `shred` tip. This tool is a must-have if you are sure that on a large `ext*` partition (with 70% free space), there got to be still be some important data on the partition when it was formatted with NTFS and used in Windows. Wiping the *whole* partition would not make any sense here, as you will have to leave the free space intact and untouched! However, __`shred` does not recurse into subdirectories__ yet. Well, so be it: `find . -type f -exec shred -uvn 2 * {} \;` and VOILA! *(Warning: with lots and lots of tiny files, this may take ages - but wth, it works.)*

    from the quoted article from @freddyb: `Since in-place updates are not possible in SSDs, the overwrite-based erasure techniques that work well for hard drives may not work properly for SSDs. Those techniques assume that overwriting a portion of the LBA space results in overwriting the same physical media that stored the original data. Overwriting data on an SSD results in logical sanitization (i.e., the data is not retrievable via the SATA or SCSI interface) but not digital sanitization.`

    I like `shred -vzun 3 filename` much better. it first writes zeros, renames the file with zeros and then deletes.

    I bled with `shred`. . . Just `shred`ed it up, Thanks!!!

  • There isn't one command that you can run which will easily clean up all the already-deleted files for you. However, there are a number of things you can do to reduce your vulnerability to this sort of attack in future.

    As others have said, using tools like shred or srm allows you to delete a specific file by actually overwriting it, rather than just removing it from the filesystem. If you're feeling bold, you can replace the rm command with shred or srm to securely delete files going forward. That means that whenever you (or another program) tries to delete something using rm, the secure delete command will run instead.

    However, if you're using a solid state disk, or even some newer mechanical disks, shred and other overwriting-based methods may not be effective, since the disk may not actually write where you think it's writing (source).


    Full-Disk Encryption

    A more convenient option is full-disk encryption. If you use the alternate installer, Ubuntu can automatically set up a fully-encrypted disk for you you, but you can also customize and configure the settings yourself. Once installed, the encryption is almost invisible to you: after you enter the passphrase (be sure to pick a good, long one) when the computer starts up, everything looks and feels just like normal Ubuntu.

    You can also encrypt external media like USB drives using Ubuntu's Disk Utility. Setting up an encrypted external disk is as simple as checking the "encrypt underlying filesystem" box when formatting the disk. You can even store the passphrase on your (encrypted) keyring, so that you don't need to enter the phrase every time you plug that disk into your computer.

    If your whole disk -- and all your removable media -- is encrypted, there's much less to worry about. A thief or police officer would need to swipe your computer while it's on, (or within a minute or two of turning it off if they're very good) in order to access your data. If you hibernate (rather than suspend) your computer when it's not in use, then you should be pretty safe.

    If you ever need to completely destroy all your data, you don't need to do a Gutmann wipe of your whole disk. Simply overwrite the very beginning of the disk, to destroy the headers for the encrypted volume. Unlike with a regular filesystem, this will actually make it impossible to recover the data.


    So, how do you go from your current setup to a safely encrypted disk? It's quite a challenge to retrofit a currently-installed operating system to use an encrypted disk. The easiest approach is to backup all your data and settings, then reinstall with an encrypted disk. When backing up, make sure to back up your data to an encrypted external drive, but don't save the passphrase in your keyring.

    After you've backed everything up, you may want to aggressively wipe your hard drive, to make sure that none of your existing data can be recovered in the future. If you're using an SSD, the process is even more challenging, so depending how much you want to invest in the process, it might be worth destroying your current disk (a challenging proposition) and starting with a new one.

    When reinstalling the OS, if you haven't aggressively wiped the disk already, you should make sure to completely fill the new encrypted partition, which will overwrite all your old data. Once you've restored your backup, you may want to aggressively wipe the start of the backup disk, to destroy the encryption header, so that it can't be recovered again.

    Can you explain the part about 'Simply overwrite the very beginning of the disk, to destroy the headers...'? I've always just discarded or returned encrypted drives when they failed, assuming nobody could recover the data. Is this a bad assumption? This is with 64 character random hex passwords, so nobody is cracking them.

    The slides (pdf) you linked to lead to the conclusion that the only way to reliably shred an SSD is to literally (physically) shred it.

  • Update: If you have not yet deleted the file that you want to be non-recoverable, use the accepted answer. If, however, you already deleted the file[s], then this is the next best method that I know of.

    If I read you right, you want to erase all your previously deleted files. Here is a simple way to do that:

    $ dd if=/dev/zero of=/path/to/mounted/partition/tmp_file bs=1M count=999999999999
    

    Let that run till it complains till it gets a disk write error [out of space]. Then delete the file! What this does is just fill up your empty disk with 000s, so all your previous files get overwritten. Make sure to delete the file now, or you will not have any disk left. You might want to do this a few times if you are really paranoid. Or if you want to write random to your disk, I'd suggest replace /dev/zero with /dev/urandom.

    However, this will take much much longer, so I'd run it overnight. Also, if you want a progress meter, do this instead:

    $ free=$( df {PARTITION OR MOUNTPOINT}  |awk '{print $3}'|tail -1 )
    $ dd if=/dev/zero bs=1M count=999999999999 | pv -s "$free" > /path/to/mounted/partition/tmp_file
    

    First you are getting your free disk space with du, awk, and tail, then using dd to get /dev/zero, piping that to pv which stands for "pipe viewer", that redirects everything to tmp_file Once again, delete the file afterwards. $ rm tmp_file

    Anyway, hope someone finds this useful! :)

    Is there a command-line utility that does this for you? Say, `overwrite_empty_space /dev/sdb1`?

    @landroni, I am not aware of such a utility. You could easily put this into a BASH script, however. [if you did, I recommend instead of using "/dev/sdb1" you just give it a path to the big file. Wherever it's mounted, that's what disk it will clean] This takes a little effort, so I will not try to go into detail here. I'm sure you can find the info by searching.

    Why not shred the tmp_file after the disk is filled up, if you really want to be sure that the file cannot be recovered?

  • First, delete files with rm on the terminal or with shift-delete on nautilus. Better yet, use srm, from the Secure-Delete tools package.

    You can install the secure delete tools like this:

    apt-get install secure-delete
    

    Second, use sfill to wipe out unused space on your disk:

    sudo sfill -f <directory>
    

    This will take some time, since it is using crypto techniques to fill out the disk.

    Warning: this is going through your whole filesystem, and you are root. Use with care!

    This will create a file which wipes out all previously existing data. The file will be created in <directory>.

    this program spends too much time. For 80 gb, I have waited approximately 18 hours. Moreover, there is no tool showing progress.

    I forget to add ; 18 hours for just filling disk with /000.0 files

    but it's secure and you can delete a file or directory.

    I question whether this is really secure. On modern filesystems (ext2/3/4, ntfs, etc), there is a journal that helps prevent accidental data loss. So, even if you wipe the free space, you can't guarantee that the file is removed from the journal. Also, I understand that some filesystems work at high enough a level that it's difficult to guarantee that you're actually overwriting the physical disk sectors that were written to. AFAIK, the only way to securely wipe something is to wipe the entire partition and all files on it.

    It should be possible to overwrite free clusters and slack space, even in a journaling filesystem. Although for slack-space you might want to clear it on use by a new file. Anyhow, you'd have to flush the journal at start, and possibly pause normal filesystem operations for the duration.

  • A GUI program for this is BleachBit (it's also in the sources). More recent (deb file) version at the BleachBit Sourceforge page.

    Beyond simply deleting files, it includes advanced features such as shredding files to prevent recovery, wiping free disk space to hide traces of files deleted by other applications.

    "BleachBit quickly frees disk space and tirelessly guards your privacy. Free cache, delete cookies, clear Internet history, shred temporary files, delete logs, and discard junk you didn't know was there. Designed for Linux and Windows systems, it wipes clean 90 applications including Firefox (vacuuming to make it faster), Internet Explorer, Adobe Flash, Google Chrome, Opera, Safari and more." (from http://bleachbit.sourceforge.net/)

    BleachBit has several advanced cleaners:

    • Clear the memory and swap on Linux
    • Delete broken shortcuts on Linux
    • Delete the Firefox URL history without deleting the whole file—with optional shredding
    • Find widely-scatted junk such as Thumbs.db and .DS_Store files.
    • Delete the OpenOffice.org recent documents list without deleting the whole Common.xcu file
    • Overwrite free disk space to hide previously files
    • Vacuum Firefox, Google Chrome, Liferea, Thunderbird, and Yum databases: shrink files without removing data to save space and improve speed
    • Surgically remove private information from .ini and JSON configuration files with deleting the whole file

    For details on how to use Bleachbit to delete securely specific files and folders see this answer.

  • If you want a solution which can't be thwarted by a pathologically obsessed individual then you should consider some combination (limited by monetary resources) of:

    • degaussing -- magnetically erasing the hard drive
    • physically disabling the hard drive -- i.e. industrial hole punch
    • acid bath <-- you can't get bits from a pile of goo.

    These solutions range drastically in cost & ingenuity. A few:

    Fun answer, but I think the asker would like to be able to keep using his hard drive after his deletions.

    where's a will there's a way: why not an atomic bomb?

    OP wants data destroyed - not his/her country! :P

    I find a 15lb sledgehammer quite effective...and therapeutic.

    Thermite is another alternative for a thorough obliviation (article in German but includes cool pictures of the process).

  • For SSDs and other flash storage media

    SSDs and many other flash storage media use a technique called “wear levelling” that reassigns unused blocks of storage cells based on their number of previous write cycles to prolong the drive’s life time. As a consequence, overwriting block ranges doesn’t work as a method to erase storage content securely (or even efficiently) like it does for hard disk drives. On the other hand, flash storage allows much faster and efficient secure erasure of blocks, block ranges, or entire drives.

    Erase the entire drive

    You should use the drive's security erase feature.

    1. Make sure the drive security supports secure erause1 and is not “frozen”. If it is, it may help to suspend and resume the computer.

      $ sudo hdparm -I /dev/sdX | grep frozen
             not     frozen 
      

      The (filtered) command output means that this drive supports secure erasure, is ”not frozen” and you can continue.

    2. Set a User Password (Eins in this example). This password is cleared too, the exact choice does not matter.

      sudo hdparm --user-master u --security-set-pass Eins /dev/sdX
      
    3. Issue the ATA Secure Erase command.

      sudo hdparm --user-master u --security-erase Eins /dev/sdX
      

    See the ATA Secure Erase article in the Linux kernel wiki for complete instructions including troubleshooting.

    (source)

    If the command output in step 1 is empty the drive does not support secure erasure but may still support the TRIM command required for the section below.

    Erase a drive partition

    The blkdiscard(8)2 can erase block devices securely if the drive supports it1:

    sudo blkdiscard --secure /dev/sdXN
    

    where /dev/sdXN is the path to the block device node referring to the drive or partition to erase.


    1 If your drive is inside an external casing with a USB or FireWire connection the translation chipset may block support of some optional features like secure erasure and TRIM even if the enclosed drive supports it.

    2 Available and installed by default since Ubuntu Xenial Xerus (16.04).

  • I would say the the solution is a combination of several answers given here. For already-deleted files and partitions still in use, I agree with Matt.

    Then, for the future I would suggest to start using secure-delete tools instead of simple 'rm'.

    Last, when it will be possible to reformat the partition, the option of encrypt the partition should be considered. Perhaps using some approach that offers Plausible deniability like in Truecrypt

  • I use truecrypt file image for such essential data. I't handy, free, crossplatform and I don't need a full disk nor using any extra software to "really delete" the file.

    Just make sure you have strong password and make make backup of the image file. I keep backup in a cloud.

    WARNING: Using TrueCrypt is not secure as it may contain unfixed security issues The development of TrueCrypt was ended in 5/2014 after Microsoft terminated support of Windows XP.

    There are various TrueCrypt derivatives that are still being maintained eg. VeraCrypt. As an aside, there was no link between Microsoft discontinuing XP and TrueCrypt development being stopped - the circumstances surrounding the latter still remain a mystery to the best of my knowledge.

    1. The first option is shred. The previous answer in regards to shred is lacking some needed details. You will not overcome the file system caching, snapshots and journaling, etc. if you run shred on the partition (as in the example of sdc5), and data will remain, lots of it potentially.

      For shred to be effective, especially on modern SSD's etc, you must run it on the device not the partition. This would be /dev/sdc ... without a number on the end (partition number).

      You will need to do this from a live USB if the drive you wish to clean is the primary device for your computer. In this case, you will be best to add persistence to the live USB when you create it so that you can download a few extra tools to do the job properly also.

      Run the command: sudo shred -vfxz /dev/sdc or sda or whatever the device name is. If you are unsure of name, open Disks or GParted to identify your device, or run the command- lsblk in the terminal and identify the name of your device there.

      Adding the letters f and x to the command is important as f= force permissions and x=exact size of files. Without these you may miss data that required permission to read or write, or miss data when the file is rounded up to the block size.

      You can add a desired number of write passes by adding -n (number you desire). Example: sudo shred -vfxz -n 5 /dev/sdc

      This is the first port of call to wipe your drive. It may help to run similar tools such as shrub, secure-delete and nwipe. They all work in the same way with very similar commands and slightly different methods and patterns. Go to the Linux man page or Google them to quickly become familiar with how to use the commands. It is not likely you will need more than the standard 4 shred passes, but if it is important you might as well go the extra mile as it could possibly help.

    2. Your RAM may well have data on it. After installing secure-delete, run the command for sdmem. This will wipe your RAM memory. However the first data you ever filled the RAM with, plus any data that stayed in RAM for prolonged periods, may well have left a trace.

      When buying new RAM(or HD's/SSD's for that matter), it is best to fill the drive to completion a few times. You could use shred, but dd is probably better here. Command:

      sudo dd if=/dev/urandom of=/dev/ # (insert your device partion name)
      

      It is also best with RAM to repeat this procedure before using sensitive data if you want to increase your security, and wipe sensitive data from RAM as quickly as possible afterwards. It is this time left in place that is mostly responsible for its ability to leave a trace.

      The only thing left to do is create a partition or install your next distro.

    3. Encryption - People often state this as a reliable method, but if you are going to continue to use your hard drive or pass it on to someone else, this is not an effective option. With regard to legal trouble, failing to unlock an encrypted device is often not an option and can be presumed guilt or an actual offense in many places. It may stop a thief, etc. though (stealing data only lol).

      Also note, an encrypted home folder is very different and protects nothing from physically being examined on your computer, it is for online/system safety essentially and can be circumvented.

    4. If disposing of the device - After wiping with shred, then encrypting with a long password that contains special characters like * etc. to break up the letters and numbers of your password, use shred again but you can just specify the first 10gig of the drive (depends on size, but this is a large safe number). Example: sudo shred -vfxzs10G /dev/sdc This is quicker and just as effective as shredding the whole drive in these circumstances.

      Then to be sure, take the hard drive and RAM out, the RAM is easy to snap and destroy, SSDs are too. You can get creative and release that paranoid energy while breaking them.

    Welcome to Ask Ubuntu! **:-)** The OP is asking about shredding a file, not a device...

    Oops lol. It was badly in need of that edit from karel too. Thanks

    @Fabby -- the OP asked about shredding all deleted files, not just one. in this case shredding the device is appropriate. ie shredding free space.

    +1 for addressing why encryption is not a solution

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM