How to backup settings and list of installed packages

  • If I want to make a backup of everything I have done since the fresh install of Ubuntu, what are the possible options? What all should I backup? I want to get all the settings that I changed, all the packages I installed, etc.

    No one has mentioned: `sudo dpkg -l > installed_software.txt`!

    @iammilind Probably because `dpkg --get-selections` solutions cover this.

    Be wary of this bug though: https://bugs.launchpad.net/ubuntu/+source/policykit-desktop-privileges/+bug/1240336 I tried the most upvoted solution and never got it working because of that bug. (This might be better suited as a comment, but I don't have enough reputation to post it as a comment)

    be aware also that if you are restoring in a newer version of the OS, some settings may be incomplete or even conflict, having a possibility to make some applications malfunction.

  • Programs

    A quick way of backing up a list of programs is to run this:

    dpkg --get-selections > ~/Package.list
    sudo cp -R /etc/apt/sources.list* ~/
    sudo apt-key exportall > ~/Repo.keys
    

    It will back them up in a format that dpkg can read* for after your reinstall, like this:

    sudo apt-key add ~/Repo.keys
    sudo cp -R ~/sources.list* /etc/apt/
    sudo apt-get update
    sudo apt-get install dselect
    sudo dselect update
    sudo dpkg --set-selections < ~/Package.list
    sudo apt-get dselect-upgrade -y
    

    * You may have to update dpkg's list of available packages or it will just ignore your selections (see this debian bug for more info). You should do this before sudo dpkg --set-selections < ~/Package.list, like this:

    apt-cache dumpavail > ~/temp_avail
    sudo dpkg --merge-avail ~/temp_avail
    rm ~/temp_avail
    

    Settings and Personal Data

    Before you reinstall, you should probably back up the settings from some of your programs, this can easily be done by grabbing folders from /etc and all the content from your user directory (not just the stuff you can see in nautilus!):

    rsync --progress /home/`whoami` /path/to/user/profile/backup/here
    

    After you reinstall, you can restore it with:

    rsync --progress /path/to/user/profile/backup/here /home/`whoami`
    

    So all together as a pseudo-bash script.

    This assumes there is only one user on the machine (remove /'whoami' otherwise) and that you used the same username on both installs (modify dest. of rsync otherwise).

    dpkg --get-selections > ~/Package.list
    sudo cp -R /etc/apt/sources.list* ~/
    sudo apt-key exportall > ~/Repo.keys
    rsync --progress /home/`whoami` /path/to/user/profile/backup/here
    
    ##  Reinstall now
    
    rsync --progress /path/to/user/profile/backup/here /home/`whoami`
    sudo apt-key add ~/Repo.keys
    sudo cp -R ~/sources.list* /etc/apt/
    sudo apt-get update
    sudo apt-get install dselect
    sudo dpkg --set-selections < ~/Package.list
    sudo dselect
    

    The last line of the reinstall script, "sudo dselect", contradicts the last line in the second script, "sudo apt-get dselect-upgrade -y". Also, running "dselect" opens some text-based GUI, where it is not clear how to continue. Is this a typo?

    I ran the reinstall script, and got into an infinite loop: [email protected]:~$ sudo dpkg --set-selections < ~/Package.list [email protected]:~$ sudo apt-get dselect-upgrade -y ... After this operation, 451 MB of additional disk space will be used. Get:1 http://il.archive.ubuntu.com/ubuntu/ precise/main libkeyutils1 i386 1.5.2-2 [7,716 B] Get:2 http://il.archive.ubuntu.com/ubuntu/ precise/main libkeyutils1 i386 1.5.2-2 [7,716 B] ......

    It tries forever to get "libkeyutils1". I tried to remove "libkeyutils1" from Package.list, but it didn't help.

    It is likely a dependency of another package so removing it from the packages.list wont fix the fact that another package will attempt to install it. I recommend you start a new question for this and link back here for context.

    This will not work on ubuntu 12.10. it says ``dpkg: warning: package not in database at line XXX: xrdp`` And there is no package by name "dselect"

    Try doing a `sudo apt-get upgrade` because both of those packages exist in 12.10 as per packages.ubuntu.com. I will add that line into the pseudo-script

    @confiq Check the above three comments

    @Huckle: in 13.04 there is a package dselect but the error is still same... ex: `dpkg: warning: package not in database at line 1484: vlc`

    blah... to solve dpkg warning, read this: http://forums.debian.net/viewtopic.php?f=17&;t=79006#p432478

    Remember to back up and re-add the sources.d directory -- ppa's especially will pile there source info into separate files in there. The re-install will fail without them.

    Will these backup ssh files too?

    ssh files are stored in ~/.ssh so as long as rsync recurses into dot-directories the ssh files will be taken. If it doesn't recurse into directories starting with a dot, there's likely a switch that will make it do so.

    Does this answer miss the sources in `/etc/apt/sources.list.d` though?

    Restored. Was disappointed as all DesktopEnv related settings (launcher, animation, compiz tweaks, desktop background) were not restored. Commoness: All reside in dconf! Solution: Within the GUI, `Log Out` via the gear icon in the main menu bar. Then switch to a CLI via `CTRL-ALT-F1`. There restore .config/dconf/user from your backup. Switch back to the GUI via `CTRL-ALT-F7`, log in again. Voila: All your Unity related settings are in effect again! Explanation attempt: While an active Unity GUI session the dconf file may be locked or ignored, thus you need to switch to a pure CLI mode?

    @porq If you replace the configuration of a program/service that is running, you will need to stop and start that program/service. Dropping to a lower run level or restarting the computer will solve some or all of these sorts of problems.

    I receive errors executing this: see paste here

    It can be also useful to backup the whole `/etc` directory.

    **note:** `dpkg: warning: package not in database` errors are likely caused due insufficient file permissions - don't forget to `chmod +r /etc/apt/sources.list* -R` after copying them to the target machine!

    at `apt-key add` I am getting "gpg: no valid OpenPGP data found."

    Sorry but there is is an error when you save the apt-key : When saving them you should use $ sudo apt-key exportall > ~/Repo.keys definitely not apt-key add which is used for restoring ...

    I suggest to do `sudo apt-key update` after `sudo apt-key add ~/Repo.keys`.

    To get the rsync command to work, I had to add a recursive flag (`-r`).

    need to add -r (recursive )at before the path to copy otherwise id does not copy anything. At best -raz or -rPz, check what them mean.

  • Who is this for: users that have normal regular use of their computer, that have done minimal or no configuration outside their home folder, did not mess up startup scripts and services. A user that wants to have his software restored to how it was when he installed it with all customizations being done and kept in their home folder.

    Who this will not fit for: servers geeks, power users with software installed by source (restoring the package list might break your system), users that have changed the startup script of some application to fit better their needs. Caution: there is a big chance any modifications outside home will be over written.

    Backup your current packages and user settings


    Once you are ready with your system and happy with the software installed you can get a list of the installed packages using the command dpkg --get-selections and save the output in to a log file.

    Create a backup folder in your home

    sudo mkdir ~/backup
    

    Save the currently installed packages list

    dpkg --get-selections > ~/backup/installed_packages.log
    

    Make a backup of your apt sources file

    sudo cp /etc/apt/sources.list ~/backup/sources.bak
    

    and a copy of your apt's list of trusted keys

    sudo apt-key exportall > ~/backup/repositories.keys
    

    Make a backup of your home folder with the integrated backup tool in Ubuntu, Deja-dup.

    • Set the backup folder

    enter image description here

    • Verify that it will make a backup of your home folder and add any folders your might not want, like for instance the Trash folder

    enter image description here

    • On the overview tab press Make Backup Now to backup your home folder

    enter image description here

    • The backup tool will ask if you want to add a password to your backup, your choice. Your backup will start after you press Continue

    enter image description here

    When this is done you will have a backup of your packages and configuration files relative to your user. If when necessary you will be able to restore your files from the ~/backup folder.

    Restoring your backup


    Start by restoring the sources file from the backup made

    sudo cp ~/backup/sources.bak /etc/apt/sources.list
    

    The backed-up keys

    sudo apt-key add ~/backup/repositories.keys
    

    Update your sources lists

    sudo apt-get update
    

    Restore the packages from the saved installed_packages.log

    sudo dpkg --clear-selections
    sudo dpkg --set-selections < ~/backup/installed_packages.log && sudo apt-get dselect-upgrade
    

    sudo dpkg --clear-selections will mark all current packages installed for removal, that way when you restore your saved package list the packages that are not on the list will be removed from your system.

    Remove your current configuration from your home creating a backup of the folder in their current state (after all, whats the use of restoring fresh files if there are other there that can affect the configuration?)

    mkdir ~/.old-gnome-config/ && mv ~/.gnome* ~/.old-gnome-config/ && mv ~/.gconf* ~/.old-gnome-config/ && mv ~/.metacity ~/.old-gnome-config/ && mv ~/.cache ~/.old-gnome-config/ && mv ~/.dbus ~/.old-gnome-config/ && mv ~/.dmrc ~/.old-gnome-config/ && mv ~/.mission-control ~/.old-gnome-config/ && mv ~/.thumbnails ~/.old-gnome-config/   && ~/.config/dconf/* ~/.old-gnome-config/
    

    After this is done restore the backup created with Deja-dup.

    • Open the backup tool and on the overview tab press restore

    enter image description here

    • Follow the prompts to restore the files to their original positions

    enter image description here

    After this is done you will have your packages back to the saved selection, our configuration restored and hopefully a working desktop, all of that without installing a single extra application and using the Ubuntu default tools.

    The only thing left is to do some clean up and check that everything is working.

    Something went wrong, my desktop is gone


    There are a few posts in Ask Ubuntu that can guide you in case something when wrong and you need to hard reset your desktop. If something when wrong and you find your self needing to do so, please have a look at these posts:

  • Backups take some planning and there are several viable strategies. You will have to decide which method works best for you.

    Be careful about "best way" , what works best for one person may not be best for another.

    At the end of the day, the "best method" is one that has been tested and known to work. You need to test your backup strategy BEFORE you need it

    Using images

    One way is to simply copy and compress an image of your partitions. You can do this with several tools, anything from dd to partimage to clonezilla

    partimage
    clonezilla

    The advantage of this strategy is that it is (relatively) easy and very complete. The disadvantage is that the back up images are large.

    Smaller backups

    You can make smaller backups by only backing up data and settings. There are several tools to do this, everything from dd to tar to rsync.

    The key here is to know what you need to back up.

    Advantage - Backups will be smaller.
    Disadvantage - This is a manual method, so it is easy to forget about a critical file (/etc/passwd).

    A list of what to back will always need to be reviewed to make sure the list is complete.

    1) List of installed packages

    Package list # Create a text list of an existing installation of all apt-get installed packages # in order to re-install on a newly installed distro

    # make the list
    [old distro] sudo dpkg --get-selections > packages
    

    You would then perform a fresh install, and restore your packages. The following commands also update all the packages on your system (so restore and full update all at once).

    # Now put them back on the new distro
    [new distro] sudo dpkg --set-selections < packages
    
    [new distro] sudo apt-get dselect-upgrade
    

    2) Data. Generally this would be /home . Most user data and customization's are going to be in your users home directory. If you save data in other locations, include that (for example /media/data).

    3) system settings . Here is where there is going to be some variation. Personally if I edit ANY system setting, I keep a copy of the original configuration file and my custom file in /root. So if I edit /etc/fstab for example, keep a copy in /root/etc/fstab and original in /root/etc/fstab.orig

    You will also need /etc/passwd, /etc/shadow , /etc/group , /etc/sudoers, /etc/hostname, and /etc/hosts (you may need more in /etc, if I forgot something I will add it)

    On a server you may need to include /var/www or other data directories.

    I understand this takes some effort, so, it may be easier to include all of /etc.

    4) A copy of your disk partition table.

    sudo fdisk -l > fdisk.bak
    

    You can use this information to restore your partition table if you replace your hard drive.

    5) A copy of your MBR

    sudo dd if=/dev/sda of=MBR.bak bs=512 count=1
    

    You would then restore with

    sudo dd if=MBR.bak of=/dev/sda bs=512 count=1
    

    6) Other files/directories - Depending on your system and customization you may need to include additional directories. Considerations might include /opt , /usr/local, /usr/share`, and/or your .desktop files. Anything on your system you downloaded or customized outside of apt-get or software manager.

    7) Put all that into an archive

    sudo tar -cvpzf backup.tar.gz /home /root /etc ./MBR.bak ./fsdisk.bak ./packages
    

    8) rsync

    As an alternate to tar, you can use rsync.

    See - https://help.ubuntu.com/community/rsync

    Network backups

    Another option is to use NFS or Samba to back up data.

    Cron

    You can automate backups by writing a backup script and running it (daily / hourly) with cron.

    Restore

    1) Using a live CD, restore your partitions using gparted or fdisk from the information in fdisk.bak
    2) Install Ubuntu.
    3) Boot your new install, update your packages.
    4) Restore your data and settings

    tar -xvpzf /home/test/backup.tar.gz -C / 
    

    Then reboot

    See also - https://help.ubuntu.com/community/BackupYourSystem/TAR

    Other Strategies

    There are many additional strategies for backup, some for backing up your home directory only, some graphical.

    see: https://help.ubuntu.com/community/BackupYourSystem

    Testing

    It is not a backup unless it is tested. This is most important when you are NOT using an image of your root partition.

    Test restoring your system in a VM, a spare computer, or a spare partition or hard drive.

    Great answer! One addition, from PartImage website: PartImage does not support Ext4 partitions, and Ubuntu defaults to the Ext4 filesystem format. I'm told PartClone (which does support Ext4) is a good alternative. You might also want to look into CloneZilla.

  • Installed Packages Selection using apt-clone

    APT-Clone. This package can be used to clone/restore the packages on a apt based system. It will save/restore the packages, sources.list, keyring and automatic-installed states. It can also save/restore no longer downloadable packages using dpkg-repack.

    APT-Clone is used by ubiquity (Ubuntu installer) for upgrade process.

    1. Install

      sudo apt-get install apt-clone
      
    2. Make backup

      sudo apt-clone clone path-to/apt-clone-state-ubuntu-$(lsb_release -sr)-$(date +%F).tar.gz
      
    3. Restore backup

      sudo apt-clone restore path-to/apt-clone-state-ubuntu.tar.gz
      

      Restore to newer release:

      sudo apt-clone restore-new-distro path-to/apt-clone-state-ubuntu.tar.gz $(lsb_release -sc)
      

    Reference: man apt-clone


    Home Data, Configuration /etc,.. using duplicity

    Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

    1. Install

      sudo apt-get install duplicity
      
    2. Backup

      duplicity full path-to/source_folder/ file:///path_to/duplicity_backups/
      

      or incremental backup (It will backup only the difference from last backup):

      duplicity incremental path-to/source_folder/ file:///path_to/duplicity_backups/
      
    3. Restore

      duplicity restore file:///path_to/duplicity_backups/ path-to/target_folder/
      

    Reference: man duplicity

    APT-Clone is much better than the `dpkg --get-selections` solution because: 1. it preserves all repositories information 2. it keeps track of what packages were automatically installed 3. it allows to repack locally installed DEB files. This should be the accepted answer!

    This is great! I'm hoping these work with 16.04, it's exactly what I'm looking for. But I'm wondering, will Apt-Clone also remove packages that aren't in the restored configuration?

    @XToro, No it it will not remove. (btw, if it was removing packages by default it will damages the upgrade, because most of upgrades have new packages installed and some dropped)

    @Sneetsher That's a shame. I would actually like if it removed all packages not on the list and then install the ones listed along with its dependencies. My issue with why I want to use it is to remove unwanted packages after playing around with things like installing new DE's. There's always tons of leftovers after removing the DE.

    @XToro, I like that, I needed that before but I went around by testing in virtual boxes. You need a modified script from Huckle's answer. It is better to ask new question and mention the issue with this one. If you are unable or don't want for some reasons, I will do it?

  • Although this question has been answered for a while, I noticed that nobody mentioned etckeeper. Run apt-get install etckeeper, configure your VCS in /etc/etckeeper/etckeeper.conf, and you will from then on have much better control of your /etc directory. This can also be used to restore your configurations within the /etc directory.

    For instance, (assuming you configured git as your VCS) back up your /etc/.git directory. Any time you need to restore your configurations onto a fresh /etc, you copy your backed-up /etc/.git directory into the fresh /etc directory. You now have many options for managing your restoration:

    • View differences
      • git diff
    • Keep all differences in the "fresh" directory in a git branch
      • git checkout -b new; git add -A; git commit -m 'new etc'
    • Unconditionally revert to your backup
      • git clean -f
  • For those wanting a nice. neat GUI...

    ...introducing Aptik.

    All you need is a backup directory, stored locally or in the cloud. Aptik will backup PPAs, downloaded packages, software selections, application settings and themes and icons. Very useful.

    You can install it through the ppa:

    sudo apt-add-repository –y ppa:teejee2008/ppa
    sudo apt-get update
    sudo apt-get install aptik
    

    Hope this helps out :)

    `sudo apt-add-repository –y ppa:teejee2008/ppa` should be `sudo apt-add-repository ppa:teejee2008/ppa`. Only 1 argument accepted.

    This is the most REALISTIC option to someone that wants to do a selectable, system-wide and user backup. Btw, the current version supports much more settings than what is shown on that screenshot.

    How to run aptik as gui? I've just installed it with commands provided and it appears to run as a console utility, non-available in Show Applications menu. You may fix aptik repo in your post, btw.

    in Ubuntu, install aptik-gtk for the GUI front end

  • To create a backup list of all your installed programs: http://savvyadmin.com/backup-and-restore-package-lists-in-ubuntu/
    You will obviously need to backup your software sources that some of those installed packages are from: Backup Software Sources

    As for your program settings, most of those are in hidden (start with a .) folders and files in your home folder. I would just backup all of them.

    And backup /etc directory also

  • If storage space isn't a concern, using dd or dc3dd will backup absolutely everything on the target disk or partition you designate. You don't want to try this on a mounted drive, so you'd do this from a bootable USB or DVD (or your install media choosing "Try Ubuntu"):

    sudo dd if=/dev/<source> of=/path/<target>.img
    

    Where is the disk or partition that you wish to backup and target is the backup filename (often the same) (sda, sda1).

    Substitute dc3dd for dd if you want a progress report. To access specific files in the backup you can typically mount the resulting .img file as a loop device by:

    mount -o loop,ro,offset=32256 filename.img /mnt/dir This works on drives with a single partition where you have backed up the entire drive.

    Or mount -o loop,ro filename.img /mnt/dir if you backed up a partition.

    You can restore the entire disk or partition by swapping the if= and of= parameters as in:

    sudo dd if=/path/<source>.img of=/dev/<target>

    Where is the image file that you wish to restore and <target> is the drive or partition you wish to restore it to.

    PROS: Easy to backup and easy to restore everything. Unlike some other solutions doesn't really rquire much in the way of planning since you are backing up everything you don't have to worry about whether you will need it or not.

    CONS: Time consuming (computer time, not yours) and not suitable for daily backup (no incremental options)

    Source: experience; I use this approach to backup client systems prior to beginning work and have never lost a bit of client data.

  • I want to have ready-to-restore apps and their data, also offline.

    That defeats the purpose of a "format-and-install". If you want to do that, just do an upgrade from your existing Ubuntu installation.

    Since "installation" in Ubuntu is as simply as going into the Software Center (or Synaptic or apt-get), and most programs are themselves updated frequently, there's no real point in backing up the programs, reinstalling Ubuntu and then installing from the backups instead of just getting the latest versions from the repositories.

    As long as your data and program settings are backed up, you should be fine. If you still want a comprehensive list of everything in your system so you know if a package is missing, just go into the terminal and type dpkg --list > mypackages.txt, and put that text file somewhere safe -- it contains every package ever installed on your current system.

  • I'll assume it's a new HDD, SSD or new system or you had on an old Ubuntu. Now you want to upgrade to Ubuntu 12.04 (Precise Pangolin)?

    I'll assume you detest installing them all again or reconfiguring? No need to.

    Open the terminal by pressing Ctrl + Alt + T, and run the following command:

    sudo dpkg --get-selections > app-backup-list.txt
    

    Then cp the text file to your home directory by:

    sudo cp app-backup-list.txt /home/username
    

    Next run the backup application from the systems settings in the side bar.

    Backup /home to Ubuntu One, a flash stick, a floppy (oops scratch that) or burn onto a CD, or other HDD in system or external HDD drive or use whatever backup program you want.

    Then reverse the backup using restore, pick in the backup program you used, and now \home is back.

    Lastly, reverse the dkpg command:

    sudo dpkg --get-selections < app-backup-list.txt
    sudo apt-get -y update
    sudo apt-get dselect-upgrade
    

    Note:

    Restoring home, gets all settings for your applications. I installed a SSD, and copied all the files, but I still did a backup just in case.

    PS: This avoids, actually backing up, massive apps' machine code, etc. Make sure your download folder is clean. Not full of machine code or videos, and then burn the download folder to a DVD using k3B.

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM