Busy Device on Umount

  • I often experience a problem to umount a directory:

    umount /mnt/dir
    umount: /mnt/dir: device is busy
    

    There are many reasons why the device is busy. Sometimes there are processes running which have open locks on it, sometimes there are other directories mounted on top of /mnt/dir.

    My question:

    What are the steps to check why a directory couldn't be unmounted.

    I know there are many reasons, but it's ok if you explain a specific solution.

    [EDIT]

    [X] running processes on mounted volumes.
    [X] another volume is mounted on top of a volume we want to unmount
    [_] NFS locks the volume we want to unmount

  • The way to check is fuser -vm /mnt/dir, which must be run as root. It will tell you which processes are accessing the mount point.

    An alternative is lsof /mnt/dir, which will show each open file on the mount. Again best run as root.

    You can run either of these as non-root, but then the output will be limited to your processes—ones from other users will just be silently not shown, even though they will prevent unmounting the filesystem.

    Example:

    Watt:~# fuser -vm /mnt/Zia/src
                         USER        PID ACCESS COMMAND
    /mnt/Zia/src:        root     kernel mount /mnt/Zia/src
                         anthony   24909 ..c.. bash
                         anthony   25041 F.c.. gvim
    

    The "access" field tells you how its being accessed. In this case, the kernel has it in use as a mount (duh, but unmount will be OK with only this). bash has it as the current working directory (will have to cd to a different directory before unmount) and gvim both has the current directory and has a file open (will need to close that gvim).

    Watt:~# lsof /mnt/Zia/src
    COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
    bash    24909 anthony  cwd    DIR   0,26    12288 3527682 /mnt/Zia/src/perl (zia.vpn.home:/home/anthony/src)
    gvim    25041 anthony  cwd    DIR   0,26    12288 3527682 /mnt/Zia/src/perl (zia.vpn.home:/home/anthony/src)
    gvim    25041 anthony    6u   REG   0,26    16384 3526219 /mnt/Zia/src/perl/.utf8.c.swp (zia.vpn.home:/home/anthony/src)
    

    In this output, you can see the current directories for both bash and gvim (as type DIR). You can also see which file gvim has open for write.

    How to force the issue:

    fuser has a -k option which will send a signal (default: SIGKILL) to each process using the mount. This is a rather forceful way to stop the mount from being busy. (And of course, be careful of what you SIGKILL!)

    umount has an -l option to perform a lazy unmount. The mount will be removed from the filesystem namespace (so you won't see it under /mnt/Zia/src anymore, in the example) but it stays mounted, so programs accessing it can continue to do so. When the last program accessing it exits, the unmount will actually occur.

    There is one final fixable cause of unmount failing, and that's an NFS server going down. Here you can use umount -f, but you risk data loss if you do so. (The client may have cached writes that haven't been confirmed by the server yet, and those writes will be discarded. Apps, however, have already been told the write is successful.)

    Note that `fuser -k` is **extremely** risky, as you'll be doing it as root and if you're not **very** sure of which processes will be killed off you can do truly spectacular damage with a careless command...

    @Shadur well, hopefully you've already run it without the `-k` option, so you'll know which processes you're going to kill. But I'll add in a warning.

    `fuser -vm` showed "kernel mount". had to do `systemctl stop opt.mount` instead of manual `umount`.

    For some reason umount -f doesn't work for me but running umount -l works perfectly.

    Thanks for the note about `umount -f`and NFS. My issue was NFS related where my dev machines changed IPs and I could not get the share removed.

    One note "F...." doesn't seem to block the unmount (I thought it would), but like you have above having the "c" does. Not sure why this is.

    using lazy method (my fav is being lazy) worked for me.

  • You should use:

    sudo umount -l <path>
    

    From the man page:

    -l, --lazy

      Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore.

      A system reboot would be expected in near future if you're going to use this option for network filesystem or local filesystem with submounts.  The recommended use-case for umount -l is to prevent hangs on shutdown due to an unreachable network share where a normal umount will hang due to a downed server or a network partition. Remounts of the share will not be possible.

    ⁺¹, I've no idea what silly person could downvote it. The `-l` is exactly the option to use when even `-f` doesn't work.

    @Hi-Angel Because this is not what the OP is asking?

    @xheinne stack exchange is not just about answering a question like a stupeed bot, this answer is helpful. many people come from google search also. I personally found this helpful. The op shoudl accept the answer which he find relevant thats why the accept button is there.

  • Another volume is mounted on top of a volume we want to unmount :

    The mount command lets you know all the mounted volumes if invoqued without arguments nor options (except -v). You can have a list of active mountpoints by adding a bit of perl :

    mount | perl -pe 's/.*on (\S+) type.*/\1/'
    

    Then, just grep over the mointpoint from which you wish to unmount and you will know if there is mounted filesystems over this one.

    mount | perl -pe 's/.*on (\S+) type.*/\1/' | grep '/mnt/dir/'
    

    Then you have two solutions. Either unmount the filesystems, or move them with mount --move olddir newdir (kernel >2.5.1)

    Yes, thanks. /etc/mtab and /proc/mounts are also possible.

    Ah that's right, I always forgot those. Let's say that typing "mount" requires less characters (but more resources to execute?)

    I have an external USB storage device "permanently" mounted on a certain directory on my laptop, Sometimes the cable gets disconnected by mistake. It was a big pain to remount the device on the directory (because of "device busy") until I read this answer. Now I known to use mount --move olddir newdir. Thanks.

  • Open files

    Processes with open files are the usual culprits. Display them:

    lsof +f -- <mountpoint or device>
    

    There is an advantage to using /dev/<device> rather than /mountpoint: a mountpoint will disappear after an umount -l, or it may be hidden by an overlaid mount.

    fuser can also be used, but to my mind lsof has a more useful output. However fuser is useful when it comes to killing the processes causing your dramas so you can get on with your life.

    List files on <mountpoint> (see caveat above):

    fuser -vmM <mountpoint>
    

    Interactively kill only processes with files open for writing:

    fuser -vmMkiw <mountpoint>
    

    After remounting read-only (mount -o remount,ro <mountpoint>), it is safe(r) to kill all remaining processes:

    fuser -vmMk <mountpoint>
    

    Mountpoints

    The culprit can be the kernel itself. Another filesystem mounted on the filesystem you are trying to umount will cause grief. Check with:

    mount | grep <mountpoint>/
    

    For loopback mounts, also check the output of:

    losetup -la
    

    Anonymous inodes (Linux)

    Anonymous inodes can be created by:

    • Temporary files (open with O_TMPFILE)
    • inotify watches
    • [eventfd]
    • [eventpoll]
    • [timerfd]

    These are the most elusive type of pokemon, and appear in lsof's TYPE column as a_inode (which is undocumented in the lsof man page).

    They won't appear in lsof +f -- /dev/<device>, so you'll need to:

    lsof | grep a_inode
    

    For killing processes holding anonymous inodes, see: List current inotify watches (pathname, PID).

  • The question how to check if NFS access a directory which is going to unmount is still unanswered.

    What I have is only this:

    Check if nfsd is running:

    pidof nfsd
    

    Show mounted directories by clients:

    showmount -a
    

    and showmount w/o arguments shows only client hosts even if they are off line. I assume this is a special behavior of NFS.

  • For me, the issue was that I was logged in more than once (via ssh) and on one of the logins I was at command prompt where the pwd was inside a folder subordinate to the mount-point.

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM