How do I delete the first n lines of an ascii file using shell commands?

  • I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

    If it's any help, I'm using RedHat and an Ubuntu linux systems.

  • As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

    $ cat t.txt
    12
    34
    56
    78
    90
    

    sed

    $ sed -e '1,3d' < t.txt
    78
    90
    

    You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

    tail

    $ tail -n +4 t.txt
    78
    90
    

    awk

    $ awk 'NR > 3 { print }' < t.txt
    78
    90
    

    You can also use sed in-place without a temp file: `sed -i -e 1,3d yourfile`. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

    As long as the file is not a symlink or hardlink.

    Thanks @YanickGirouard, @IgnacioVazquezAbrams! You two have just saved me a ton of manual labor on my research! :)

    @jw013 What is the symlink limitation about? Sed? Awk? Tail? All of them?

    @Svetlana `sed -i` specifically. Most implementations just delete the file and replace it with a new one, which doesn't work for links since you end up leaving the original at its other location.

    how about explaining what '1,3d', +4, et.c. means? The question was for n lines, but you didn't tell what n is (as apparently n is 2 in your examples, though it's not obvious for a noob what to change in order to change n)

    @Ignacio Vazquez-Abrams what would the command look like if I need to remove a variable number of lines. e.g. sed -i -e "1,${n}d" yourfile. what is the correct syntax to remove n number of lines

    This uses a temp file so not very useful for a 100% util disk space. Would be interesting to have a solution that does this literally "in-place".

    @Shai yes `sed` doesn't do it in-place at all, it creates a temp file.

    You can also do a sed -e '1,3d' t.txt

    Dont try on 'live' (log) files as this will replace the actual file with truncated file.

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM