Use config file for my shell script

  • I need to create a config file for my own script: here an example:

    script:

    #!/bin/bash
    source /home/myuser/test/config
    echo "Name=$nam" >&2
    echo "Surname=$sur" >&2
    

    Content of /home/myuser/test/config:

    nam="Mark"
    sur="Brown"
    

    that's works!

    My question: is this the correct way to do this or there're other ways?

    The variables should be at the top. I'm surprised it works. Anyway, why do you need a config file? Are you planning to use these variables somewhere else?

    Faheem, I need the variables because my script has many options: using a config file will semplify the script. Thanks

    IMHO its fine. I would do this way.

    `abcde` also does it this way and that is a quite big program (for a shell script). You can have a look at it here.

  • source is not secure as it will execute arbitrary code. This may not be a concern for you, but if file permissions are incorrect, it may be possible for an attacker with filesystem access to execute code as a privileged user by injecting code into a config file loaded by an otherwise-secured script such as an init script.

    So far, the best solution I've been able to identify is the clumsy reinventing-the-wheel solution:

    myscript.conf

    password=bar
    echo rm -rf /
    PROMPT_COMMAND='echo "Sending your last command $(history 1) to my email"'
    hostname=localhost; echo rm -rf /
    

    Using source, this would run echo rm -rf / twice, as well as change the running user's $PROMPT_COMMAND. Instead, do this:

    myscript.sh (Bash 4)

    #!/bin/bash
    typeset -A config # init array
    config=( # set default values in config array
        [username]="root"
        [password]=""
        [hostname]="localhost"
    )
    
    while read line
    do
        if echo $line | grep -F = &>/dev/null
        then
            varname=$(echo "$line" | cut -d '=' -f 1)
            config[$varname]=$(echo "$line" | cut -d '=' -f 2-)
        fi
    done < myscript.conf
    
    echo ${config[username]} # should be loaded from defaults
    echo ${config[password]} # should be loaded from config file
    echo ${config[hostname]} # includes the "injected" code, but it's fine here
    echo ${config[PROMPT_COMMAND]} # also respects variables that you may not have
                   # been looking for, but they're sandboxed inside the $config array
    

    myscript.sh (Mac/Bash 3-compatible)

    #!/bin/bash
    config() {
        val=$(grep -E "^$1=" myscript.conf 2>/dev/null || echo "$1=__DEFAULT__" | head -n 1 | cut -d '=' -f 2-)
    
        if [[ $val == __DEFAULT__ ]]
        then
            case $1 in
                username)
                    echo -n "root"
                    ;;
                password)
                    echo -n ""
                    ;;
                hostname)
                    echo -n "localhost"
                    ;;
            esac
        else
            echo -n $val
        fi
    }
    
    echo $(config username) # should be loaded from defaults
    echo $(config password) # should be loaded from config file
    echo $(config hostname) # includes the "injected" code, but it's fine here
    echo $(config PROMPT_COMMAND) # also respects variables that you may not have
                   # been looking for, but they're sandboxed inside the $config array
    

    Please reply if you find a security exploit in my code.

    FYI this is a Bash version 4.0 solution which sadly is subject to insane licensing issues imposed by Apple and is not available by default on Macs

    @Sukima Good point. I've added a version that is compatible with Bash 3. Its weakness is that it will not handle `*` in inputs properly, but then what in Bash handles that character well?

    The first script fails if the password contains a backslash.

    @Kusalananda What if the backslash is escaped? `my\\password`

    This version takes several seconds to process my config file, I found the solution from gw0 to be much faster.

  • Here is a clean and portable version which is compatible with Bash 3 and up, on both Mac and Linux.

    It specifies all defaults in a separate file, to avoid the need for a huge, cluttered, duplicated "defaults" config function in all of your shell scripts. And it lets you choose between reading with or without default fallbacks:

    config.cfg:

    myvar=Hello World
    

    config.cfg.defaults:

    myvar=Default Value
    othervar=Another Variable
    

    config.shlib (this is a library, so there is no shebang-line):

    config_read_file() {
        (grep -E "^${2}=" -m 1 "${1}" 2>/dev/null || echo "VAR=__UNDEFINED__") | head -n 1 | cut -d '=' -f 2-;
    }
    
    config_get() {
        val="$(config_read_file config.cfg "${1}")";
        if [ "${val}" = "__UNDEFINED__" ]; then
            val="$(config_read_file config.cfg.defaults "${1}")";
        fi
        printf -- "%s" "${val}";
    }
    

    test.sh (or any scripts where you want to read config values):

    #!/usr/bin/env bash
    source config.shlib; # load the config library functions
    echo "$(config_get myvar)"; # will be found in user-cfg
    printf -- "%s\n" "$(config_get myvar)"; # safer way of echoing!
    myvar="$(config_get myvar)"; # how to just read a value without echoing
    echo "$(config_get othervar)"; # will fall back to defaults
    echo "$(config_get bleh)"; # "__UNDEFINED__" since it isn't set anywhere
    

    Explanation of the test script:

    • Note that all usages of config_get in test.sh are wrapped in double quotes. By wrapping every config_get in double quotes, we ensure that text in the variable value will never be misinterpreted as flags. And it ensures that we preserve whitespace properly, such as multiple spaces in a row in the config value.
    • And what's that printf line? Well, it's something you should be aware of: echo is a bad command for printing text that you have no control over. Even if you use double quotes, it will interpret flags. Try setting myvar (in config.cfg) to -e and you will see an empty line, because echo will think that it's a flag. But printf doesn't have that problem. The printf -- says "print this, and don't interpret anything as flags", and the "%s\n" says "format the output as a string with a trailing newline, and lastly the final parameter is the value for printf to format.
    • If you aren't going to be echoing values to the screen, then you'd simply assign them normally, like myvar="$(config_get myvar)";. If you're going to print them to the screen, I suggest using printf to be totally safe against any echo-incompatible strings that may be in the user config. But echo is fine if the user-provided variable isn't the first character of the string you are echoing, since that's the only situation where "flags" could be interpreted, so something like echo "foo: $(config_get myvar)"; is safe, since the "foo" doesn't begin with a dash and therefore tells echo that the rest of the string isn't flags for it either. :-)

    @user2993656 Thanks for spotting that my original code still had my private config filename (environment.cfg) in it instead of the correct one. As for the "echo -n" edit you did, that depends on the shell used. On Mac/Linux Bash, "echo -n" means "echo without trailing newline", which I did to avoid trailing newlines. But it seems to work exactly the same without it, so thanks for the edits!

    Actually, I just went through and rewrote it to use printf instead of echo, which ensures that we'll get rid of the risk of echo misinterpreting "flags" in the config values.

    I really like this version. I dropped the `config.cfg.defaults` in lieu of defining them at the time of calling `$(config_get var_name "default_value")`. https://tritarget.org/static/Bash%2520script%2520configuration%2520file%2520format.html

    Likewise - this is great.

  • Parse the configuration file, don't execute it.

    I'm currently writing an application at work that uses an extremely simple XML configuration:

    <config>
        <username>username-or-email</username>
        <password>the-password</password>
    </config>
    

    In the shell script (the "application"), this is what I do to get at the username (more or less, I've put it in a shell function):

    username="$( xml sel -t -v '/config/username' "$config_file" )"
    

    The xml command is XMLStarlet, which is available for most Unices.

    I'm using XML since other parts of the application also deals with data encoded in XML files, so it was easiest.

    If you prefer JSON, there's jq which is an easy to use shell JSON parser.

    My configuration file would look something like this in JSON:

    {                                 
      "username": "username-or-email",
      "password": "the-password"      
    }                
    

    And then I'd be getting the username in the script:

    username="$( jq -r '.username' "$config_file" )"
    

    Executing the script has a number of advantages and disadvantages. The main disadvantages are security, if someone can alter the config file then they can execute code, and it is harder to make it idiot proof. The advantages are speed, on a simple test it is more than 10,000 times faster to source a config file than to run pq, and flexibility, anyone who likes monkey patching python will appreciate this.

    @icarus How big configuration files do you usually encounter, and how often do you need to parse them in one session? Notice too that several values may be had out of XML or JSON in one go.

    Usually only a few (1 to 3) values. If you are using `eval` to set multiple values then you are executing selected parts of the config file :-).

    @icarus I was thinking arrays... No need to `eval` anything. The performance hit of using a standard format with an existing parser (even though it's an external utility) is negligible in comparison to the robustness, the amount of code, the ease of use, and maintainability.

    +1 for "parse the config file, don't execute it"

    Use `xmlstarlet` instead of `xml`. Please fix this mistake

  • The most common, efficient and correct way is to use source, or . as a shorthand form. For example:

    source /home/myuser/test/config
    

    or

    . /home/myuser/test/config
    

    Something to consider, however, is the security issues that using an additional externally-sourced configuration file can raise, given that additional code can be inserted. For more information, including on how to detect and resolve this issue, I would recommend taking a look at the 'Secure it' section of http://wiki.bash-hackers.org/howto/conffile#secure_it

    I had high hopes for that article (came up in my search results too), but the author's suggestion of attempting to use regex to filter out malicious code is an exercise in futility.

    The procedure with dot, requires an absolute path? With the relative one it doesn't work

  • I use this in my scripts:

    sed_escape() {
      sed -e 's/[]\/$*.^[]/\\&/g'
    }
    
    cfg_write() { # path, key, value
      cfg_delete "$1" "$2"
      echo "$2=$3" >> "$1"
    }
    
    cfg_read() { # path, key -> value
      test -f "$1" && grep "^$(echo "$2" | sed_escape)=" "$1" | sed "s/^$(echo "$2" | sed_escape)=//" | tail -1
    }
    
    cfg_delete() { # path, key
      test -f "$1" && sed -i "/^$(echo $2 | sed_escape).*$/d" "$1"
    }
    
    cfg_haskey() { # path, key
      test -f "$1" && grep "^$(echo "$2" | sed_escape)=" "$1" > /dev/null
    }
    

    Should support every character combination, except keys can't have = in them, since that's the seperator. Anything else works.

    % cfg_write test.conf mykey myvalue
    % cfg_read test.conf mykey
    myvalue
    % cfg_delete test.conf mykey
    % cfg_haskey test.conf mykey || echo "It's not here anymore"
    It's not here anymore
    

    Also, this is completely safe since it doesn't use any kind of source/eval

  • This is succinct and secure:

    # Read common vars from common.vars
    # the incantation here ensures (by env) that only key=value pairs are present
    # then declare-ing the result puts those vars in our environment 
    declare $(env -i `cat common.vars`)
    

    The -i ensures you only get the variables from common.vars

    Update: An illustration of the security is that

    env -i 'touch evil1 foo=omg boo=$(touch evil2)'
    

    Will not produce any touched files. Tested on mac with bash, i.e. using bsd env.

    Just see how evil1 and evil2 files are created if you put this to common.vars ``` touch evil1 foo=omg boo=$(touch evil2) ```

    @pihentagy For me the following produces no touched files `env -i 'touch evil1 foo=omg boo=$(touch evil2)'`. Running on mac.

    indeed, but cannot access foo. I've tried `env -i ... myscript.sh` and inside that script foo is not defined. However, if you remove "garbage", it will work. So thanks for explaining. :+1:

  • For my scenario, source or . was fine, but I wanted to support local environment variables (ie, FOO=bar myscript.sh) taking precedence over configured variables. I also wanted the config file to be user editable and comfortable to someone used to sourced config files, and to keep it as small/simple as possible, to not distract from the main thrust of my very small script.

    This is what I came up with:

    CONF=${XDG_CONFIG_HOME:-~/config}/myscript.sh
    if [ ! -f $CONF ]; then
        cat > $CONF << CONF
    VAR1="default value"
    CONF
    fi
    . <(sed 's/^\([^=]\+\) *= *\(.*\)$/\1=${\1:-\2}/' < $CONF)
    

    Essentially - it checks for variable definitions (without being very flexible about whitespace) and rewrites those lines so that the value is converted to a default for that variable, and the variable is unmodified if found, like the XDG_CONFIG_HOME variable above. It sources this altered version of the config file and continues on.

    Future work could make the sed script more robust, filter out lines that look weird or aren't definitions, etc, not break on end of line comments - but this is good enough for me for now.

  • This one seems safe and short. Feel free to crack this ruthlessly. I'd like to know of a better way.

    TL;DR;

    while read LINE; do declare "$LINE"; done < evil.conf
    

    I'm using bash 4.3.48.

    It's also compliant with bash --posix. See bottom for test.

    But sh doesn't support it because of declare.


    Basic test for those who want proof

    Create file evil.conf

    echo > evil.conf '
    A=1
    B=2
    C=$(echo hello)
    
    # Could produce side-effect
    D=`touch evil`
    C=$((1+2))
    E=$(ping 8.8.8.8 -n 3)
    echo hello
    
    # Could produce visible side-effect
    touch evil2
    ping 8.8.8.8 -n 3
    F=ok'
    
    

    Load the config with the snippet

    while read LINE; do declare "$LINE"; done < evil.conf
    
    

    Output (see sanitizer in action)

    bash: declare: `': not a valid identifier
    bash: declare: `': not a valid identifier
    bash: declare: `# Could produce side-effect': not a valid identifier
    bash: declare: `echo hello': not a valid identifier
    bash: declare: `': not a valid identifier
    bash: declare: `# Could produce visible side-effect': not a valid identifier
    bash: declare: `touch evil2': not a valid identifier
    bash: declare: `ping 8.8.8.8 -n 3': not a valid identifier
    

    Let's check values now

    for V in A B C D E F; do declare -p $V; done
    
    
    declare -- A="1"
    declare -- B="2"
    declare -- C="\$((1+2))"
    declare -- D="\`touch evil\`"
    declare -- E="\$(ping 8.8.8.8 -n 3)"
    declare -- F="ok"
    

    Check side effects (no side effects):

    ls evil evil2
    
    
    ls: cannot access 'evil': No such file or directory
    ls: cannot access 'evil2': No such file or directory
    

    Appendix. Test bash --posix

    bash -c 'while read LINE; do declare "$LINE"; done < evil.conf; for V in A B C D E F; do declare -p $V; done' --posix
    

    Thanks to @Marcin for kick-off idea.

  • You can do it:

    #!/bin/bash
    name="mohsen"
    age=35
    cat > /home/myuser/test/config << EOF
    Name=$name
    Age=$age
    EOF
    

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM