How to view core files for debugging purposes in Linux?

  • I want to view the contents of a core file while debugging a program. How can I view the contents of a core file?

  • objdump + gdb minimal runnable example

    TLDR:

    Now for a the full educational test setup:

    main.c

    #include <stddef.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    
    int myfunc(int i) {
        *(int*)(NULL) = i; /* line 7 */
        return i - 1;
    }
    
    int main(int argc, char **argv) {
        /* Setup some memory. */
        char data_ptr[] = "string in data segment";
        char *mmap_ptr;
        char *text_ptr = "string in text segment";
        (void)argv;
        mmap_ptr = (char *)malloc(sizeof(data_ptr) + 1);
        strcpy(mmap_ptr, data_ptr);
        mmap_ptr[10] = 'm';
        mmap_ptr[11] = 'm';
        mmap_ptr[12] = 'a';
        mmap_ptr[13] = 'p';
        printf("text addr: %p\n", text_ptr);
        printf("data addr: %p\n", data_ptr);
        printf("mmap addr: %p\n", mmap_ptr);
    
        /* Call a function to prepare a stack trace. */
        return myfunc(argc);
    }
    

    Compile, and run to generate core:

    gcc -ggdb3 -std=c99 -Wall -Wextra -pedantic -o main.out main.c
    ulimit -c unlimited
    rm -f core
    ./main.out
    

    Output:

    text addr: 0x4007d4
    data addr: 0x7ffec6739220
    mmap addr: 0x1612010
    Segmentation fault (core dumped)
    

    GDB points us to the exact line where the segfault happened, which is what most users want while debugging:

    gdb -q -nh main.out core
    

    then:

    Reading symbols from main.out...done.
    [New LWP 27479]
    Core was generated by `./main.out'.
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  0x0000000000400635 in myfunc (i=1) at main.c:7
    7           *(int*)(NULL) = i;
    (gdb) bt
    #0  0x0000000000400635 in myfunc (i=1) at main.c:7
    #1  0x000000000040072b in main (argc=1, argv=0x7ffec6739328) at main.c:28
    

    which points us directly to the buggy line 7.

    Binutils analysis

    First:

    file core
    

    tells us that the core file is actually an ELF file:

    core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './main.out'
    

    which is why we are able to inspect it more directly with usual binutils tools.

    A quick look at the ELF standard shows that there is actually an ELF type dedicated to it:

    Elf32_Ehd.e_type == ET_CORE
    

    Further format information can be found at:

    man 5 core
    

    Then:

    readelf -Wa core
    

    gives some hints about the file structure. Memory appears to be contained in regular program headers:

    Program Headers:
      Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
      NOTE           0x000468 0x0000000000000000 0x0000000000000000 0x000b9c 0x000000     0
      LOAD           0x002000 0x0000000000400000 0x0000000000000000 0x001000 0x001000 R E 0x1000
      LOAD           0x003000 0x0000000000600000 0x0000000000000000 0x001000 0x001000 R   0x1000
      LOAD           0x004000 0x0000000000601000 0x0000000000000000 0x001000 0x001000 RW  0x1000
    

    and there is some more metadata present in a notes area, notably prstatus contains the PC:

    Displaying notes found at file offset 0x00000468 with length 0x00000b9c:
      Owner                 Data size       Description
      CORE                 0x00000150       NT_PRSTATUS (prstatus structure)
      CORE                 0x00000088       NT_PRPSINFO (prpsinfo structure)
      CORE                 0x00000080       NT_SIGINFO (siginfo_t data)
      CORE                 0x00000130       NT_AUXV (auxiliary vector)
      CORE                 0x00000246       NT_FILE (mapped files)
        Page size: 4096
                     Start                 End         Page Offset
        0x0000000000400000  0x0000000000401000  0x0000000000000000
            /home/ciro/test/main.out
        0x0000000000600000  0x0000000000601000  0x0000000000000000
            /home/ciro/test/main.out
        0x0000000000601000  0x0000000000602000  0x0000000000000001
            /home/ciro/test/main.out
        0x00007f8d939ee000  0x00007f8d93bae000  0x0000000000000000
            /lib/x86_64-linux-gnu/libc-2.23.so
        0x00007f8d93bae000  0x00007f8d93dae000  0x00000000000001c0
            /lib/x86_64-linux-gnu/libc-2.23.so
        0x00007f8d93dae000  0x00007f8d93db2000  0x00000000000001c0
            /lib/x86_64-linux-gnu/libc-2.23.so
        0x00007f8d93db2000  0x00007f8d93db4000  0x00000000000001c4
            /lib/x86_64-linux-gnu/libc-2.23.so
        0x00007f8d93db8000  0x00007f8d93dde000  0x0000000000000000
            /lib/x86_64-linux-gnu/ld-2.23.so
        0x00007f8d93fdd000  0x00007f8d93fde000  0x0000000000000025
            /lib/x86_64-linux-gnu/ld-2.23.so
        0x00007f8d93fde000  0x00007f8d93fdf000  0x0000000000000026
            /lib/x86_64-linux-gnu/ld-2.23.so
      CORE                 0x00000200       NT_FPREGSET (floating point registers)
      LINUX                0x00000340       NT_X86_XSTATE (x86 XSAVE extended state)
    

    objdump can easily dump all memory with:

    objdump -s core
    

    which contains:

    Contents of section load1:
    
     4007d0 01000200 73747269 6e672069 6e207465  ....string in te
     4007e0 78742073 65676d65 6e740074 65787420  xt segment.text 
    
    Contents of section load15:
    
     7ffec6739220 73747269 6e672069 6e206461 74612073  string in data s
     7ffec6739230 65676d65 6e740000 00a8677b 9c6778cd  egment....g{.gx.
    
    Contents of section load4:
    
     1612010 73747269 6e672069 6e206d6d 61702073  string in mmap s
     1612020 65676d65 6e740000 11040000 00000000  egment..........
    

    which matches exactly with the stdout value in our run.

    Tested in Ubuntu 16.04 amd64, GCC 6.4.0, binutils 2.26.1.

    The PC is part of the prstatus structure, which contains all registers. See the elf_gregset_t field: https://github.com/torvalds/linux/blob/master/include/uapi/linux/elfcore.h

  • gdb is the GNU debugger which can be used to examine the core file. BTW bt (backtrace) is a useful gdb command to examine the program call stack.

    gdb binary-file core-file
    
  • When you compile the program use -g option

    gcc -g program.c

    If core file is created then you can debug using gdb whithout using -g option debug flags wont be enabled.

  • If prefer to use command line tool, then you can use gdb :

    gdb <program> <core file>
    

    or

    gdb <program> -c <core file>
    

    If you like gui, then install ddd, and from there open the program to debug and the core file.

  • #-------------------------------------------------------------------------
    #!/usr/bin/ksh
    # -------------------------------------------------------------------------
    
    _OUTFILE=XXXX-XXXX-Audit-`date +"%Y%m%d%H%M"`.log
    >$_OUTFILE
    MAILLIST=""
    COREPATH=$PKMS/logs/cores
    MARKER=$COREPATH/marker
    
    function Parse
    {
       while getopts :p:u:s:l: name
          do
        case $name in
            p) PKMS="$OPTARG" ;;       # $PKMS
            u) DBUSER="$OPTARG" ;;     # $DBUSER 
            s) DBPSWD="$OPTARG" ;;     # $DBPSWD
            l) DBLOCN="$OPTARG" ;;     # $DBLOC 
            *) Usage ;;                     # display usage and exit
           esac
          done
       if [[ -z "${PKMS}"  || -z "${DBUSER}" || -z "${DBPSWD}" || -z "${DBLOCN}" ]] 
       then
        echo $Usage
        exit -1
       fi
    }
    
    
    function getCoreDumps
    {
       COREFILES=$COREPATH/newcores.txt
       STACKS=$COREPATH/stacks.txt
       DATE=$(date +%y%m%d%H%M%S)
       >$COREFILES
       >$STACKS
       umask 002
    
       find $COREPATH -type f -newer $MARKER -name "core" > $COREFILES
       find $COREPATH -type f -newer $MARKER -name "core.?" >> $COREFILES
    
       rm $STACKS 2>/dev/null
    
       for i in $(<$COREFILES)
       do
            mv $i $i.$DATE
            chmod g+r,g+w $i.$DATE
            #echo "Coredump recently found at" `date` '\n'>> $STACKS
            echo $i.$DATE >> $STACKS
        #echo >> $STACKS
       done
    
       NL=$(wc -l $COREFILES  | awk '{ print $1 }')
       if [ "$NL" -gt 0 ]
       then
        echo "New CORE files found:" >> $_OUTFILE
        echo "--- ---- ----- ------" >> $_OUTFILE
        cat $STACKS >> $_OUTFILE
       else
        echo "No new CORE files found" >> $_OUTFILE
        echo "-- --- ---- ----- -----" >> $_OUTFILE
       fi
    
    }
    
    
    
    #/usr/bin/clear
    
    echo "\t\t\t\t---------------------------------\t" >> $_OUTFILE
    echo "\t\t\t\t
    echo "\t\t\t\t---------------------------------\t" >> $_OUTFILE
    
    date "+                             %d/%m/%Y %H:%M:%S"  >> $_OUTFILE
    
    echo "===================" >> $_OUTFILE
    echo " APPICATION MACHINES" >> $_OUTFILE
    echo "===================" >> $_OUTFILE
    echo >> $_OUTFILE
    echo >> $_OUTFILE
    
    
    
    getCoreDumps
    echo >> $_OUTFILE
    echo >> $_OUTFILE
    
    
    
    echo "===================" >> $_OUTFILE
    echo "XXXX APP DataBase Info" >> $_OUTFILE
    echo "===================" >> $_OUTFILE
    
    echo >> $_OUTFILE
    getAPPDBInfo
    echo >> $_OUTFILE
    echo >> $_OUTFILE
    
    MAILDATE=$(date +%d/%m/%Y)
    
    
    mailx -s "XXXX Monitor Log for $PKMS Environment - Dated $MAILDATE" $MAILLIST < $_OUTFILE
    
    touch $MARKER
    rm /tmp/XXXXtempOUTFILE
    exit 0
    

    Please update this answer with what this script does and how it answers the question

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM