Write Python stdout to file immediately

  • When trying to write the stdout from a Python script to a text file (python script.py > log), the text file is created when the command is started, but the actual content isn't written until the Python script finishes. For example:

    script.py:

    import time
    for i in range(10):
        print('bla')
        time.sleep(5)
    

    prints to stdout every 5 seconds when called with python script.py, but when I call python script.py > log, the size of the log file stays zero until the script finishes. Is it possible to directly write to the log file, such that you can follow the progress of the script (e.g. using tail)?

    EDIT It turns out that python -u script.py does the trick, I didn't know about the buffering of stdout.

    @jezmck, I could have understood question wrong.

  • This is happening because normally when process STDOUT is redirected to something other than a terminal, then the output is buffered into some OS-specific-sized buffer (perhaps 4k or 8k in many cases). Conversely, when outputting to a terminal, STDOUT will be line-buffered or not buffered at all, so you'll see output after each \n or for each character.

    You can generally change the STDOUT buffering with the stdbuf utility:

    stdbuf -oL python script.py > log
    

    Now if you tail -F log, you should see each line output immediately as it is generated.


    Alternatively explicit flushing of the output stream after each print should achieve the same. It looks like sys.stdout.flush() should achieve this in Python. If you are using Python 3.3 or newer, the print function also has a flush keyword that does this: print('hello', flush=True).

    Thanks, I didn't know about the buffering! Knowing that, Google pretty quickly told me that `python -u script.py` does the trick. *EDIT* So many answers at once, I accepted yours since it pointed me in the direction of the buffering.

    @julbra Cool, yes I didn't know python had that option either. Some command-line programs also have similar options - e.g. `--line-buffered` for `grep`, but some others don't. `stdbuf` is the general catchall utility to deal with those that don't.

    @DigitalTrauma: Isn't it better to use no buffering at all i.e. `stdbuf -o0 python script.py > log` in this kind of determined circumstances?

    @heemayl `-oL` is a compromise. In general larger buffers will provide better performance when redirecting somewhere (fewer system calls and fewer I/O operations). However if it is absolutely necessary to see each character as it is output then yes, `-o0` would be required.

    @Paul Please avoid copy pasting contents between answers, or at the bery least mention the original authors that provided the content.

    @Bakuriu I wrote the edit myself. What answer do you think I copy-pasted the text from? **Edit 1:** I see you mentioned the same thing in your own answer. Again, I didn't copy anything. In fact, DigitalTrauma's answer was the only one I read when I edited it. **Edit 2:** And to be fair, you didn't mention anything about this being available only since Python 3.3.

    @Paul Before introducing *new content* in an answer you *should* look the other answers, to avoid these kind of situations. Only edits about formatting/adding references and such don't need the context from other answers. (Btw: Python<3.3 is my definition of *ancient version of python*.)

    Oh my, many thanks! I was pulling my hair out for hours: why would I not see any output from a simple pipe from a python program streaming output piped to sed? Worked in Ubuntu, but not on Debian, at least not until I killed the python program. Turns out the output was just caught in the stdout buffer on Debian! But not on Ubuntu. Go figure! Your stdbuf -oL solution fixed it instantly.

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM