Python3.5 subprocess not writing on terminal while writing print logs in file

You can read in the middle of the process stdout, stderr and you can write to stdin. The problem are always the buffers.

First where you have to deactivate the buffers from tests.py. The -u option deactivates all buffers, so the output is directly written to stdout. No program change in your tests.py.

cmd = ['python', '-u', './tests.py']
proc = Popen(cmd, stdout=PIPE) 

The second buffer “problem” is with writing to an open file. When you open a file, even in binary mode, fd.write(data) does not immediately write to disk.

But you can control the buffer:

fd = open('tests.log', 'ab', buffering=0)

You can only deactivate the buffer only in binary mode.

Additionally you can read always one byte. This is only interesting, when the output of one line takes veeeery long or if you handle with real binary data.

You may want to question your expectations.

Popen() will not call “tests.py” to run and then continue with the execution of “My_Tests.py”. This is called “blocking”, what you want is a “non-blocking” way to call a new process from a python script.

From the “subprocess” documentation (https://docs.python.org/3/library/subprocess.html): Run the command described by args. Wait for command to complete, then return a CompletedProcess instance.

Heya,

An update on an older topic,

It seems like your issue is related to output buffering. The standard Python output (stdout) is usually line-buffered when attached to a console (your terminal), but fully buffered when directed to a file or pipe. This can lead to situations where output appears in chunks or all at once, rather than line-by-line.

You can try to use flush() method to force Python to write out the buffered data immediately.

Something like that should work

from subprocess import Popen, PIPE, STDOUT
import sys

with Popen("./tests.py", stdout=PIPE, stderr=STDOUT, bufsize=1, universal_newlines=True) as p, \
    open('tests.log', 'w') as file:  # Change 'ab' to 'w' since you are using universal_newlines=True
    for line in p.stdout:  # b'\n'-separated lines
        print(line, end='')  # new addition
        sys.stdout.flush()  # Ensure the output is displayed immediately
        file.write(line)
        file.flush()  # Ensure the file is written immediately

This forces both the terminal output and file output to be written immediately after each line is processed.

Also, please note that I changed the mode of opening your log file from ‘ab’ (append binary) to ‘w’ (write text) because you’re using universal_newlines=True, which means you’re treating the input/output as text, not binary.

Please, make sure to adjust these settings according to your needs and the nature of your input/output data.