Watching something be written to a file live with tailRedirect output of remote python app started through ssh to fileUbuntu RFID Screensaver lock-unlockHow can I offer an XP-compatible migration to Linux Mint?Is there a command in Linux which waits till it will be terminated?Ctrl+c in a sub process is killing a nohup'ed process earlier in the scripthow to get cronjob running every minuteCtrl-C'd an in-place recursive gzip - is this likely to have broken anything?Getting output of another script while preserving line-breakssocat and rich terminal again“Ctrl + c” combination works different on different SSH clients

How to model explosives?

Were any external disk drives stacked vertically?

Neighboring nodes in the network

Python: return float 1.0 as int 1 but float 1.5 as float 1.5

Why does Kotter return in Welcome Back Kotter

Can one be a co-translator of a book, if he does not know the language that the book is translated into?

Forgetting the musical notes while performing in concert

What is going on with Captain Marvel's blood colour?

Withdrawals from HSA

Why is consensus so controversial in Britain?

Took a trip to a parallel universe, need help deciphering

Fully-Firstable Anagram Sets

Can I ask the recruiters in my resume to put the reason why I am rejected?

Twin primes whose sum is a cube

Does a druid starting with a bow start with no arrows?

Is it canonical bit space?

Would Slavery Reparations be considered Bills of Attainder and hence Illegal?

Increase size of symbol intercal when in superscript position

In Romance of the Three Kingdoms why do people still use bamboo sticks when papers are already invented?

How do I write bicross product symbols in latex?

Western buddy movie with a supernatural twist where a woman turns into an eagle at the end

How to draw the figure with four pentagons?

Cronab fails because shell path not found

Stopping power of mountain vs road bike



Watching something be written to a file live with tail


Redirect output of remote python app started through ssh to fileUbuntu RFID Screensaver lock-unlockHow can I offer an XP-compatible migration to Linux Mint?Is there a command in Linux which waits till it will be terminated?Ctrl+c in a sub process is killing a nohup'ed process earlier in the scripthow to get cronjob running every minuteCtrl-C'd an in-place recursive gzip - is this likely to have broken anything?Getting output of another script while preserving line-breakssocat and rich terminal again“Ctrl + c” combination works different on different SSH clients






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








20















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question



















  • 7





    How about python myprog.py | tee output.txt instead?

    – n8te
    22 hours ago






  • 2





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    18 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    2 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    2 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    2 hours ago

















20















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question



















  • 7





    How about python myprog.py | tee output.txt instead?

    – n8te
    22 hours ago






  • 2





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    18 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    2 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    2 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    2 hours ago













20












20








20


1






I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?










share|improve this question
















I have a python program which is, slowly, generating some output.



I want to capture that in a file, but I also thought I could watch it live with tail.



So in one terminal I'm doing :



python myprog.py > output.txt


and in another terminal :



tail -f output.txt


But it seems like the tail isn't showing me anything while the python program is running.



If I hit ctrl-c to kill the python script, suddenly the tail of output.txt starts filling up. But not while python is running.



What am I doing wrong?







linux command-line redirection stdout






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 2 hours ago









Peter Cordes

2,4651621




2,4651621










asked 22 hours ago









interstarinterstar

410413




410413







  • 7





    How about python myprog.py | tee output.txt instead?

    – n8te
    22 hours ago






  • 2





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    18 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    2 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    2 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    2 hours ago












  • 7





    How about python myprog.py | tee output.txt instead?

    – n8te
    22 hours ago






  • 2





    @n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

    – JPhi1618
    18 hours ago






  • 1





    stdbuf can be used to alter the buffering status of file descriptors.

    – studog
    2 hours ago











  • Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

    – Peter Cordes
    2 hours ago












  • Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

    – Mark Wagner
    2 hours ago







7




7





How about python myprog.py | tee output.txt instead?

– n8te
22 hours ago





How about python myprog.py | tee output.txt instead?

– n8te
22 hours ago




2




2





@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

– JPhi1618
18 hours ago





@n8te tee might show the same problem if the program isn't flushing the output buffer regularly. This needs flush() and tee.

– JPhi1618
18 hours ago




1




1





stdbuf can be used to alter the buffering status of file descriptors.

– studog
2 hours ago





stdbuf can be used to alter the buffering status of file descriptors.

– studog
2 hours ago













Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

– Peter Cordes
2 hours ago






Terminology: There is no pipe anywhere in this scenario. There's a redirect to a regular file. (Which causes C stdio and Python to decide to make stdout full-buffered instead of line-buffered because it's not a TTY). Pipes are a different type of file (a buffer inside the kernel). I edited your question to correct that.

– Peter Cordes
2 hours ago














Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

– Mark Wagner
2 hours ago





Probably not needed in your situation but if you don't want to terminate the program you can use gdb and call fflush: see stackoverflow.com/questions/8251269/…

– Mark Wagner
2 hours ago










5 Answers
5






active

oldest

votes


















25














You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



import sys
...
print('Some message')
sys.stdout.flush()





share|improve this answer










New contributor




Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.















  • 4





    If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

    – mckenzm
    17 hours ago






  • 1





    You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

    – Dan
    9 hours ago







  • 4





    It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

    – glglgl
    7 hours ago


















17














Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



From man tee:




tee(1) - Linux man page



Name tee - read from standard input and write to standard output and files



Synopsis



tee [OPTION]... [FILE]...


Description



Copy standard input to each FILE, and also to standard output.



-a, --append 
append to the given FILEs, do not overwrite
-i, --ignore-interrupts
ignore interrupt signals
--help
display this help and exit
--version
output version information and exit


If a FILE is -, copy again to standard output.




So in your case you'd run:



python myprog.py | tee output.txt


EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






share|improve this answer

























  • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

    – Baldrickk
    13 hours ago






  • 5





    That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

    – eckes
    12 hours ago






  • 3





    This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

    – Barmar
    3 hours ago


















5














Run python with the unbuffered flag:



python -u myprog.py > output.txt


Output will then print in real time.






share|improve this answer








New contributor




BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



























    2














    Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



    This is a redirect to a regular file.



    C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



    You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



    On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



    Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




    Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






    share|improve this answer






























      0














      Another option (if you don't care about the contents, just the progress) is pv:



      NAME
      pv - monitor the progress of data through a pipe

      SYNOPSIS
      pv [OPTION] [FILE]...
      pv [-h|-V]


      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






      share|improve this answer


















      • 1





        There's no pipeline, just file redirection. And it won't solve the buffering problem.

        – Barmar
        3 hours ago











      Your Answer








      StackExchange.ready(function()
      var channelOptions =
      tags: "".split(" "),
      id: "3"
      ;
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function()
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled)
      StackExchange.using("snippets", function()
      createEditor();
      );

      else
      createEditor();

      );

      function createEditor()
      StackExchange.prepareEditor(
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader:
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      ,
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      );



      );













      draft saved

      draft discarded


















      StackExchange.ready(
      function ()
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');

      );

      Post as a guest















      Required, but never shown

























      5 Answers
      5






      active

      oldest

      votes








      5 Answers
      5






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      25














      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        17 hours ago






      • 1





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        9 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        7 hours ago















      25














      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.















      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        17 hours ago






      • 1





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        9 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        7 hours ago













      25












      25








      25







      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()





      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.










      You may also need to explicitly flush the buffer for it to get piped upon generation. This is because output is typically only printed when the pipe's buffer fills up (which is in kilobytes I belive), and when the stdin message ends. This is probably to save on read/writes. You could do this after every print, or if you are looping, after the last print within the loop.



      import sys
      ...
      print('Some message')
      sys.stdout.flush()






      share|improve this answer










      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this answer



      share|improve this answer








      edited 17 hours ago









      user2313067

      2,1001911




      2,1001911






      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      answered 21 hours ago









      DaveyDavey

      31625




      31625




      New contributor




      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Davey is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.







      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        17 hours ago






      • 1





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        9 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        7 hours ago












      • 4





        If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

        – mckenzm
        17 hours ago






      • 1





        You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

        – Dan
        9 hours ago







      • 4





        It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

        – glglgl
        7 hours ago







      4




      4





      If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

      – mckenzm
      17 hours ago





      If you have read this far, please don't be thinking of closing and re-opening the file to do this, the seeks will be a problem, especially for very large files. (I've seen this done!).

      – mckenzm
      17 hours ago




      1




      1





      You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

      – Dan
      9 hours ago






      You can also use print's flush parameter to do just as well. For example, print('some message', flush=True).

      – Dan
      9 hours ago





      4




      4





      It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

      – glglgl
      7 hours ago





      It has nothing to do with the pipe's buffer, but with the stdout mechanism which doesn't flush after newline if it doesn't write to a tty.

      – glglgl
      7 hours ago













      17














      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer

























      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        13 hours ago






      • 5





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        12 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        3 hours ago















      17














      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer

























      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        13 hours ago






      • 5





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        12 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        3 hours ago













      17












      17








      17







      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.






      share|improve this answer















      Instead of trying to tail a live file, use tee instead. It was made to do exactly what you're trying to do.



      From man tee:




      tee(1) - Linux man page



      Name tee - read from standard input and write to standard output and files



      Synopsis



      tee [OPTION]... [FILE]...


      Description



      Copy standard input to each FILE, and also to standard output.



      -a, --append 
      append to the given FILEs, do not overwrite
      -i, --ignore-interrupts
      ignore interrupt signals
      --help
      display this help and exit
      --version
      output version information and exit


      If a FILE is -, copy again to standard output.




      So in your case you'd run:



      python myprog.py | tee output.txt


      EDIT: As others have pointed out, this answer will run into the same issue OP was originally having unless sys.stdout.flush() is used in the python program as described in Davey's accepted answer. The testing I did before posting this answer did not accurately reflect OP's use-case.



      tee can still be used as an alternative--albeit less than optimal--method of displaying the output while also writing to the file, but Davey's answer is clearly the correct and best answer.







      share|improve this answer














      share|improve this answer



      share|improve this answer








      edited 6 mins ago

























      answered 21 hours ago









      n8ten8te

      5,32972235




      5,32972235












      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        13 hours ago






      • 5





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        12 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        3 hours ago

















      • tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

        – Baldrickk
        13 hours ago






      • 5





        That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

        – eckes
        12 hours ago






      • 3





        This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

        – Barmar
        3 hours ago
















      tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

      – Baldrickk
      13 hours ago





      tail in another thread is a good solution for when you've started the application before you decide you want to see the output though.

      – Baldrickk
      13 hours ago




      5




      5





      That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

      – eckes
      12 hours ago





      That requires a permanent console session, this is why it’s often much easier to use tail -F or even better the follow function of less. But in all cases the flush should be used.

      – eckes
      12 hours ago




      3




      3





      This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

      – Barmar
      3 hours ago





      This won't solve the problem that the OP is having. Python's output to the pipe will be buffered just like output to the file.

      – Barmar
      3 hours ago











      5














      Run python with the unbuffered flag:



      python -u myprog.py > output.txt


      Output will then print in real time.






      share|improve this answer








      New contributor




      BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.
























        5














        Run python with the unbuffered flag:



        python -u myprog.py > output.txt


        Output will then print in real time.






        share|improve this answer








        New contributor




        BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.






















          5












          5








          5







          Run python with the unbuffered flag:



          python -u myprog.py > output.txt


          Output will then print in real time.






          share|improve this answer








          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.










          Run python with the unbuffered flag:



          python -u myprog.py > output.txt


          Output will then print in real time.







          share|improve this answer








          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.









          share|improve this answer



          share|improve this answer






          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.









          answered 5 hours ago









          BHCBHC

          511




          511




          New contributor




          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





          New contributor





          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.






          BHC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.





















              2














              Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



              This is a redirect to a regular file.



              C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



              You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



              On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



              Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




              Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






              share|improve this answer



























                2














                Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



                This is a redirect to a regular file.



                C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



                You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



                On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



                Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




                Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






                share|improve this answer

























                  2












                  2








                  2







                  Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



                  This is a redirect to a regular file.



                  C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



                  You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



                  On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



                  Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




                  Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.






                  share|improve this answer













                  Terminology: There is no pipe anywhere in this scenario. (I edited the question to fix that). Pipes are a different type of file (a buffer inside the kernel).



                  This is a redirect to a regular file.



                  C stdio, and Python, default to making stdout line-buffered when it's connected to a TTY, otherwise it's full-buffered. Line-buffered means the buffer is flushed after a newline. Full-buffered means it's only flushed to become visible to the OS (i.e. with a write() system call) when it's full.



                  You will see output eventually, in chunks of maybe 4kiB at a time. (I don't know the default buffer size.) This is generally more efficient, and means fewer writes to your actual disk. But not great for interactive monitoring, because output is hidden inside the memory of the writing process until it's flushed.



                  On Stack Overflow, there's a Disable output buffering Python Q&A which lists many ways to get unbuffered (or line-buffered?) output to stdout in Python. The question itself summarizes the answers.



                  Options include running python -u (Or I guess putting #!/usr/bin/python -u at the top of your script), or using the PYTHONUNBUFFERED environment variable for that program. Or explicit flushing after some/all print functions, like @Davey's answer suggests.




                  Some other programs have similar options, e.g. GNU grep has --line-buffered, and GNU sed has -u / --unbuffered, for use-cases like this, or for example piping the output of your python program. e.g. ./slowly-output-stuff | grep --line-buffered 'foo.*bar'.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered 2 hours ago









                  Peter CordesPeter Cordes

                  2,4651621




                  2,4651621





















                      0














                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






                      share|improve this answer


















                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        3 hours ago















                      0














                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






                      share|improve this answer


















                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        3 hours ago













                      0












                      0








                      0







                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.






                      share|improve this answer













                      Another option (if you don't care about the contents, just the progress) is pv:



                      NAME
                      pv - monitor the progress of data through a pipe

                      SYNOPSIS
                      pv [OPTION] [FILE]...
                      pv [-h|-V]


                      Introduce this in your pipeline and it will show you the number of bytes processed as well as the speed they go through the pipeline.



                      If the content is what you actually want to monitor, then tee is the best choice, as the other answer indicates.







                      share|improve this answer












                      share|improve this answer



                      share|improve this answer










                      answered 6 hours ago









                      rrauenzarrauenza

                      1313




                      1313







                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        3 hours ago












                      • 1





                        There's no pipeline, just file redirection. And it won't solve the buffering problem.

                        – Barmar
                        3 hours ago







                      1




                      1





                      There's no pipeline, just file redirection. And it won't solve the buffering problem.

                      – Barmar
                      3 hours ago





                      There's no pipeline, just file redirection. And it won't solve the buffering problem.

                      – Barmar
                      3 hours ago

















                      draft saved

                      draft discarded
















































                      Thanks for contributing an answer to Super User!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid


                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.

                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function ()
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1421123%2fwatching-something-be-written-to-a-file-live-with-tail%23new-answer', 'question_page');

                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      Popular posts from this blog

                      名間水力發電廠 目录 沿革 設施 鄰近設施 註釋 外部連結 导航菜单23°50′10″N 120°42′41″E / 23.83611°N 120.71139°E / 23.83611; 120.7113923°50′10″N 120°42′41″E / 23.83611°N 120.71139°E / 23.83611; 120.71139計畫概要原始内容臺灣第一座BOT 模式開發的水力發電廠-名間水力電廠名間水力發電廠 水利署首件BOT案原始内容《小檔案》名間電廠 首座BOT水力發電廠原始内容名間電廠BOT - 經濟部水利署中區水資源局

                      Prove that NP is closed under karp reduction?Space(n) not closed under Karp reductions - what about NTime(n)?Class P is closed under rotation?Prove or disprove that $NL$ is closed under polynomial many-one reductions$mathbfNC_2$ is closed under log-space reductionOn Karp reductionwhen can I know if a class (complexity) is closed under reduction (cook/karp)Check if class $PSPACE$ is closed under polyonomially space reductionIs NPSPACE also closed under polynomial-time reduction and under log-space reduction?Prove PSPACE is closed under complement?Prove PSPACE is closed under union?

                      Is my guitar’s action too high? Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)Strings too stiff on a recently purchased acoustic guitar | Cort AD880CEIs the action of my guitar really high?Μy little finger is too weak to play guitarWith guitar, how long should I give my fingers to strengthen / callous?When playing a fret the guitar sounds mutedPlaying (Barre) chords up the guitar neckI think my guitar strings are wound too tight and I can't play barre chordsF barre chord on an SG guitarHow to find to the right strings of a barre chord by feel?High action on higher fret on my steel acoustic guitar