Python Forum
What does .flush do? How can I change this to write to the file?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
What does .flush do? How can I change this to write to the file?
#1
Looking at using yield, I have this simulated log file: logsim.py

logsim.py has 2 very long lists, ips and docs

ips like this: ips = ['129.194.8.73', '198.37.27.153', '217.127.12.71', ...
docs like this: docs = ['"GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 -',
'"GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538', ....

The tutorial is here.

logsim.py code, apart from the lists ips and docs:

import time, random
# a fake log programme

f = open("/home/pedro/temp/access-log.log","w")

while True:
    time.sleep(random.random())
    n = random.randint(0,len(ips)-1)
    m = random.randint(0,len(docs)-1)
    t = time.time()
    date = time.strftime("[%d/%b/%Y:%H:%M:%S -0600]",time.localtime(t))
    #print(f,"%s - - %s %s" % (ips[n],date,docs[m]))
    print(f'{f} {ips[n]},{date},{docs[m]}')      
    f.flush()
For the tutorial it says, leave this running in the background, so I run it in bash.
But nothing is written to /home/pedro/temp/access-log.log when I run this in bash

Should I change f.flush() for f.write(ips[n] + ',' + date + ',' + docs[m])

Often, the file does not change until f.close() is called, but I can't do that, because logsim.py is still writing to f!

How to get the data actually written to /home/pedro/temp/access-log.log without closing the file?
How to dynamically update the file, without closing it?
Reply
#2
Try
print(f'{ips[n]},{date},{docs[m]}', file=f)
f.flush()
« We can solve any problem by introducing an extra level of indirection »
Reply
#3
Can also write it like this,updatet for Python 3.
log_sim.py:
import time
import random

ips = [
    '99.167.100.246', '74.6.23.48', '77.91.224.3', '67.176.147.11',
    '84.237.120.134', '64.92.161.138', '87.244.30.18', '84.110.229.30',
    '201.213.27.208', '80.99.119.201', '74.6.28.156', '213.217.40.133',
    '62.224.58.211', '74.6.28.151', '65.166.139.21', '192.25.206.10',
    '66.79.171.46', '80.91.229.6', '201.63.117.142', '144.51.43.161',
    '200.13.243.76', '213.67.242.79', '159.149.89.40', '41.222.70.189',
    '201.18.38.242', '220.239.245.127', '74.6.26.203', '216.160.75.249',
    '203.101.103.2', '74.6.24.228'
]

docs = [
    '"GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 -',
    '"GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538',
]

with open("access-log", "w") as f:
    while True:
        time.sleep(random.random())
        n = random.randint(0, len(ips) - 1)
        m = random.randint(0, len(docs) - 1)
        t = time.time()
        date = time.strftime("[%d/%b/%Y:%H:%M:%S -0600]", time.localtime(t))
        f.write(f"{ips[n]} - - {date} {docs[m]}\n")
        f.flush()
follow.py:
import time

def follow(thefile):
    thefile.seek(0, 2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

if __name__ == '__main__':
    with open("access-log", "r") as logfile:
        for line in follow(logfile):
            # print function in Python 3
            print(line, end='')
Test make sure that start log_sim.py first.
Output:
G:\div_code\reader_env λ python follow.py 203.101.103.2 - - [22/Apr/2024:13:31:37 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 144.51.43.161 - - [22/Apr/2024:13:31:38 -0600] "GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 - 84.237.120.134 - - [22/Apr/2024:13:31:38 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 67.176.147.11 - - [22/Apr/2024:13:31:38 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 159.149.89.40 - - [22/Apr/2024:13:31:39 -0600] "GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 - 213.217.40.133 - - [22/Apr/2024:13:31:39 -0600] "GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 - 220.239.245.127 - - [22/Apr/2024:13:31:40 -0600] "GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 - 66.79.171.46 - - [22/Apr/2024:13:31:40 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 201.18.38.242 - - [22/Apr/2024:13:31:40 -0600] "GET /ply/ply-2.0.tar.gz HTTP/1.0" 304 - 99.167.100.246 - - [22/Apr/2024:13:31:41 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 74.6.24.228 - - [22/Apr/2024:13:31:41 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 87.244.30.18 - - [22/Apr/2024:13:31:42 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 77.91.224.3 - - [22/Apr/2024:13:31:42 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 192.25.206.10 - - [22/Apr/2024:13:31:42 -0600] "GET /python/tutorial/beazley_advanced_python/Slides/SLIDE066.HTM HTTP/1.0" 200 1538 Traceback (most recent call last): File "G:\div_code\reader_env\follow.py", line 14, in <module> for line in follow(logfile): File "G:\div_code\reader_env\follow.py", line 8, in follow time.sleep(0.1) KeyboardInterrupt ^C G:\div_code\reader_env
Reply
#4
Also for fun as this is written to be like tail -f which is command line tool.
Then can write to be a command line tool,eg i think Typer is cool.
So Typer is tool like eg argparse in standard library.
# py_tail.py
import typer
from collections import deque
import time

app = typer.Typer()

def follow_file(filename: str) -> None:
    with open(filename, 'r') as fh:
        # Go to the end of the file
        fh.seek(0, 2)
        while True:
            line = fh.readline()
            if not line:
                time.sleep(1)  # Wait a bit for new content
                continue
            print(line.rstrip('\n'))

@app.command()
def tail(
    filename: str,
    n: int = typer.Option(10, "--number", "-n", help="The last n lines of file"),
    follow: bool = typer.Option(False, "--follow", "-f", help="Show data as the file grows.")
    ):
    """
    Print the last n lines from a file. Optionally follow the file for new lines.
    """
    lines = deque(maxlen=n)
    try:
        with open(filename) as fh:
            for line in fh:
                lines.append(line.rstrip('\n'))
        for line in lines:
            print(line)
    except FileNotFoundError:
        typer.echo(f"File not found: {filename}")
        raise typer.Exit(code=1)
    if follow:
        follow_file(filename)

if __name__ == "__main__":
    app()
Now it's a command line tool and have also added eg tail -n 5 show the n lines of the file.
See that color is added automatic(it use Rich) which make it a good looking tool.
[Image: qUXPoG.png]
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Last record in file doesn't write to newline gonksoup 3 535 Jan-22-2024, 12:56 PM
Last Post: deanhystad
  write to csv file problem jacksfrustration 11 1,762 Nov-09-2023, 01:56 PM
Last Post: deanhystad
  python Read each xlsx file and write it into csv with pipe delimiter mg24 4 1,693 Nov-09-2023, 10:56 AM
Last Post: mg24
  logging: change log file permission with RotatingFileHandler erg 0 1,157 Aug-09-2023, 01:24 PM
Last Post: erg
  How can I change the uuid name of a file to his original file? MaddoxMB 2 1,022 Jul-17-2023, 10:15 PM
Last Post: Pedroski55
  How do I read and write a binary file in Python? blackears 6 7,608 Jun-06-2023, 06:37 PM
Last Post: rajeshgk
  Reading data from excel file –> process it >>then write to another excel output file Jennifer_Jone 0 1,186 Mar-14-2023, 07:59 PM
Last Post: Jennifer_Jone
  Read text file, modify it then write back Pavel_47 5 1,773 Feb-18-2023, 02:49 PM
Last Post: deanhystad
  unittest generates multiple files for each of my test case, how do I change to 1 file zsousa 0 1,010 Feb-15-2023, 05:34 PM
Last Post: zsousa
  how to read txt file, and write into excel with multiply sheet jacklee26 14 10,614 Jan-21-2023, 06:57 AM
Last Post: jacklee26

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020