How can I add a progress bar for my software?

[aquerci]

Hi all,

I wrote a software in Python 3 with a CLI interface. It reads a text file given by the user, takes from it many information, processes them, and produces an excel file with the same information but written in a "human language".

Unfortunately my script is a little bit slowly. For example, to process a text file with 76000 lines it takes around 3 minutes.

To complete the software I wanted to add a progress bar but how can I do it? My source cose has composed by around 1000 lines, with many cicles and custom functions. Can you give me a simple examples please? I need a step by step procedure.

Keep in mind taht I'm still a beginner with the Python. It has been my first and only programming language that I learnt.

[perfringo]

I remember the days when “beginner” was something which didn’t comprise ability to write 1000 lines of code with CLI interface....

[buran]

I remember the days when “beginner” was something which didn’t comprise ability to write 1000 lines of code with CLI interface....

well, I don't know if this is the case with OP code, but actually not so difficult to imagine it - e.g. just imagine some huge repetitive if blocks.

@OP: look at https://pypi.org/project/progressbar2/

also Click has built-in progressbar support https://click.palletsprojects.com/en/7.x...gress-bars
I don't know how you created the CLI

[jefsummers]

Agree with Buran, don't recommend progress as that does not seem to support Python 3.7+.
Install progressbar2 by pip install progressbar2
Then can create the progress bar, and update it every few thousand lines processed.

import time
import progressbar
for i in progressbar.progressbar(range(10)) :
    time.sleep(1)
print("Done")

[Gribouillis]

There is another module named tqdm that could be worth exploring.

Also if you're in linux, you could try pythondialog.

import dialog
import time


if __name__ == '__main__':
    d = dialog.Dialog()
    d.set_background_title('Spam')
    d.gauge_start('Spammish Inquisition', percent=0)
    for i in range(0, 101,):
        d.gauge_update(i)
        time.sleep(0.1)
    d.gauge_stop()

[snippsat]

tqdm(updated for Python 3.7) as mention bye @Gribouillis.
I have used tqdm several time,it's easy to plug into exiting code,
here a example as was downloading a lot of planes a progress bar did help.

[aquerci]

hi guys and thanks a lot for your quickly replies. I followed your advices and I tried to study how the "tqdm" library works but I really don't understand how to use it with my script.

from what I understood with tqdm I can create a progress bar to check a "for" loop in real time. I wrote an example below:

text = "hola\nciao\nhi\nsalut\nпривет"

import time
from tqdm import tqdm

for line in tqdm(text.split()):
    time.sleep(2)

for each line in "text", the script waits 2 seconds. when I run it the progress bar created by tqdm go ahead every 2 seconds and in 10 seconds it covers all the bar.

ok, it's simple, but what about if I have many "for" loops in my script? most of my code is composed by many "for" loops. maybe I could apply tqdm for all of them, but how can I do it and at the same time have an unique loading bar?

then I have another dubt. even if I applied qtdm for each "for" loop, what about the rest of the code? I have some "while" loops too, and other "if/elif/else" instructions. they help me to read some text files, and create new variables.

maybe I could simply print an empty progress bar at the first time and then full it every some pice of code read, i don't know.

I remember the days when “beginner” was something which didn’t comprise ability to write 1000 lines of code with CLI interface....

yeah, let me explain me better ahah.. in Python I know many topics but i'm not an expert. I read the free book "Think Python" and I followed many tutorials online. now I can do many things but currently I can't still program with the classes. I'm still studying.

[snippsat]

ok, it's simple, but what about if I have many "for" loops in my script? most of my code is composed by many "for" loops. maybe I could apply tqdm for all of them, but how can I do it and at the same time have an unique loading bar?

You add to where it make sense usually one or two bar's,or it can soon get confusing to watch.
To test my code that poster link to,to see if it still work as it's 1,5 year sine i wrote it.
This is a example of two bar's working together,one main that stop at 72(all planes downloaded).
Then a bar for each plane type eg 18/18 will count 1 on the way to 72(all planes).

from bs4 import BeautifulSoup
import requests
from tqdm import tqdm, trange
from itertools import islice
 
def all_planes():
    '''Generate url links for all planes'''
    url = 'http://web.archive.org/web/20041225023002/http://www.projectai.com:80/libraries/acfiles.php?cat=6'
    url_get = requests.get(url)
    soup = BeautifulSoup(url_get.content, 'lxml')
    td = soup.find_all('td', width="50%")
    plain_link = [link.find('a').get('href') for link in td]
    for ref in tqdm(plain_link):
         url_file_id = 'http://web.archive.org/web/20041114195147/http://www.projectai.com:80/libraries/{}'.format(ref)
         yield url_file_id
 
def download(all_planes):
    '''Download zip for 1 plain,feed with more url download all planes'''
    # A_300 = next(all_planes())  # Test with first link
    last_3 = islice(all_planes(), 65, 72)
    for plane_url in last_3:
        url_get = requests.get(plane_url)
        soup = BeautifulSoup(url_get.content, 'lxml')
        td = soup.find_all('td', class_="text", colspan="2")
        zip_url = 'http://web.archive.org/web/20041108022719/http://www.projectai.com:80/libraries/download.php?fileid={}'
        for item in tqdm(td):
            zip_name = item.text
            zip_number = item.find('a').get('href').split('=')[-1]
            with open(zip_name, 'wb')  as f_out:
                down_url = requests.get(zip_url.format(zip_number))
                f_out.write(down_url.content)
 
if __name__ == '__main__':
    download(all_planes)

[aquerci]

I see, so with tqdm am I not able to create a single progress bar for each "for" loop in my code? today, I wrote this progress bar, it's a really simple function but, it works. let me know what do you think:

import os
import math
from datetime import datetime, timezone

# it can clear the terminal screen:
def clear(): 
    # for windows 
    if os.name == 'nt': 
        _ = os.system('cls') 
    # for mac and linux(here, os.name is 'posix') 
    else: 
        _ = os.system('clear')

# it can create a progress bar in the terminal:
def progress_bar(header, current_time, number):
    remains = 60 - number
    percentage = str(math.ceil(100*number//60))
    progress_bar = "\nLoading: [" + "#"*number + " "*remains + "]"
    clear()
    print(header, progress_bar, percentage + "% -", current_time + ",", datetime.utcnow().strftime('%H:%M:%S'))

to use the "progress_bar" function you have to call it many times in the script. for example:

from datetime import datetime, timezone
current_time = datetime.utcnow().strftime('%H:%M:%S')

menu = "-------------------\nEXAMPLE\n-------------------\ntext..\n"

progress_bar(menu, current_time, 0)
# code ..
# code ..
progress_bar(menu, current_time, 4)
# code ..
# code ..
progress_bar(menu, current_time, 8)
# code ..
# code ..
# .......
progress_bar(menu, current_time, 60)

I used this progress bar in my script and it works fine. you can see it in the attachments. unfortunately the percentuage shown by my progress bar is not the real percentuage, it depends by how many times did you use the function in the script and where, but it's ok, it doesn't matter. the main goal is to see the progress about my software.

thanks to this progress bar I could see where my script is slow. probably i will open another topic to understand better how to improve that pice of code, and if it's possible also how to use more than one CPU with python.