Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
os.list dir not working
#1
Hello,

For some reason this isn't working, it shows files list as 0 []. I have 6 .xlsx files in the input_dir. If I put the .xlsx file where the script is it will work, but I need these .xlsx files in their own folder not with the script.

import openpyxl as xl 
import os
import pandas as pd
import xlsxwriter

input_dir = 'C:\\Users\\\work\\comparison'

files = [file for file in os.listdir(input_dir)
         if os.path.isfile(file) and file.endswith(".xlsx")]


for file in files: 
   input_file =  os.path.join(input_dir, file)
   wb1=xl.load_workbook(input_file)
   ws1=wb1.worksheets[0]
Reply
#2
(Jul-29-2020, 01:41 PM)Kristenl2784 Wrote: C:\\Users\\\work\\comparison

3 x backslash

try

C:/Users/work/comparison

Python lets you use OS-X/Linux style slashes "/" even in Windows
Reply
#3
(Jul-29-2020, 02:00 PM)Axel_Erfurt Wrote:
(Jul-29-2020, 01:41 PM)Kristenl2784 Wrote: C:\\Users\\\work\\comparison

3 x backslash

try

C:/Users/work/comparison

Python lets you use OS-X/Linux style slashes "/" even in Windows

I removed the backslash still didn’t work and tried forward slashes and didn’t work
Reply
#4
To debug a problem like this the first thing I would do is verify I am getting a list of files.
input_dir = 'C:/Users/work/comparison'
for file in os.listdir(input_dir):
    print(file, os.path.isfile(file), file.endswith(".xlsx"))
Well, that isn't true. The first thing I would do is replace os with pathlib

from pathlib import Path

input_dir = Path('C:/Users/work/comparison')
for file in input_dir.iterdir():
    print(file, Path.isfile(file), file.endswith(".xlsx"))
Either way the first thing is make sure I am getting a list of files. If that doesn't work nothing else is going to work. If I am getting files I want to verify that my test for spreadsheet files is going to work.

Usually I don't wait for errors to occur to do debugging. If I am unsure about anything I usually write it as test. With your code I probably would have started like this:
import openpyxl as xl 
import os
import pandas as pd
import xlsxwriter
 
input_dir = 'C:/Users/work/comparison'
files = []

for file in os.listdir(input_dir):
    input_file =  os.path.join(input_dir, file)
    print(file, input_file, os.path.isfile(file), file.endswith(".xlsx"))

for file in files: 
   input_file =  os.path.join(input_dir, file)
   wb1=xl.load_workbook(input_file)
   ws1=wb1.worksheets[0]
When I was confident about finding spreadsheet files in the directory I would then remove the test and replace with the list comprehension. When I run this on my computer (with a different input_dir) it was pretty obvious why "files" in your code is an empty list.
Reply
#5
(Jul-29-2020, 02:50 PM)deanhystad Wrote: To debug a problem like this the first thing I would do is verify I am getting a list of files.
input_dir = 'C:/Users/work/comparison'
for file in os.listdir(input_dir):
    print(file, os.path.isfile(file), file.endswith(".xlsx"))
Well, that isn't true. The first thing I would do is replace os with pathlib

from pathlib import Path

input_dir = Path('C:/Users/work/comparison')
for file in input_dir.iterdir():
    print(file, Path.isfile(file), file.endswith(".xlsx"))
Either way the first thing is make sure I am getting a list of files. If that doesn't work nothing else is going to work. If I am getting files I want to verify that my test for spreadsheet files is going to work.

Usually I don't wait for errors to occur to do debugging. If I am unsure about anything I usually write it as test. With your code I probably would have started like this:
import openpyxl as xl 
import os
import pandas as pd
import xlsxwriter
 
input_dir = 'C:/Users/work/comparison'
files = []

for file in os.listdir(input_dir):
    input_file =  os.path.join(input_dir, file)
    print(file, input_file, os.path.isfile(file), file.endswith(".xlsx"))

for file in files: 
   input_file =  os.path.join(input_dir, file)
   wb1=xl.load_workbook(input_file)
   ws1=wb1.worksheets[0]
When I was confident about finding spreadsheet files in the directory I would then remove the test and replace with the list comprehension. When I run this on my computer (with a different input_dir) it was pretty obvious why "files" in your code is an empty list.

I don't think I follow what you're doing or trying to explain. Also, my code works fine if the .xlsx files are in with the code, one directory back. But when I place the .xlsx files one directory up into their own folder, and update the input_dir path it doesn't find any files.

This worked:
files = [file for file in os.listdir(input_dir)
if os.path.isfile(os.path.join(input_dir, file)) and file.endswith(".xlsx")]
Reply
#6
I think your original problem is the os.path.isfile() check. You are just feeding it a filename that doesn't have the path joined to it yet so it's returning false. That's why your last code snippet worked. You're using isfile() on the joined path now.

I agree with dean that pathlib module is preferred here.
"So, brave knights, if you do doubt your courage or your strength, come no further, for death awaits you all with nasty, big, pointy teeth!" - Tim the Enchanter
Reply
#7
What I was trying to communicate (unsuccessfully) is that I think the best way to debug a problem in code is to fully understand the problem. To help me understand the problem I add extra code. Your problem was that file did not have a path, but unless you were really familiar with the workings of os.path you might not know that listdir() returns the filename sans path, or maybe you knew that and the need to add the path got lost in the complication of the list comprehension.

To find the source of a problem when I don't understand exactly what's going on, I add some extra test code that prints information out to help me understand. This code is purely for my information:
for file in os.listdir(input_dir):
    input_file =  os.path.join(input_dir, file)
    print(file, input_file, os.path.isfile(file), file.endswith(".xlsx"))
When I run the program I will see that "file" is just the filename and extension. I will see that the file.endswith(".xlsx") logic works. And I will see that os.path.isfile(file) is False for EVERY FILE. Once I know the "isfile" fails I can limit the scope of my debugging. Maybe for my next test I change "input_dir" to "." and I see that fixes the the isfile() problem. That should trigger my memory that if you don't specify a path, files are opened/created in the current working directory, and that would (hopefully) thinking "Do I need to provide a path for the isfile()?"

I see a lot of good advice in this forum, but I don't see a lot of advice about how to isolate a problem and determine the cause. I was trying to provide the latter type of advice.
Reply
#8
I forgot to add in my last post that in addition to the pathlib module tip I did exactly what deanhystad did and filled your code with lots of print statements to verify what the functions were actually returning rather that assuming it was right. Another option is to use the debug tools in your editor to monitor the state of variables as you step through the program but for simple issues like this just adding some print statements often reveals the errors.
"So, brave knights, if you do doubt your courage or your strength, come no further, for death awaits you all with nasty, big, pointy teeth!" - Tim the Enchanter
Reply
#9
I also think there are times to use list comprehension and times when it is better to use a loop. This is ugly:
files = [file for file in os.listdir(input_dir)
         if os.path.isfile(file) and file.endswith(".xlsx")]
I know it is ugly because later on you used os.path.join so you knew it was needed. The only reason you didn't use it in the list comprehension is that the ungainly syntax hid the need.

This is longer, but it is easier to read:
files = []
for file in os.listdir(input_dir)
    file = os.path.join(input_dir, file)
    if os.path.isfile(file) and file.endswith(".xlsx"):
        files.append(file)
And when you are done with the loop you have a list of filenames with paths.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  python3: iterating through list not working wardancer84 3 2,302 Jul-08-2020, 04:30 PM
Last Post: DPaul
  Finding MINIMUM number in a random list is not working Mona 5 2,986 Nov-18-2019, 07:27 PM
Last Post: ThomasL
  Appending to list not working and causing a infinite loop eiger23 8 3,941 Oct-10-2019, 03:41 PM
Last Post: eiger23
  list not working Zman350x 2 2,326 Mar-10-2018, 12:21 AM
Last Post: Zman350x

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020