Python Forum
pathlib destpath.exists() true even file does not exist
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
pathlib destpath.exists() true even file does not exist
#1
I have a Windows file server (Win Server 2003 Standard x64 Edition) that contains csv files.

On a Debian machine (9.13 Stretch) I mounted a Windows share pointing to abovementioned machine by adding this line in file /etc/fstab:
//192.168.254.10/DATA /mnt/fsdata cifs uid=postgres,username=<*user*>,password=<*pwd*>,iocharset=utf8,sec=ntlm 0 0

I am running a python script on the Debian machine to check if the csv files exist on the windows machine.

On the windows machine, when I “cut” or “drag” all the files away from folder, and run Path.exists() on /mnt/fsdata/IT/Servers/PostGres/CDR/, the result is always true. (=incorrect)

On the windows machine, when I delete all the files, and run Path.exists() on /mnt/fsdata/IT/Servers/PostGres/CDR/, the result is false. (=correct)

It seems that cut or drag is not recognized on Debian side, and python still “sees” the files as being there.

This is the code excerpt:

#!/usr/bin/env python3

import pathlib

src = "/mnt/fsdata/IT/Servers/PostGres/CDR/"
dest = src + "procd/"

srcpath = pathlib.Path(src)

for file in list(srcpath.glob('*.csv')):
                  destpath = pathlib.Path(dest + file.name)
                  
                  # check if file already exists in /procd folder:
                  if destpath.exists():
                        # something happens...
What can be done to avoid this?

I searched a lot online but found no answers.

Thank you
NaN
Reply
#2
pathlib will return the 'address' of where you want a file to be.
To see if file is there, use exists.

example (untested):
from pathlib import Path

homepath = Path('.')
datapath = homepath / 'data'
datapath.mkdir(exist_ok=True)    # make data directory (only if it is not already there)
myfile = datapath / 'myfile.text'

# check if file exists:
if myfile.exists():
    with myfile.open() as fp:
        data = myfile.read()
else:
    print("myfile.txt does not exist")
Reply
#3
I suspect it has something to do with the way samba servers work, or samba clients. Some answers on the web, such as this one indicate that it could be a cache problem. I'm not proficient in these samba issues, but you can perhaps either fine tune the configuration of the shared directory on the Windows side or find a way for the Linux client to force the server to reread the contents of the directory. This link could be helpful too.
Reply
#4
(Nov-26-2020, 10:47 PM)Gribouillis Wrote: I suspect it has something to do with the way samba servers work, or samba clients. Some answers on the web, such as this one indicate that it could be a cache problem. I'm not proficient in these samba issues, but you can perhaps either fine tune the configuration of the shared directory on the Windows side or find a way for the Linux client to force the server to reread the contents of the directory. This link could be helpful too.

I added cache=none in /etc/fstab and restarted the machine. Nothing changed.
My opinion is that the problem is on python side, because when I ls the folder it is empty. After that, running the py script still gives destpath.exists() as being true.

I added following lines to try to open files:
with destpath.open() as f:
    print("File name: " + str(destpath))
it returns an error:
Error:
An error occurred: [Errno 2] No such file or directory: '/mnt/fsdata/IT/Servers/PostGres/CDR/procd/Trunks-2020-11-01.csv'
Could it be a bug in python? Who can I contact for that?
Reply
#5
You could perhaps first perform a os.stat() call on the file and print all the fields of the returned stat_result object to see what it contains. A bug in Python is by far the least plausible explanation.
Reply
#6
stats are as follows:
Output:
os.stat_result(st_mode=33261, st_ino=1407374883714381, st_dev=41, st_nlink=1, st_uid=118, st_gid=0, st_size=1621, st_atime=1606730312, st_mtime=1606730312, st_ctime=1606730608) os.stat_result(st_mode=33261, st_ino=1125899907003726, st_dev=41, st_nlink=1, st_uid=118, st_gid=0, st_size=6249, st_atime=1606730312, st_mtime=1606730312, st_ctime=1606730608) os.stat_result(st_mode=33261, st_ino=1125899907003737, st_dev=41, st_nlink=1, st_uid=118, st_gid=0, st_size=8594, st_atime=1606730312, st_mtime=1606730312, st_ctime=1606730608)
etc...
Reply
#7
This kind of checks leads very often into problems. More than you think.

# check if file exists:
if myfile.exists():   # <--- file could exist during this moment
    # <--- maybe the file is now deleted, has changed permission or something else.
    with myfile.open() as fp: # <-- will definitely raise an Exception if there is a problem
        data = myfile.read() # <-- could also raise an Exception
Don't ask for permission, ask for forgiveness:
from pathlib import Path


...
myfile = Path("C:")
# try other not working Paths
...


try:
    with myfile.open() as fp:
        data = fp.read()
except FileNotFoundError:
    print("File not found")
except PermissionError:
    print("I do not have the permission to access", myfile)
except UnicodeDecodeError:
    print("Could not decode UTF-8. Binary file or wrong encoding?")
Output:
I do not have the permission to access C:
Try this with binary files and you'll get a UnicodeDecodeError.
Try this with your Samba-Share. It should raise FileNotFoundError, but maybe it's an OSError.
Just try it and observe which Exception you get.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#8
What happens if you try os.listdir(...) or list(os.scandir(...)) and if you mix calls to subprocess.check_output(['ls', ...]) or subprocess.check_output("ls ...", shell=True) in between?
Reply
#9
I tried your suggestion:

from pathlib import Path

src = "/mnt/fsdata/IT/Servers/PostGres/CDR/"
dest = src + "procd"

# define the path
currentDirectory = Path(src)
currentPattern = "*.csv"
destDirectory = Path(dest)

for currentFile in currentDirectory.glob(currentPattern):
    destFile = destDirectory / currentFile.name  
    if destFile.exists():
        try:
          with destFile.open() as fp:
            data = fp.read()
            print(data)
        except FileNotFoundError:
            print("File not found")
        except PermissionError:
            print("I do not have the permission to access", myfile)
        except UnicodeDecodeError:
            print("Could not decode UTF-8. Binary file or wrong encoding?")
I get File not found.
I am the only user manipulating files in these folders. It is a test environment.
What am I doing wrong? Confused
Reply
#10
DeaD_EyE

OK, I tried your "Don't ask for permission, ask for forgiveness"-approach and rewrote my script:
#!/usr/bin/env python3.8

import pathlib
from pathlib import Path

src = "/mnt/fsdata/IT/Servers/PostGres/CDR/"
dest = src + "procd"
dup = src + "dup" 

# define the paths:
currentDirectory = pathlib.Path(src)
destDirectory = pathlib.Path(dest)
dupDirectory = pathlib.Path(dup) 
currentPattern = "*.csv"

for currentFile in currentDirectory.glob(currentPattern):
    destFile = destDirectory / currentFile.name  
    try:
        with destFile.open() as fp:
          data = fp.read()
    except IOError:
        currentFile.rename(destFile)
        print('File moved to procd.')
        continue    
    # Move file to dup folder:
    duppath = dupDirectory / currentFile.name
    currentFile.rename(duppath)
    print('File moved to dup.')
print("Finished !")
When I upload files to /mnt/fsdata/IT/Servers/PostGres/CDR everything works fine.
Files are moved to /procd.
Now, when I drag those files from /procd back to /mnt/fsdata/IT/Servers/PostGres/CDR, nothing happens. The files remain where they are. It is as if the drag and drop never happened! (note that they are not moved to /dup either...)
What I basicaly want to achieve here is that I import files to /CDR folder. Check if they are not already in /procd. (if already in /procd, move them to /dup) Import them into a database. Move them to /procd.
What's wrong?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  UndefinedEnvironmentName: 'extra' does not exist in evaluation environment EarthAndMoon 3 1,621 Oct-09-2023, 05:38 PM
Last Post: snippsat
  Pathlib import not working chriswrcg 9 3,524 May-29-2022, 07:37 PM
Last Post: snippsat
  check if a file exist on the internet and get the size kucingkembar 6 1,714 Apr-16-2022, 05:09 PM
Last Post: kucingkembar
  SQLALCHEMY - Column doesn't exist jamesaarr 9 7,437 Nov-04-2021, 09:20 AM
Last Post: ndc85430
  deleting an empty Dir, using pathlib.Path tester_V 9 5,673 Jul-01-2021, 01:53 PM
Last Post: Gribouillis
  Trying to pathlib instead of os.path tester_V 4 2,430 Jun-22-2021, 04:15 AM
Last Post: tester_V
  path.exists returning True when it shouldn't natha18 0 1,457 Sep-21-2020, 01:04 PM
Last Post: natha18
  Check if a file exists. Pedroski55 5 3,251 Sep-08-2020, 10:01 AM
Last Post: Pedroski55
  p]Why os.path.exists("abc/d") and os.path.exists("abc/D") treat same rajeev1729 1 2,140 May-27-2020, 08:34 AM
Last Post: DeaD_EyE
  Question about if with () or without () / pathlib Tecuma 3 2,157 Apr-02-2020, 10:02 AM
Last Post: Tecuma

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020