Downloading time zone aware files, getting wrong files(by date))s

[tester_V]

Greetings!
With a lot of help from this forum, I finally made a script for downloading (yesterday in Kuala_Lumpur) my today's files.
Still, my script downloads files with the wrong date. It looks OK but it does not really work as intended .
I'm comparing the dates of the files and if matched - download...
Here is a snipped for downloading files:

from datetime import datetime, date, time, timedelta
from zoneinfo import ZoneInfo
from pathlib import Path
import shutil

dt_to_find = dt.datetime.now().date() 
print(f" ========================= Target Day {dt_to_find} ========================= ")

from_timezone = ZoneInfo('US/Pacific')
to_timezone   = ZoneInfo('Asia/Kuala_Lumpur')

itmes = '\\\\somehost\\c$\\directory\\'  # <--- Some Host in Asia/Kuala_Lumpur 
ndir_p = 'D:\\somedir\\'     # <--------------- Directory for files from Kuala_Lumpur

for echf in Path(items).iterdir():	
	if echf.is_file() :                               
        # Converting Time stamp to Date(Kuala_Lumpur)
        ef_time = echf.stat().st_mtime  # <-------- Getting timeStamp
        dt3 = datetime.fromtimestamp(ef_time, to_timezone)
        dt33 = dt3.date()  # <--------------------- Getting Date to match
        # Comparing two dates, Target date with the date from the Timestamp
        if dt33 == dt_to_find :
            try :
                print(f" {dt33} <-Time_ST.. {echf} .. Copying file")
                shutil.copy2(echf,ndir_p)                           
            except OSError as st :
                print(f" -- {st}")  

Any help is appreciated.
Thank you!

[deanhystad]

What is wrong?

It looks OK but it does not really work as intended

is not much of a description.

[tester_V]

For some reason, I do get duplicate files. I added one more script to keep a record of downloaded files and I see I'm downloading some of the files again on the next day... That is really puzzling...

Thank you.

[deanhystad]

There are a few things wrong.

1. You are downloading files modified today. You can't do that. Today is not a 24 hour period. The duration of today is now - start of day. You should download files that were modified yesterday. By using now() you risk missing files or copying a file multiple times based on when you run the script.

2. dt_to_find is in the local timezone, but dt_to_find is in Kualua Lumpur timezone. These cannot be direclty compared.

Your program should look like this:

import pendulum
from pathlib import Path
import shutil
 
itmes = '\\\\somehost\\c$\\directory\\'  # <--- Some Host in Asia/Kuala_Lumpur 
ndir_p = 'D:\\somedir\\'     # <--------------- Directory for files from Kuala_Lumpur

yesterday = pendulum.yesterday().timestamp()
today = pendulum.today().timestamp()
for echf in Path(items).iterdir():  
    if echf.is_file() :                               
        if yesterday <= echf.stat().st_mtime < today:  # There will be zero gap and zero overlap if you run this once every day.
            try :
                shutil.copy2(echf, ndir_p)                           
            except OSError as st :
                print(f" -- {st}")  

If you want more recent files you can set today and yesterday using the Kuala Lumpur timezone. Just make sure you wait until after midnight in Kuala Lumpur before running the script.

[tester_V]

I see your point.
I just cannot install 'pendulum' on my server... Getting too many errors, I'm behind a corporate firewall.
I have to use "from zoneinfo import ZoneInfo"
Thank you.

[tester_V]

I made some changes and replaced Date (local date)

dt_to_find = dt.datetime.now().date() 
print(f" ========================= Target Day {dt_to_find} ========================= ")

with Kuala_Lumpur Date as you suggested.

kul_tz = datetime.now(tz=ZoneInfo('Asia/Kuala_Lumpur'))
print(' DateTime now Kuala_Lumpur ',kul_tz)
dt_to_find = kul_tz.date() - timedelta(1)
print(' DateTime Yesterday Kuala_Lumpur ',dt_to_find)

Do you think it would do the trick?

from datetime import datetime, date, time, timedelta
from zoneinfo import ZoneInfo
from pathlib import Path
import shutil

to_timezone   = ZoneInfo('Asia/Kuala_Lumpur')

kul_tz = datetime.now(tz=ZoneInfo('Asia/Kuala_Lumpur'))
print(' DateTime now Kuala_Lumpur ',kul_tz)
dt_to_find = kul_tz.date() - timedelta(1)
print(' DateTime Yesterday Kuala_Lumpur ',dt_to_find)
print(f" ========================= Target Day {dt_to_find} ========================= ")

itmes = '\\\\somehost\\c$\\directory\\'  # <--- Some Host in Asia/Kuala_Lumpur 
ndir_p = 'D:\\somedir\\'     # <--------------- Directory for files from Kuala_Lumpur

for echf in Path(items).iterdir():	
	if echf.is_file() :                               
        # Converting Time stamp to Date(Kuala_Lumpur)
        ef_time = echf.stat().st_mtime  # <-------- Getting timeStamp
        dt3 = datetime.fromtimestamp(ef_time, to_timezone)
        dt33 = dt3.date()  # <--------------------- Getting Date to match
        # Comparing two dates, Target date with the date from the Timestamp
        if dt33 == dt_to_find :
            try :
                print(f" {dt33} <-Time_ST.. {echf} .. Copying file")
                shutil.copy2(echf,ndir_p)                           
            except OSError as st :
                print(f" -- {st}") 

Thank you.

[deanhystad]

If you can't install pendulum you could fall back to constructing a start of day timestamp.

from datetime import datetime, date, time, timedelta
from zoneinfo import ZoneInfo
from pathlib import Path
 
items = '/somehost/c$/directory/'  # <--- Some Host in Asia/Kuala_Lumpur 
ndir_p = 'D:somedir/'     # <--------------- Directory for files from Kuala_Lumpur

# Get timestamps for start of today and start of yesterday.
# Use tzinfo=ZoneInfo('Asia/Kuala_Lumpur') to get these for Kuala Lumpur.
today = datetime.combine(date.today(), time(), tzinfo=ZoneInfo("America/Vancouver"))
yesterday = (today - timedelta(hours=24)).timestamp()
today = today.timestamp()
for echf in Path(items).iterdir():
    if echf.is_file():                           
        if yesterday <= echf.stat().st_mtime < today:  # There will be zero gap and zero overlap if you run this once every day.
            try:
                shutil.copy2(echf, ndir_p)                           
            except OSError as st :
                print(f" -- {st}")

[tester_V]

Thank you for your help! I really appreciate it!
Someone shared this construct with me before:

today = datetime.combine(date.today(), time(), tzinfo=ZoneInfo("America/Vancouver"))

I just thought filtering files by the Date only would be enough. I'll go with what you say...
You ARE DA MAN!

Thank you for sharing and coaching!

[tester_V]

Have one more question.
If I ran both snippets I have different days?
I'm not worrying about it but would be nice to understand it.
Would you elaborate?
Snipped one:

today = datetime.combine(date.today(), time(), tzinfo=ZoneInfo("Asia/Kuala_Lumpur"))
print(f" Today in KL - {today} ")

Prints:
Today in KL - 2023-07-22 00:00:00+08:00
Snipped two:

kul_tz = datetime.now(tz=ZoneInfo('Asia/Kuala_Lumpur')) # <-- DateTime NOW in Kuala_Lumpur
print(' DateTime Kuala_Lumpur now ',kul_tz)

Prints:
DateTime Kuala_Lumpur now 2023-07-23 14:41:27.687291+08:00

I would expect to have both dates for 'today's' date to be 2023-07-23 because it is "today" in Kuala Lumpur.
I'm missing something...
If it is not too much to ask...
Thank you again...
Tester_V

[deanhystad]

Dates are affected by timezones too.

date.today() returns the current date for the timezone you are in. datetime.now(tz=timezone).date() returns the current date for the specified timezone. If you are in Seattle, these will generate the same date for 9 hours each day, but for 15 hours each day "today" in Kuala Lumpur is "tomorrow" in Seattle. If you want to make your computer act like it is in Kuala Lumpur you need to specify the timezone for the date and the datetime object.

kl = ZoneInfo("Asia/Kuala_Lumpur")
start_of_today = datetime.combine(datetime.now(tz=kl), time(), tzinfo=kl)