Python Forum
Copy only hidden files and folders with rsync
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Copy only hidden files and folders with rsync
#1
New to Python but not to coding.

I would like to copy only the hidden files and folders in my home directory with rsync from a Python script.
The issues is that using a .* after the source directory name (/home/jer/.*) causes an rsync error when executed from Python.
However, in the Linux terminal, this statement works fine:

rsync -an /home/jer/.*/ /Temp

After some research, I tried the following syntax:

rsync -an /home/jer/.[^.]*/ /Temp
This also causes an rsync error but in the Linux terminal also works fine.

Below is my relevant code snippet:

bk_args = "-avn"
bk_src_dir = "/home/jer/.[^.]*/"
bk_tar_dir = "/mnt/md0"
bk_options = ['--itemize-changes=%i%f', '--info=stats2', '--progress']

try:
    subprocess.run(["rsync", bk_args, *bk_options, bk_src_dir, bk_tar_dir])
This causes the following rsync error:
sending incremental file list
rsync: [sender] change_dir "/home/jer/.[^.]*" failed: No such file or directory (2)
I have tried various combinations of escaping ".*" without success so I thought it was time to come to the mountain for some ideas!
Reply
#2
The * in your shell is replaced with filenames, and then your shell starts the program with the arguments. You have to do this step manually.

from pathlib import Path

bk_args = "-avn"


# Path("somepath").glob(".*") return a Generator
# the elements are Path objects
# they need to be str to use them with subprocess as arguments
bk_src_files = list(map(str, Path("/home/jer").glob(".*")))

bk_tar_dir = "/mnt/md0"
bk_options = ['--itemize-changes=%i%f', '--info=stats2', '--progress']

print(bk_options)

subprocess.run(["rsync", bk_args, *bk_options, *bk_src_dir, bk_tar_dir])
Larz60+ likes this post
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
(Mar-04-2023, 01:20 PM)DeaD_EyE Wrote: The * in your shell is replaced with filenames, and then your shell starts the program with the arguments. You have to do this step manually.

from pathlib import Path

bk_args = "-avn"


# Path("somepath").glob(".*") return a Generator
# the elements are Path objects
# they need to be str to use them with subprocess as arguments
bk_src_files = list(map(str, Path("/home/jer").glob(".*")))

bk_tar_dir = "/mnt/md0"
bk_options = ['--itemize-changes=%i%f', '--info=stats2', '--progress']

print(bk_options)

subprocess.run(["rsync", bk_args, *bk_options, *bk_src_dir, bk_tar_dir])

Brilliant solution DeaD_EyE! This thorn in my side finally resolved with your help!

For completeness, I wanted to provide another solution for anyone that may stumble onto this thread in the future.
The following code will also work but involves explicitly calling the shell in the subprocess.
Note: bk_src_dir = "/home/jer/.*/"

bk_str = "rsync " +  bk_args + " " + bk_options + " " + bk_src_dir + " " + bk_tar_dir
subprocess.run(bk_str, shell=True) 
I did not want to use the explicit call to the shell in the subprocess so did not implement this solution.

Thanks again!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Using zipfile module - finding folders not files darter1010 2 282 Apr-06-2024, 07:22 AM
Last Post: Pedroski55
  Why is the copy method name in python list copy and not `__copy__`? YouHoGeon 2 287 Apr-04-2024, 01:18 AM
Last Post: YouHoGeon
  Copy Paste excel files based on the first letters of the file name Viento 2 455 Feb-07-2024, 12:24 PM
Last Post: Viento
  Create new folders and copy files cocobolli 3 1,481 Mar-22-2023, 10:23 AM
Last Post: Gribouillis
  Can ZipFile be used to extract hidden files? AiedailEclipsed 0 1,637 Mar-22-2022, 05:21 PM
Last Post: AiedailEclipsed
  Compare filename with folder name and copy matching files into a particular folder shantanu97 2 4,509 Dec-18-2021, 09:32 PM
Last Post: Larz60+
  Moving files to Folders giddyhead 13 9,192 Mar-07-2021, 02:50 AM
Last Post: giddyhead
  code to read files in folders and transfer the file name, type, date created to excel Divya577 0 1,870 Dec-06-2020, 04:14 PM
Last Post: Divya577
  Copy files from subfolders into same name of subfolders at other directory rathoreanil 1 2,354 Oct-12-2020, 01:30 AM
Last Post: Larz60+
  need to define date range of files to copy over OTH 4 3,139 Aug-07-2020, 12:29 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020