Python Forum
Upload Files to Azure Storage Container
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Upload Files to Azure Storage Container
#1
Hello,

I found the script online that uploads multiple JPG files. I changed it so I can upload multiple BLOB files, i.e.) csv, xls, etc. However, I am receiving the following error message when I run the script:

Error:
azure.core.exceptions.HttpResponseError: The specifed resource name contains invalid characters. RequestId:62 Time:2023-12-21T15:42:33.3507947Z ErrorCode:InvalidResourceName Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters. RequestId:62 Time:2023-12-21T15:42:33.3507947Z</Message></Error>
The file names are lowercase and alphanumeric: bcp-notes-batch-0.csv

Here is the script:

# Replace with blob container
MY_IMAGE_CONTAINER = "source_system"

# Replace with the local folder which contains the files for upload
LOCAL_FILE_PATH = 'I:/bcp'

class AzureBlobFileUploader:
    def __init__(self):
        print("Initializing AzureBlobFileUploader")
        # Initialize the connection to Azure storage account
        self.blob_service_client = BlobServiceClient.from_connection_string(MY_CONNECTION_STRING)

    def upload_all_files_in_folder(self):
        all_file_names = [f for f in os.listdir(LOCAL_FILE_PATH)
                          if os.path.isfile(os.path.join(LOCAL_FILE_PATH, f))]
        result = self.run(all_file_names)
        print(result)

    def run(self, all_file_names):
        with ThreadPool(processes=int(10)) as pool:
            return pool.map(self.upload_file, all_file_names)

    def upload_file(self, file_name):
        blob_client = self.blob_service_client.get_blob_client(container=MY_IMAGE_CONTAINER, blob=file_name)
        upload_file_path = os.path.join(LOCAL_FILE_PATH, file_name)
        content_settings = ContentSettings(content_type='application/octet-stream')
        print(f"Uploading file - {file_name}")
        with open(upload_file_path, "rb") as data:
            blob_client.upload_blob(data, overwrite=True, content_settings=content_settings)
        return file_name

# Initialize class and upload files
azure_blob_file_uploader = AzureBlobFileUploader()
azure_blob_file_uploader.upload_all_files_in_folder()
Reply
#2
I would start with replacing the list comprehension on line 14 with a regular loop. Catch the exception, and when it happens, print out the filename with repr() in case it has weird things that aren't obvious.
Reply
#3
These are the only two files in the directory. All characters in the name are valid:

Error:
Initializing AzureBlobFileUploader Uploading file - ibrtfnotesbatch00.csv Uploading file - ibrthnotesbatch01.csv Exception uploading file - 'ibrthnotesbatch01.csv': The specifed resource name contains invalid characters. RequestId:70 Time:2023-12-21T18:18:25.7180043Z ErrorCode:InvalidResourceName Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters. RequestId:70 Time:2023-12-21T18:18:25.7180043Z</Message></Error> Exception uploading file - 'ibrtfnotesbatch00.csv': The specifed resource name contains invalid characters. RequestId:28 Time:2023-12-21T18:18:25.7217421Z ErrorCode:InvalidResourceName Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters. RequestId:28 Time:2023-12-21T18:18:25.7217421Z</Message></Error> [None, None]
Reply
#4
According to azure documentation:
Quote:A path name is composed of one or more path name components (directory or file name) separated by the forward-slash (/) character. All path name components other than the last path name component denote directories. The last path name component denotes a directory or a file.
Is the problem that os.path.join(LOCAL_FILE_PATH, file_name) uses '\'?
Reply
#5
(Dec-21-2023, 06:50 PM)deanhystad Wrote: According to azure documentation:
Quote:A path name is composed of one or more path name components (directory or file name) separated by the forward-slash (/) character. All path name components other than the last path name component denote directories. The last path name component denotes a directory or a file.
Is the problem that os.path.join(LOCAL_FILE_PATH, file_name) uses '\'?

Still no luck:

class AzureBlobFileUploader:
    def __init__(self):
        print("Initializing AzureBlobFileUploader")
        self.blob_service_client = BlobServiceClient.from_connection_string(MY_CONNECTION_STRING)

    def upload_all_files_in_folder(self):
        all_file_names = [f for f in os.listdir(LOCAL_FILE_PATH) if os.path.isfile(os.path.join(LOCAL_FILE_PATH, f))]
        result = self.run(all_file_names)
        print(result)

    def run(self, all_file_names):
        with ThreadPool(processes=int(10)) as pool:
            return pool.map(self.upload_file, all_file_names)

    def upload_file(self, file_name):
            try:
                print(repr(file_name))  # Print the repr() of the file name
                blob_client = self.blob_service_client.get_blob_client(container=MY_IMAGE_CONTAINER, blob=file_name)
                upload_file_path = os.path.join(LOCAL_FILE_PATH, file_name).replace('\\', '/')  # Replace backslashes
                print(f"Uploading file - {file_name}")
                with open(upload_file_path, "rb") as data:
                    blob_client.upload_blob(data, overwrite=True)
                return file_name
            except Exception as e:
                print(f"Exception uploading file - {repr(file_name)}: {e}")

    def get_containers(self):
        try:
            containers = self.blob_service_client.list_containers()
            print([container.name for container in containers])
        except Exception as e:
            print(e.message)

# Initialize class and upload files
azure_blob_file_uploader = AzureBlobFileUploader()
azure_blob_file_uploader.get_containers()
azure_blob_file_uploader.upload_all_files_in_folder()
Error:
Initializing AzureBlobFileUploader 'ibrtfnotesbatch00.csv' 'ibrthnotesbatch01.csv' Uploading file - ibrtfnotesbatch00.csv Uploading file - ibrthnotesbatch01.csv Exception uploading file - 'ibrtfnotesbatch00.csv': The specifed resource name contains invalid characters. RequestId:1b87ffae-901e-0077-4242-34969d000000 Time:2023-12-21T19:17:32.3704303Z ErrorCode:InvalidResourceName Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters. RequestId:1b87ffae-901e-0077-4242-34969d000000 Time:2023-12-21T19:17:32.3704303Z</Message></Error> Exception uploading file - 'ibrthnotesbatch01.csv': The specifed resource name contains invalid characters. RequestId:74f78216-701e-0040-3a42-344431000000 Time:2023-12-21T19:17:41.6794092Z ErrorCode:InvalidResourceName Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters. RequestId:74f78216-701e-0040-3a42-344431000000 Time:2023-12-21T19:17:41.6794092Z</Message></Error> [None, None]
Reply
#6
I looked at the problem wrong. The \ has nothing to do with the issue because that is a local filename, not the name of the blob resource. I think our problem is the ".". From the azure docs
Quote:Azure Storage resource names must follow certain rules, such as:

Must be between 3 and 63 characters long.
Must start with a letter or number.
Can only contain lowercase letters, numbers, and hyphens (-).
Must not start or end with a hyphen (-).
Reply
#7
I don't know what azure storage is. If you need to change file names to suit, that can be done.

I believe rsync will work under MS Windoze, but I have never tried.

For syncing files I don't think rsync can be beat. Just use it on the command line, or make a bash script to copy from multiple folders.

You must have write permission on the remote machine.

For example, to send all files from /home/pedro/jpgs to another machine, you will be asked for your user, in this case pedro's, password.

If the remote machine doesn't have the directory /home/pedro/jpgs, it will be created, as will any and all sub directories.

rsync will only send (or receive) files which have changed since the last time you used rsync. This one-liner sends files over my LAN to another laptop, but I could put the address of my little cloud server. I just need to know the ip4 address of the other machine:

Quote:rsync -av -e "ssh" --progress /home/pedro/jpgs [email protected]:/home/pedro/

You can even use rsync to send files to a usb stick!

I believe -e "ssh" is redundant now, as all transfers use ssh, but I leave it there anyway!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  uploading files from a ubuntu local directory to Minio storage container dchilambo 0 471 Dec-22-2023, 07:17 AM
Last Post: dchilambo
  Making a question answering chatbot based on the files I upload into python. Joejones 1 1,248 May-19-2023, 03:09 PM
Last Post: deanhystad
  azure TTS from text files to mp3s mutantGOD 2 1,714 Jan-17-2023, 03:20 AM
Last Post: mutantGOD
Lightbulb shutdown host from docker container cosmin1805 0 962 Nov-27-2022, 06:34 PM
Last Post: cosmin1805
  networkx package is not visible in singularity container image erdemath 11 2,308 Oct-14-2022, 12:04 PM
Last Post: Larz60+
  python installation/running inside singularity container erdemath 2 1,812 Sep-21-2022, 08:13 AM
Last Post: erdemath
  Python in Singularity Container on Ubuntu erdemath 0 914 Aug-31-2022, 02:17 PM
Last Post: erdemath
  Get Azure activity logs using python script raham3406 4 3,596 Apr-27-2021, 05:10 AM
Last Post: raham3406
  Wanted to tag azure resources with creator name raham3406 0 1,460 Apr-25-2021, 02:24 PM
Last Post: raham3406
  How can I upload some files to my website? brunolelli 3 2,281 Mar-17-2021, 01:40 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020