validate large json file with millions of records in batches

[herobpv]

Greetings...

I'm working on validating large json file with millions of records in batches using jsonschema Draft202012Validator.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

import json from azure.storage.blob import BlobServiceClient from azure.identity import DefaultAzureCredential   account = "myAccount" container = "myContainer" blob_name = "myBlob.json"   default_credential = DefaultAzureCredential() blob_service_client = BlobServiceClient(account, credential=default_credential) container_client = blob_service_client.get_container_client(container) blob_client = container_client.get_blob_client(blob_name) data = bytearray(blob_client.download_blob().readall())       batch_size = 1000       process = json.loads(data)       for batch in [process[i:i+batch_size] for i in range(0, len(process), batch_size)]:            # process data

I'm running into convert object of type bytearray to string JSON serializable error.

[bowlofred]

What type of data is coming from the .readall()? Does it look like ascii?

By default loads() will try to decode the data as UTF-8, but you can force a different decode if that is better.

[herobpv]

What type of data is coming from the .readall()? Does it look like ascii?

By default loads() will try to decode the data as UTF-8, but you can force a different decode if that is better.

Hello, The data is actually UTF-8.

[bowlofred]

What's the error you're getting?