API Loop Help

[tturner2304]

Hi All

Firstly i am a complete beginner to Python (2 Weeks), but have experience in other languages.
I have the following piece of code which i am trying to build to pull data from an API. The API in question has a 100 record limit per call limit with a total of around 46,000 records. i have managed to build the loop but it is not stopping when there is no more data. is someone able to help to see where i am going wrong

Thanks in advance

# Define the base URL and parameters
base_url = 'https://api_urlhere.com
headers = {
   # 'Authorization': 'Bearer YOUR_API_TOKEN',
    'Content-Type': 'application/json'
}
params = {
    'limit': 100,  # Number of records to fetch per request
    'offset': 0    # Initial offset
}

# Function to fetch data from the API
def fetch_data(url, headers, params):
    response = requests.get(url, headers=headers, params=params)
    if response.status_code == 200:
        return response.json()
    else:
        print(f"Error: {response.status_code}")
        print("Response content:", response.content)
        return None

# Initialize variables
all_data = []
has_more_data = True

# Loop to fetch data with pagination
while has_more_data:
    # Fetch data
    data = fetch_data(base_url, headers, params)
    if data:
        # Debug: Print the response structure
        print(params)
        #print("Response JSON structure:", data)
        # Check if 'results' key exists in the response
        if 'ranking' in data:
            # Process the data (example: extend the all_data list with new records)
            all_data.extend(data['ranking'])
            
            # Check if there is more data to fetch

            if len(data['ranking']) < params['limit']:
                has_more_data = False
                break
            else:
                # Update the offset for the next request
                 params['offset'] += params['limit']
        else:
            print("Key 'ranking' not found in the response")
            has_more_data = False
            break
    else:
        has_more_data = False
        break

# Example: Save all data to a JSON file
with open('ladder.json', 'w') as f:
    json.dump(all_data, f, indent=4)

print("Data fetching complete. Total records fetched:", len(all_data))

with open('ladder.json', encoding='utf-8') as inputfile:
    df = pd.read_json(inputfile)

df.to_csv('ladder.csv', encoding='utf-8', index=False)

[Pedroski55]

Don't know what all the other stuff is or does just from a quick glance. Concerning the while loop: if your while loop doesn't stop then the condition:

has_more_data = False

is not met. Quite why that is is not immediately apparent to me, but if you provide some sample data to work on, that may clarify itself.

As an example with random data:

from string import ascii_lowercase
from random import choices, choice

# reset to run again
all_data = []
# reset to run again
has_more_data = True
# if 1 of these letters is chosen, adieu while loop!
empty = choices(ascii_lowercase, k=3)

# if has_more_data = False the while loop will exit, no need for break
while has_more_data:
    print(f'empty = {empty}')
    # Fetch data
    data = choice(ascii_lowercase)
    if data in empty:
        # Debug: Print the response structure
        has_more_data = False
        print("Leaving the while loop ...")
    else:
        all_data.append(data)
        print(f'all_data = {all_data}')