API Loop Help

[tturner2304]

Hi All

Firstly i am a complete beginner to Python (2 Weeks), but have experience in other languages.
I have the following piece of code which i am trying to build to pull data from an API. The API in question has a 100 record limit per call limit with a total of around 46,000 records. i have managed to build the loop but it is not stopping when there is no more data. is someone able to help to see where i am going wrong

Thanks in advance

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

# Define the base URL and parameters base_url = 'https://api_urlhere.com headers = {    # 'Authorization': 'Bearer YOUR_API_TOKEN',     'Content-Type': 'application/json' } params = {     'limit': 100,  # Number of records to fetch per request     'offset': 0    # Initial offset }   # Function to fetch data from the API def fetch_data(url, headers, params):     response = requests.get(url, headers=headers, params=params)     if response.status_code == 200:         return response.json()     else:         print(f"Error: {response.status_code}")         print("Response content:", response.content)         return None   # Initialize variables all_data = [] has_more_data = True   # Loop to fetch data with pagination while has_more_data:     # Fetch data     data = fetch_data(base_url, headers, params)     if data:         # Debug: Print the response structure         print(params)         #print("Response JSON structure:", data)         # Check if 'results' key exists in the response         if 'ranking' in data:             # Process the data (example: extend the all_data list with new records)             all_data.extend(data['ranking'])                           # Check if there is more data to fetch               if len(data['ranking']) < params['limit']:                 has_more_data = False                 break             else:                 # Update the offset for the next request                  params['offset'] += params['limit']         else:             print("Key 'ranking' not found in the response")             has_more_data = False             break     else:         has_more_data = False         break   # Example: Save all data to a JSON file with open('ladder.json', 'w') as f:     json.dump(all_data, f, indent=4)   print("Data fetching complete. Total records fetched:", len(all_data))   with open('ladder.json', encoding='utf-8') as inputfile:     df = pd.read_json(inputfile)   df.to_csv('ladder.csv', encoding='utf-8', index=False)

[Pedroski55]

Don't know what all the other stuff is or does just from a quick glance. Concerning the while loop: if your while loop doesn't stop then the condition:

1	has_more_data = False

is not met. Quite why that is is not immediately apparent to me, but if you provide some sample data to work on, that may clarify itself.

As an example with random data:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

from string import ascii_lowercase from random import choices, choice   # reset to run again all_data = [] # reset to run again has_more_data = True # if 1 of these letters is chosen, adieu while loop! empty = choices(ascii_lowercase, k=3)   # if has_more_data = False the while loop will exit, no need for break while has_more_data:     print(f'empty = {empty}')     # Fetch data     data = choice(ascii_lowercase)     if data in empty:         # Debug: Print the response structure         has_more_data = False         print("Leaving the while loop ...")     else:         all_data.append(data)         print(f'all_data = {all_data}')