Python Forum
Unable to use random.choice(list) in async method
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Unable to use random.choice(list) in async method
#1
I need to pull a random proxy from a list in an async method but the code exits the method as soon as it hits the line of code:
proxy = random.choice(proxy_list)
Here's the full method and the next line (the "print" line) never executes.
async def download_site(session, url):
    proxy_list = [
    (754, '38.39.205.220:80'),
    (681, '38.39.204.100:80'),
    (682, '38.39.204.101:80'),
    (678, '38.39.204.94:80')
    ]
    proxy = random.choice(proxy_list)
    print(proxy[1])
    async with session.get(url, proxy="http://" + proxy[1]) as response:
        print("Read {0} from {1}".format(response.content_length, url))
Can someone please share a work-around or a fix for this?
Thanks.
Reply
#2
But how are you calling that function? Unless you have something await on it, it won't be scheduled to execute. Can you post the full code? How do you know that is the line that is causing problems? Do you have a traceback?

(It seems to run that section for me when I call it via asyncio.run())
Reply
#3
This works

import asyncio
import random as rnd

async def download_site():
    proxy_list = [
        (754, '38.39.205.220:80'),
        (681, '38.39.204.100:80'),
        (682, '38.39.204.101:80'),
        (678, '38.39.204.94:80')
        ]
    await asyncio.sleep(1)
    proxy = rnd.choice(proxy_list)
    print(proxy)

asyncio.run(download_site())
Output:
(682, '38.39.204.101:80')
I welcome all feedback.
The only dumb question, is one that doesn't get asked.
My Github
How to post code using bbtags


Reply
#4
Thanks @bowlofred and @menator01. This is a small app and I should have just posted all the code the first time.

I added the: await asyncio.sleep(1)
and it still did not work. Please see the code below and THANK YOU very much.

Line 34 does not execute.
Also, once I get this working, I would like to move the list out of this method can get the proxy from the method "get_proxy():

import asyncio
import time
import aiohttp

# This code taken from:
# https://realpython.com/python-concurrency/#what-is-parallelism

# Info for adding headers for the proxy
# https://docs.aiohttp.org/en/stable/client_advanced.html

# Metod to provide random proxies
def get_proxy(self):
    proxy_list = [
    (754, '38.39.205.220:80'),
    (681, '38.39.204.100:80'),
    (682, '38.39.204.101:80'),
    (678, '38.39.204.94:80')
    ]
    proxy = random.choice(proxy_list)
    print(proxy[1])
    return proxy

async def download_site(session, url):
    #For test, use this rather than calling get_proxy
    proxy_list = [
    (754, '38.39.205.220:80'),
    (681, '38.39.204.100:80'),
    (682, '38.39.204.101:80'),
    (678, '38.39.204.94:80')
    ]
    await asyncio.sleep(1)
    proxy = random.choice(proxy_list)
    # This line does not execute
    print(proxy[1])
    async with session.get(url, proxy="http://" + proxy[1]) as response:
        print("Read {0} from {1}".format(response.content_length, url))


async def download_all_sites(sites):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in sites:
            task = asyncio.ensure_future(download_site(session, url))
            tasks.append(task)
        await asyncio.gather(*tasks, return_exceptions=True)

if __name__ == "__main__":
    sites = [
        "https://www.jython.org",
       # "http://olympus.realpython.org/dice",
    ] #* 80
    start_time = time.time()
    asyncio.get_event_loop().run_until_complete(download_all_sites(sites))
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} sites in {duration} seconds")
Note: I modified the main method to return a list of one item rather than 160 to make it easier to step through and debug.
Reply
#5
thanks all
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  unable to remove all elements from list based on a condition sg_python 3 373 Jan-27-2024, 04:03 PM
Last Post: deanhystad
  Sample random, unique string pairs from a list without repetitions walterwhite 1 401 Nov-19-2023, 10:07 PM
Last Post: deanhystad
Question mypy unable to analyse types of tuple elements in a list comprehension tomciodev 1 427 Oct-17-2023, 09:46 AM
Last Post: tomciodev
  combobox_callback(choice choice part of openfile name (kind of dependency) janeik 9 1,348 Sep-10-2023, 10:27 PM
Last Post: janeik
  queue for async function python telegram.ext noctious 0 1,512 Jun-11-2023, 02:58 PM
Last Post: noctious
  find random numbers that are = to the first 2 number of a list. Frankduc 23 3,013 Apr-05-2023, 07:36 PM
Last Post: Frankduc
  List of random numbers astral_travel 17 2,533 Dec-02-2022, 10:37 PM
Last Post: deanhystad
  [split] why can't i create a list of numbers (ints) with random.randrange() astral_travel 7 1,428 Oct-23-2022, 11:13 PM
Last Post: Pedroski55
  Question about List.reverse() method tomliuwhite 1 1,303 Dec-07-2021, 08:20 AM
Last Post: ndc85430
  get data from 2 async functions korenron 0 1,202 Sep-22-2021, 08:39 AM
Last Post: korenron

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020