Python Forum
Async / Await usage with asyncio to retrieve urls in parallel
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Async / Await usage with asyncio to retrieve urls in parallel
#5
I sure would love to post my experiments, but I'm really struggling to put it all together. I don't even know where to start. What I want seems fairly simple (just by looking at the sync version), and I'm new to python so I'm really in pain to make all of this work. In the end I just want a async getResults() function that can be called from elsewhere just as if it were a sync getResults().

From what I understand of your answers, I have no choice but to use the aiohttp. So the other pieces of code are irrelevant.

In the last example you're giving, it just prints the result, whereas I want to return it as a dictionary url -> response, and it doesn't handle a loop.

Here is what I've tried, but I don't know how wrong I am :

async def getResultsAsync(urls):
    async def getResultAsync(url):
        async with aiohttp.ClientSession() as session:
            resp = await session.get(url)
            res = url, await resp.json()
        return res
    return dict(await asyncio.gather(*[getResultAsync(url) for url in urls]))

def getResults(urls):
    return asyncio.run(getResultsAsync(url))
Reply


Messages In This Thread
RE: Async / Await usage with asyncio to retrieve urls in parallel - by lukee - Sep-20-2020, 07:08 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  get data from 2 async functions korenron 0 1,246 Sep-22-2021, 08:39 AM
Last Post: korenron
  Async requests lukee 0 1,524 Oct-06-2020, 04:40 AM
Last Post: lukee
  Using Python to search through a list of urls jeremy 4 2,909 Dec-18-2019, 11:52 AM
Last Post: Malt
  Async IO writing to two Different Tables Help TiagoV 0 2,652 Oct-09-2019, 04:45 AM
Last Post: TiagoV
  Urls in a file to be executed pyseeker 2 2,073 Sep-09-2019, 03:38 PM
Last Post: pyseeker
  user validation for opening urls Ashley 6 2,746 Jul-08-2019, 09:08 PM
Last Post: metulburr
  Issues with async and yielding from API GSerum 1 2,159 Dec-18-2018, 08:37 PM
Last Post: nilamo
  async question on raspberry pi baukeplugge 2 61,246 Nov-07-2018, 07:58 PM
Last Post: baukeplugge
  How I can limit quantity of parallel executable tasks in asyncio? AlekseyPython 1 2,451 Oct-24-2018, 10:22 AM
Last Post: AlekseyPython
  Looping URLs breaks them PythonStudent 2 2,953 Apr-21-2018, 02:54 PM
Last Post: PythonStudent

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020