Python Forum
Async / Await usage with asyncio to retrieve urls in parallel
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Async / Await usage with asyncio to retrieve urls in parallel
#8
1.) yes good point, you definitely don't want to create a session per request, so in this case you can just move the context manager in the gather_coro and pass it as a argument to get_url_result function:
def get_results(urls):

  async def gather_coro():
    async with aiohttp.ClientSession() as session:   # moved here from get_url_result
      return await asyncio.gather(*(get_url_result(url, session) for url in urls))

  url_results = asyncio.run(gather_coro())
  return dict(url_results)
2.) Yes, get_url_result can return just resp, creation of dictionary really shouldn't be a problem! But remember that in order to access response content like resp.text() you need to await it, this means you won't be able to access it from a normal function but only from coroutine, from docs -
Quote:aiohttp loads only the headers when .get() is executed, letting you decide to pay the cost of loading the body afterward, in a second asynchronous operation. Hence the await response.text().
Reply


Messages In This Thread
RE: Async / Await usage with asyncio to retrieve urls in parallel - by mlieqo - Sep-21-2020, 06:38 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  get data from 2 async functions korenron 0 1,245 Sep-22-2021, 08:39 AM
Last Post: korenron
  Async requests lukee 0 1,524 Oct-06-2020, 04:40 AM
Last Post: lukee
  Using Python to search through a list of urls jeremy 4 2,908 Dec-18-2019, 11:52 AM
Last Post: Malt
  Async IO writing to two Different Tables Help TiagoV 0 2,652 Oct-09-2019, 04:45 AM
Last Post: TiagoV
  Urls in a file to be executed pyseeker 2 2,073 Sep-09-2019, 03:38 PM
Last Post: pyseeker
  user validation for opening urls Ashley 6 2,745 Jul-08-2019, 09:08 PM
Last Post: metulburr
  Issues with async and yielding from API GSerum 1 2,158 Dec-18-2018, 08:37 PM
Last Post: nilamo
  async question on raspberry pi baukeplugge 2 61,032 Nov-07-2018, 07:58 PM
Last Post: baukeplugge
  How I can limit quantity of parallel executable tasks in asyncio? AlekseyPython 1 2,451 Oct-24-2018, 10:22 AM
Last Post: AlekseyPython
  Looping URLs breaks them PythonStudent 2 2,952 Apr-21-2018, 02:54 PM
Last Post: PythonStudent

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020