Python Forum
I need help with a script that reads and searches through an api.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
I need help with a script that reads and searches through an api.
#4
By async I assume you're referring to asyncio? If that's the case, then I don't think asyncio would help here at all. asyncio is similar to threading, but also much different since instead of running multiple threads at once, it jumps between threads, still performing only one action at a time. You should use threading, though also be careful about how many threads you are running at one time. You can create a class to manage all your threads, here's an example below:
from threading import Thread, Event
from RedditAPI import Client
import time


class RedditThread(Thread):
    def __init__(self, sort):
        Thread.__init__(self)

        self.end_thread = Event()
        self.client = Client("sx2iI2emTgSsSA", '65xxWV8QgP4BfMIvlsL0xKSli9bJjg', "Python Reddit API")
        self.sort = sort

    def run(self):
        posts_obj = self.client.make_request(f'r/hololive/{self.sort}', 'get')
        print(posts_obj.posts[0].title, time.perf_counter())
        self.finish()

    def finish(self):
        self.end_thread.set()

    @property
    def is_finished(self):
        return self.end_thread.is_set()


class ThreadManager(Thread):
    # You can increase this number
    max_threads = 2

    def __init__(self):
        Thread.__init__(self)
        self.running_threads = []
        self.queue = []

        self.end_thread = Event()

    def new_thread(self, thread):
        self.queue.append(thread)

    def run(self):
        while not self.end_thread.is_set():
            threads_to_remove = []
            for thread in self.running_threads:
                if thread.is_finished:
                    threads_to_remove.append(thread)

            for thread in threads_to_remove:
                self.running_threads.remove(thread)

            for _ in range(self.max_threads - len(self.running_threads)):
                if self.queue:
                    _thread = self.queue.pop(0)
                    _thread.start()
                    self.running_threads.append(_thread)

    def stop(self):
        self.end_thread.set()

    @property
    def finished(self):
        return not self.running_threads and not self.queue


tm = ThreadManager()

for sort in ['hot', 'best', 'new', 'rising', 'top']:
    t = RedditThread(sort)
    tm.new_thread(t)

tm.start()
while not tm.finished:
    time.sleep(3)
tm.stop()
Output:
Calliope Mori�'s 2nd EP Announcement 2.6218656 Calliope Mori�'s 2nd EP Announcement 2.7002329 Today's a good day to be a HoloID fan 3.5399011 New ASMR stream<3 3.8415417 Casual Shark 4.7890268
As you can see, it runs two threads, they finish and it moves onto the next two, and so on. You could increase this number, though I would say don't run too many threads at one time.
Reply


Messages In This Thread
RE: I need help with a script that reads and searches through an api. - by SheeppOSU - Apr-06-2021, 01:51 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Is it possible to scrape this data from Google Searches rosjo 1 2,208 Nov-06-2020, 06:51 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020