Python Forum
I need help with a script that reads and searches through an api.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
I need help with a script that reads and searches through an api.
#1
New coder here who needs help with api scanning project. I've been racking my brain with https://pastebin.com/Nk4yGAzz for four months, could use some help and/or direction.

What's it do and how does it work?:
-------------

The trading system of World of Warcraft is known as the Auction House. This system lets every person in the game trade items between each other without having to meet up with one another. So that means you can sell any item from any city with an auctioneer and someone on the other side of the world can bid on it/buy it. My script looks at the hourly published data from every realm (there are 83 realms) and scans for a list of items that I'm interested in buying. There's a popular website called the https://theunderminejournal.com/#us/ if you want to see what I'm going for.


I'm having a lot of difficulty optimizing my code and a lot of what I'm reading is too advance for my understanding of programming. Any help is greatly appreciated.
Reply
#2
What exactly is not working in your code?
Reply
#3
(Apr-05-2021, 03:05 PM)SheeppOSU Wrote: What exactly is not working in your code?

My code has a delay from the time the json downloads to when I see the results. There's a consistent 2 to 3 seconds delay on most of the urls. I'm just trying to speed things up a bit but I'm having trouble understanding how to implement async.
Reply
#4
By async I assume you're referring to asyncio? If that's the case, then I don't think asyncio would help here at all. asyncio is similar to threading, but also much different since instead of running multiple threads at once, it jumps between threads, still performing only one action at a time. You should use threading, though also be careful about how many threads you are running at one time. You can create a class to manage all your threads, here's an example below:
from threading import Thread, Event
from RedditAPI import Client
import time


class RedditThread(Thread):
    def __init__(self, sort):
        Thread.__init__(self)

        self.end_thread = Event()
        self.client = Client("sx2iI2emTgSsSA", '65xxWV8QgP4BfMIvlsL0xKSli9bJjg', "Python Reddit API")
        self.sort = sort

    def run(self):
        posts_obj = self.client.make_request(f'r/hololive/{self.sort}', 'get')
        print(posts_obj.posts[0].title, time.perf_counter())
        self.finish()

    def finish(self):
        self.end_thread.set()

    @property
    def is_finished(self):
        return self.end_thread.is_set()


class ThreadManager(Thread):
    # You can increase this number
    max_threads = 2

    def __init__(self):
        Thread.__init__(self)
        self.running_threads = []
        self.queue = []

        self.end_thread = Event()

    def new_thread(self, thread):
        self.queue.append(thread)

    def run(self):
        while not self.end_thread.is_set():
            threads_to_remove = []
            for thread in self.running_threads:
                if thread.is_finished:
                    threads_to_remove.append(thread)

            for thread in threads_to_remove:
                self.running_threads.remove(thread)

            for _ in range(self.max_threads - len(self.running_threads)):
                if self.queue:
                    _thread = self.queue.pop(0)
                    _thread.start()
                    self.running_threads.append(_thread)

    def stop(self):
        self.end_thread.set()

    @property
    def finished(self):
        return not self.running_threads and not self.queue


tm = ThreadManager()

for sort in ['hot', 'best', 'new', 'rising', 'top']:
    t = RedditThread(sort)
    tm.new_thread(t)

tm.start()
while not tm.finished:
    time.sleep(3)
tm.stop()
Output:
Calliope Mori�'s 2nd EP Announcement 2.6218656 Calliope Mori�'s 2nd EP Announcement 2.7002329 Today's a good day to be a HoloID fan 3.5399011 New ASMR stream<3 3.8415417 Casual Shark 4.7890268
As you can see, it runs two threads, they finish and it moves onto the next two, and so on. You could increase this number, though I would say don't run too many threads at one time.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Is it possible to scrape this data from Google Searches rosjo 1 2,168 Nov-06-2020, 06:51 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020