Python Forum
How to reallocating memory space in multithreading program in python ?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to reallocating memory space in multithreading program in python ?
#1
#from multiprocessing.pool import ThreadPool
#from multiprocessing.dummy import Pool as ThreadPool
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool 
import tweepy
import random
import json
import simplejson
import re
from tweepy import OAuthHandler
from tweepy.auth import OAuthHandler
import gc
import sys

import time
from datetime import datetime
import threading
import psycopg2

# Consumer keys and access tokens, used for OAuth

consumer_key = "xxxxxxxx"
consumer_secret = "xxxxxxxx"
access_token = "xxxxxxxxxx"
access_token_secret = "xxxxxxx"


# OAuth process, using the keys and tokens
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)


# Creation of the actual interface, using authentication
try:
    api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True, retry_count=10, retry_delay=60)
    #auth = api.VerifyCredentials()
except Exception as e:
    print("Exception is occured",e)
count =0 

def twitterjson(string):

            results = tweepy.Cursor(api.user_timeline, screen_name=sm_handle, count=100).items()
            for tweet in results:
                with open(sm_handle+'.json', 'w') as outfile:
                    outfile.write(json.dumps(tweet._json))
                    print (tweet)
     return 1;
            
if __name__ == "__main__":
  while True:
    try:
        # Establishing a PSQL connection
        conn = psycopg2.connect("dbname='sample' user='postgres' host='localhost' port='5432'")
        print("opened database successfully")

        # create a psycopg2 cursor that can execute queries
        cursor = conn.cursor()

        # twitter_users_query = "select twitter_user_id, twitter_screen_name from twitter_users;"
        twitter_users_query = "select usernames from test;"
        cursor.execute(twitter_users_query)
        #pool = ThreadPool(10) 

        rows = cursor.fetchall()
        names=[]
        for row in rows:
            sm_handle = row[0]
            names.append(sm_handle)
            print("sm_handles: "+sm_handle)
            threads = []

           # for i in rows:
        for name in names:
            print ("name:"+name)
     
    except Exception as e:
        print("unable to connect to the database : ", e)
    try:   
        pool = ThreadPool(100) 
        gc.collect()
        #pool.terminate()
        results=pool.map(twitterjson, names)
        pool.terminate(100)
        print ("Completed"+results)
        pool.terminate()
        pool.close() 
        pool.join()
        print("This is sleeping for 1 minute..............")
        #time.sleep(60)
        print('Done')
        
        #pool.terminate()
        #pool.close() 
        #pool.join()
        #pool.terminate()

    except Exception as e:
         print("Exception is occured",e)
         pool.terminate();
Reply
#2
Could you be more explicit when you say "reallocating memory space"?
If it ain't broke, I just haven't gotten to it yet.
OS: Windows 10, openSuse 42.3, freeBSD 11, Raspian "Stretch"
Python 3.6.5, IDE: PyCharm 2018 Community Edition
Reply
#3
How to free the memory utilzation(RAM) in completed threads in above Example
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  multithreading xlwings error Equivocal 0 346 Jan-25-2025, 05:10 PM
Last Post: Equivocal
  Getting an error while trying to process data, low memory when memory is not low? bkeith12 0 489 Dec-20-2024, 03:06 PM
Last Post: bkeith12
  Python rule about the space character surrounding the equal sign ineuw 10 4,427 Sep-21-2023, 09:17 AM
Last Post: ineuw
  Python Tkinter Simple Multithreading Question AaronCatolico1 5 3,220 Dec-14-2022, 11:35 PM
Last Post: deanhystad
  python list out top 10 Memory utilisation mg24 0 1,343 Nov-20-2022, 10:51 PM
Last Post: mg24
  python Multithreading on single file mg24 3 3,332 Nov-05-2022, 01:33 PM
Last Post: snippsat
Bug Embedded Python Memory Leaks Alexei 1 1,674 Sep-16-2022, 01:15 PM
Last Post: Alexei
  Removing Space between variable and string in Python coder_sw99 6 10,493 Aug-23-2022, 01:15 PM
Last Post: louries
  multithreading Hanyx 4 2,098 Jul-29-2022, 07:28 AM
Last Post: Larz60+
Question Problems with variables in multithreading Wombaz 2 2,021 Mar-08-2022, 03:32 PM
Last Post: Wombaz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020