Python Forum
How to work with large lists without crashing Python?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to work with large lists without crashing Python?
#1
Whenever I try to make a list of >5,000 items in python idle, it freezes and becomes unresponsive. Is there a way to work with large python lists without using a lot of data or memory?

I was thinking to create a list you could have python print each element of your list one by one into a text file, and it would print each item in a new line. This would create a text file with 10,000+ lines with 4-5 words in each line. For accessing the list, you could have python pull one item at a time, and just iterate over every line of the text file.

My question is, is the above method of dealing with lists I described as doable? If so, can someone direct me to a module/tutorial involving that? If not, is there a better way to deal with large lists?

BTW I looked at numpy arrays, but I don't if I can use that as I will not know the size of the list until it is created. I only know that it will be in the ballpark of 10,000+ items.
Reply
#2
it depends what elements are, but in most cases 5000 items is not that much and should not crash python. of course it also depend on your machine RAM. Maybe you are doing something wrong.
also, depending on how you create your list, you could create generator/iterator and iterate over it one element at a time without creating the entire list in memory
can you provide more info on how you create the list/what its elements are, etc. maybe provide sample data
Reply
#3
The list is a list of stock symbols. From that list of symbols. I form a smaller list. That smaller list needs to be stored in memory all at once. The larger list is 10,000+ elements of 3-5 letter acryonyms. The larger list is around 5,000. They are parsed for online.

edit: typo.
Reply
#4
that is not a problem at all and will not crash python. however as I said you don't need to store the small list in memory at once.
without seeing your code I cannot tell what is wrong with your code
Reply
#5
I’m trying to import the list. Im just copying 10,000 elements into idle and hitting enter. Thats where it lags. it also lags when i print the list. If i make the list one element at a time it is fine, however I need to import a premade list. When I get home I can post the code, aorry for the inconvinence.
Reply
#6
maybe it's a problem with IDLE. It's terrible IDE. try something else as IDE https://wiki.python.org/moin/IntegratedD...vironments
Reply
#7
Why do you hardcode the list? Just read it from a file and iterate over it.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#8
(Mar-14-2018, 09:08 PM)wavic Wrote: Why do you hardcode the list? Just read it from a file and iterate over it.

Don't know how to do that. Was asking for help on how to do that.
Downloaded pycharm, problem solved. Thank you buran.
Reply
#9
Put the data into a file. Let's call it data.txt.

with open('data.txt', 'r') as data:
    for line in data:
        # do something
Here is how you read from a file. You create a file object called data and iterate over each line. The 'r' parameter is for reading.
To write to a file you do almost the same.
with open('some_file.txt', 'w') as output: # this time in 'w' (write) mode
    for element in iterable:
        output.write(element)
The with statement is called context manager and it will close the file automatically when its code block is executed. You can stack them.

with open('file_one.txt', 'r') as r:
    with open('file_two.txt', 'w') as w:
        for line in r:
            w.write(line)
This read from one file and write to another. Basically, it's copying the file.
tester_V likes this post
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python/Numpy have I already written the swiftest code for large array? justforkicks1 1 2,780 Dec-12-2017, 11:56 PM
Last Post: squenson

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020