Python Forum
How to automate list separation. NOOB
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to automate list separation. NOOB
#6
"The problem is that the enormous data set text files I copy/paste from have dates and values but no commas."

Why do you copy data (manually)? Where do you paste it?

There is no such datastructure in Python:

2009 09 23 11 45 00 1 9999.000
2009 09 23 12 00 00 1 9999.000
2009 09 23 12 15 00 1 4593.017
2009 09 23 12 30 00 1 4593.005
2009 09 23 12 45 00 1 4592.993

What is it? String? Multiline string? Text file?

If data is in file then one should iterate over it row by row.

Snippet of code I provided is working example how to solve this type of problems. If you don't know how to use it then you should say so. Nobody is 'playing' here.

How big is your file? Or there are many of them? Appr how many rows? Is my calculation correct:

>>> 4000 * 96 * 365 * 20   #  number of buoys * water height measurements per day * days in year * years                                           
2803200000
Do you need only two numbers out of all these rows?

EDIT:

Let's assume that I have a file which has rows as you provided in raw data and this is stored in file named buoys.txt, then I could do:

with open('buoys.txt', 'r') as f:
    rows = (row.strip() for row in f)     
    nums = (float(row.split()[-1]) for row in rows if float(row.split()[-1]) != 9999.0)
    max_value = max(nums)
This should be gentle on memory as it's generator piping i.e. rows are consumed one at the time.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Messages In This Thread
RE: How to automate list separation. NOOB - by perfringo - Sep-20-2019, 12:47 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  [split] Automate the boring stuff, inserting commas in list srikanth 1 2,131 Jul-02-2019, 02:29 PM
Last Post: metulburr
  Automate the boring stuff, inserting commas in list DJ_Qu 3 4,724 Apr-21-2019, 03:52 PM
Last Post: perfringo
  1. How can I automate this batch creation of documents? 2. How can I automate posting SamLearnsPython 2 3,444 Jul-02-2018, 11:36 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020