Python Forum
How to filter specific rows from large data file
Thread Rating:
  • 1 Vote(s) - 3 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to filter specific rows from large data file
#1
Hi I have a large data file and I'm only interested in rows with specific x values of 4.125 as shown below. Now because the value of 4.125 relates to the stop position of the ion, the corresponding start position is also of interest to me and I want to keep this information in the array. How do I write a program which effectively finds the x stop position of 4.125 and retains the ion start position. It is a 120982 * 9 array and in the example shown below I would be interested in keeping the information of ion # 3849096.

"Ion N","Mass","Charge","X","Y","Z","Azm","Elv","KE" 3849094,0.00054858,-1,66.5216,-51,-3.8,-180,88.7,18160 3849094,0.00054858,-1,27.3925,30.3532,-4.07076,-177.1,41.5494,17697.2 3849095,0.00054858,-1,66.5216,-51,-3.7,-180,88.7,18160 3849095,0.00054858,-1,26.6277,31.0039,-3.91402,-177.096,40.8293,17699.4 3849096,0.00054858,-1,66.5216,-51,-3.6,-180,88.7,18160 3849096,0.00054858,-1,4.125,44.9887,-2.47517,-176.363,25.715,17711.1

This is the code I have developed so far but does not work:

import pandas as pd 
import numpy as np

opts = pd.read_csv('Ambre_2.dat',sep = ',', low_memory = False)
df = pd.DataFrame(opts)

X = df.iloc[:,3]
IonN = df.iloc[:,0]
tol = 1e-6
Fltr = 4.125

filterreddata = df[abs(df.X-Fltr)<tol,:]
filteredions = df(np.in1d(df.IonN, filterreddata.IonN), :]
filteredions[2:2:end, :] = []
f = open('ions.csv', 'w')
f.write(tabulate(filteredions))
f.close()
Reply


Messages In This Thread
How to filter specific rows from large data file - by Ariane - Jun-29-2018, 07:39 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Errors if an xlsx file has blank rows in the beginning…. tester_V 1 984 Aug-28-2023, 06:22 PM
Last Post: deanhystad
  Sorting data by specific variables using argparse Bearinabox 5 1,571 Jan-01-2023, 07:44 PM
Last Post: Bearinabox
  Counting Duplicates in large Data Set jmair 3 1,268 Dec-07-2022, 09:42 AM
Last Post: paul18fr
  Reading large crapy text file in anaconda to profile data syamatunuguntla 0 916 Nov-18-2022, 06:15 PM
Last Post: syamatunuguntla
  Training a model to identify specific SMS types and extract relevant data? lord_of_cinder 0 1,070 Oct-10-2022, 04:35 AM
Last Post: lord_of_cinder
  Split excel file and write output at specific row and set sheet position DSCA 0 2,118 May-12-2022, 07:29 PM
Last Post: DSCA
  Searching Module to plot large data G_rizzle 0 1,545 Dec-06-2021, 08:00 AM
Last Post: G_rizzle
  how to filter data frame dynamically with the columns psahay 0 2,490 Aug-24-2020, 01:10 PM
Last Post: psahay
  Dropping Rows From A Data Frame Based On A Variable JoeDainton123 1 2,341 Aug-03-2020, 02:05 AM
Last Post: scidam
  Chunking and Sorting a large file Robotguy 1 3,704 Jul-29-2020, 12:48 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020