Python Forum
created a pandas series instead of pandas DataFrame
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
created a pandas series instead of pandas DataFrame
#1
i created a pandas DataFrame for GPS satellite from a text file and i wanted to create a single DataFrame for distinct satellite, all goes as per my expectation but if for a given satellite there is only one observation it created a pandas Series instead of DataFrame. I want DataFrame because i need it to save it in HDF5 file.
Also i get a warning
FutureWarning: convert_objects is deprecated. To re-infer data dtypes for object columns, use DataFrame.infer_objects()
For all other conversions use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric.


import pandas as pd
from textwrap import wrap
from datetime import datetime
def STF(a):
    '''STF takes input as a number string containing 'D' or 'E' and return the
    value as float and if the number string doesn't contain "D" or "E" it return 
    the original number in float.
    for example: 198701D-02=1987.01
                 198701E-02=1987.01 
                 198839=198839.00'''
    if 'D' in a:
        b = a.split("D")
        c = (float(b[0]))*(10**(int(b[1])))
        return(c)
    elif 'E' in a:
        b = a.split("E")
        c = (float(b[0]))*(10**(int(b[1])))
        return(c)
def gpsnav3df(filepath): 
    '''gpsnav3df takes input as filepath of Rinex3 navigation file of GPS  and
    give output in the form of dataframe containing all the information about the
    satellite like Satellite ID, Epoch, Clock Information etc.'''
    with open(filepath) as fp: #open file and returns the corresponding file object  
    
        data=fp.read() # read all the data into single string
        data1=data.split('END OF HEADER') # split a string into list
    
        #data section
        satdata=data1[1]
    
        satdata=satdata.strip() # remove the whitespaces from beginning and end of the string
        satdata=wrap(satdata,610) # split string into nth character of each word
    

        #creating Empty list    
        SatId ,Epoch,SV_clock_Bias,SV_clock_Drift,SV_clock_drift_Rate,IODE,Crs,Delta_n, M0, Cuc, e,Cus, sqrt_A ,Toe, Cic ,OMEGA_0, Cis, i0, Crc ,omega ,OMEGA_dot,IDOT,Codes ,GPS_Week ,L2P_data_Flag,SV_accuracy,SV_health, TGD, TODC ,Transmission_time , Fit_interval  = ([] for i in range(31) )
        
        for sat in satdata:
            #SV 
            satId = sat[0:3]  
            SatId.append(satId)
        
            #EPOCH
        
            time  = sat[4:23].replace(' ' , '')
            epoch = datetime.strptime(time,'%Y%m%d%H%M%S')
            Epoch.append(epoch)
    
            #SV_clock
        
            SV_clock_bias       = STF(sat[23:42]) # second
            SV_clock_Bias.append(SV_clock_bias)
        
            SV_clock_drift      = STF(sat[42:61]) #sec/sec
            SV_clock_Drift.append(SV_clock_drift)
                
            SV_clock_drift_rate = STF(sat[61:80]) # sec/sec2   
            SV_clock_drift_Rate.append(SV_clock_drift_rate)
            
            # Broadcast orbit-1
    
            IOD_E    =  STF(sat[85:104]) # Issue of Data,Ephemeris
            IODE.append(IOD_E)
            
            Cr_s     =  STF(sat[104:124]) #meters
            Crs.append(Cr_s)
            
            Delta_N =  STF(sat[124:142])# rad/sec
            Delta_n.append(Delta_N)
            
            
            M_0      =  STF(sat[142:161]) # radians
            M0.append(M_0)
            #broadcast orbit-2 
    
            Cu_c    = STF(sat[166:185]) #radians
            Cuc.append(Cu_c)
            
            e_      = STF(sat[185:204]) #Eccentricity
            e.append(e_)
            
            Cu_s    = STF(sat[204:223]) #radians
            Cus.append(Cu_s)
        
            sqrtA = STF(sat[223:242]) # sqrt(m)    
            sqrt_A.append(sqrtA)
    
            #broadcast orbit-3
    
            To_e     = STF(sat[247:266]) #Time of Ephemeris(sec of GPS week)
            Toe.append(To_e)
            
            Ci_c     = STF(sat[266:285]) #radians
            Cic.append(Ci_c)
            
            OMEGA0 = STF(sat[285:304]) #radians
            OMEGA_0.append(OMEGA0)
            
            Ci_s     = STF(sat[304:323]) #radians/sec
            Cis.append(Ci_s)
            
    
            #broadcast orbit-4
            
            i_0        = STF(sat[328:347]) #radian
            i0.append(i_0)
            
            Cr_c       = STF(sat[347:366]) #meters 
            Crc.append(Cr_c)
            
            omegA     = STF(sat[366:385]) #radians
            omega.append(omegA)
            
            OMEGA_Dot = STF(sat[385:404]) #radians/sec
            OMEGA_dot.append(OMEGA_Dot)
            
            #broadcast orbit-5
    
            IDO_T          = STF(sat[409:428])
            IDOT.append(IDO_T)
            
            codes         = STF(sat[428:447])
            Codes.append(codes)
            
            GPS_week      = STF(sat[447:466])
            GPS_Week.append(GPS_week)
            
            L2P_data_flag = STF(sat[466:485])
            L2P_data_Flag.append(L2P_data_flag)
        
        #broadcast orbit-6
        
            SV_Accuracy = STF(sat[490:509])
            SV_accuracy.append(SV_Accuracy)
            
            SV_Health   = STF(sat[509:528])
            SV_health.append(SV_Health)
            
            
            TG_D         = STF(sat[528:547])
            TGD.append(TG_D)
            
            IOD_C        = STF(sat[547:566])
            TODC.append(IOD_C)
        
        #broadcast orbit-7
            Transmission_Time = STF(sat[571:590])
            Transmission_time.append(Transmission_Time)
            
            Fit_Interval      = STF(sat[590:609])
            Fit_interval.append(Fit_Interval)
        
    df = pd.DataFrame(list(zip(SatId ,Epoch,SV_clock_Bias,SV_clock_Drift,SV_clock_drift_Rate,IODE,Crs,Delta_n, M0, Cuc, e,Cus, sqrt_A ,Toe, Cic ,OMEGA_0, Cis, i0, Crc ,omega ,OMEGA_dot,IDOT,Codes ,GPS_Week ,L2P_data_Flag,SV_accuracy,SV_health, TGD, TODC ,Transmission_time , Fit_interval )),
                      columns =['SatId' ,'Epoch','SV_clock_Bias','SV_clock_Drift','SV_clock_drift_Rate','IODE','Crs','Delta_n', 'M0', 'Cuc', 'e','Cus', 'sqrt_A' ,'Toe', 'Cic' ,'OMEGA_0', 'Cis',' i0',' Crc' ,'omega' ,'OMEGA_dot','IDOT','Codes' ,'GPS_Week' ,'L2P_data_Flag','SV_accuracy','SV_health', 'TGD', 'TODC' ,'Transmission_time' , 'Fit_interval']
                      ,dtype=object).convert_objects()
    
    df=df.set_index('SatId')
    
    return (df) 


filepath='E:/Project/RAW_Data/ABMF00GLP_R_20190201400_01H_GN.txt'
GPS=gpsnav3df(filepath)
G01=GPS.loc['G01']
print(G01)
print(type(G01))
G10=GPS.loc['G10']
print(G10)
print(type(G10))
Output:
Epoch 2019-01-20 06:00:00 SV_clock_Bias -0.000148412 SV_clock_Drift -6.48015e-12 SV_clock_drift_Rate 0 IODE 87 Crs -152.625 Delta_n 4.16696e-09 M0 1.87066 Cuc -7.94232e-06 e 0.00830994 Cus 7.9684e-06 sqrt_A 5153.67 Toe 21600 Cic -2.30968e-07 OMEGA_0 -0.722293 Cis 3.1665e-08 i0 0.974264 Crc 234.531 omega 0.687125 OMEGA_dot -7.9539e-09 IDOT -2.78583e-10 Codes 1 GPS_Week 2037 L2P_data_Flag 0 SV_accuracy 2 SV_health 0 TGD 5.58794e-09 TODC 87 Transmission_time 14400 Fit_interval None Name: G01, dtype: object <class 'pandas.core.series.Series'> Epoch SV_clock_Bias ... Transmission_time Fit_interval SatId ... G10 2019-01-20 13:59:44 0.000132 ... 47580.0 None G10 2019-01-20 16:00:00 0.000132 ... 50400.0 None [2 rows x 30 columns] <class 'pandas.core.frame.DataFrame'> E:/Project/code/n.py:172: FutureWarning: convert_objects is deprecated. To re-infer data dtypes for object columns, use DataFrame.infer_objects() For all other conversions use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric.

Attached Files

.txt   ABMF00GLP_R_20190201400_01H_GN.txt (Size: 27.69 KB / Downloads: 0)
Reply
#2
Docstring of first function states “STF takes input as a number string containing 'D' or 'E' and return the value as float and if the number string doesn't contain "D" or "E" it return the original number in float.”

I don’t observe returning original number as float.
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#3
@perfringo:
yeah, you are right i changed it but the main problem of getting series instead of DataFrame is still there.
Reply
#4
Maybe that helps Convert Series to DataFrame.
Reply
#5
Or you can try call .loc correctly:

>>> import pandas as pd
>>> df = pd.DataFrame({'chars': (*'abc',), 'nums': (*range(3),)})
>>> df
 chars  nums
0     a     0
1     b     1
2     c     2
pandas.core.frame.DataFrame
>>> type(df.loc[0])
<class 'pandas.core.series.Series'>
>>> type(df.loc[[0]])                  # note additional brackets
<class 'pandas.core.frame.DataFrame'>
This and some other examples available in pandas documenation: pandas.DataFrame.loc
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#6
Keeping blinders on to your code and just answering the question posed, I am with ThomasL but this link Series.to_frame gives more detail and some examples. Hope this helps.
Reply
#7
@perfringo:
thanks, it works
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Parsing and summing time deltas (duration) onto bar + pie charts using pandas - - DRY Drone4four 2 514 Feb-10-2024, 06:04 PM
Last Post: Drone4four
  Pandas hierarchical sum mariostg 1 384 Jan-26-2024, 03:47 PM
Last Post: mariostg
  Pandas keep existing format of Excel AlphaInc 2 883 Jan-11-2024, 03:44 AM
Last Post: plonkarchivist
  Grouping in pandas/multi-index data frame Aleqsie 3 607 Jan-06-2024, 03:55 PM
Last Post: deanhystad
  [pandas] TypeError: list indices must be integers or slices, not str but type is int. cspower 4 764 Dec-30-2023, 09:38 AM
Last Post: Gribouillis
  Assigning conditional values in Pandas Scott 3 725 Dec-19-2023, 03:10 AM
Last Post: Larz60+
  HTML Decoder pandas dataframe column mbrown009 3 962 Sep-29-2023, 05:56 PM
Last Post: deanhystad
  FutureWarning: Logical ops (and, or, xor) between Pandas objects and dtype-less seque NewBiee 5 1,490 Sep-12-2023, 03:15 PM
Last Post: deanhystad
  [solved] duplication in pandas BSDevo 0 523 Sep-06-2023, 10:47 PM
Last Post: BSDevo
  Help to control pandas.melt function vanphuht91 0 672 Jun-17-2023, 08:41 AM
Last Post: vanphuht91

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020