Python Forum
How to check for nested dataframe density?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to check for nested dataframe density?
#1
I have attached a csv file where this data is being stored as a nested dataframe in a main dataframe which i cannot include in here. main_col is the column from the main dataframe that has the data in this csv file stored in it as a nested df. what I want to achieve is to measure the data density but i am getting an index positional error. The code I am currently using looks like below and I am not sure what is causing the problem.


.csv   testdata.csv (Size: 310 bytes / Downloads: 32)

import pandas as pd

df = pd.read_csv('test_data.csv')

def data_density(thresh=None):
    counter = 0
    counter_1 = 0
    ix = []
    for ixn, data in df.iterrows():
        counter = counter + 1
        total_matrix = data['main_col'].loc[:, 'A1']['Game1'].shape[0] * \
                       data['main_col'].loc[:, 'A1']['Game2'].shape[1] + \
                       data['main_col'].loc[:, 'A2']['Game1'].shape[0] * \
                       data['main_col'].loc[:, 'A2']['Game2'].shape[1]
        total_values = data['main_col'].loc[:, 'A1']['Game1'].count().sum() + \
                       data['main_col'].loc[:, 'A2']['Game1'].count().sum()


        if total_values != 0:
            data_density = float(total_values) / float(total_matrix)

            if data_density > threshold:
                counter_1 = counter_1 + 1
                ix.append(ixn)
    ratio = float(counter_1) / counter
    return ix, ratio


df3 = pd.DataFrame()
for i in range(80, 100, 5):
    i = float(i) / 100
    ix, ratio = data_density(thresh=i)
    print('data density for', ratio, 'when threshold is:', i)
    print(len(ix))
    df = pd.DataFrame()
    for j in range(0, len(ix)):
        df2 = df[(df.index == ix[j])]
        df = df.append(df2)
    print(df)
    df3 = df3.append(df)
Reply


Messages In This Thread
How to check for nested dataframe density? - by python_newbie09 - Aug-27-2018, 07:34 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  inserting something into probability density SchroedingersLion 1 2,064 Jan-06-2020, 09:15 AM
Last Post: Gribouillis
  Get underlying function from Kernel Density Estimation jpython 3 2,761 Dec-05-2019, 11:23 AM
Last Post: jpython
  How to get the probability density function of my data set jpython 1 2,256 Dec-04-2019, 12:49 PM
Last Post: Larz60+
  finding the integral of probability density function Staph 3 3,595 Aug-11-2019, 09:19 AM
Last Post: buran
  MatPlotLib 2d plot of current density? ruben 0 2,171 May-13-2019, 06:47 AM
Last Post: ruben
  Pandas nested json data to dataframe FrankC 1 10,115 Aug-14-2018, 01:37 AM
Last Post: scidam
  Plot Probability Density of an Histogram Python_TLS 1 2,853 Jun-28-2018, 06:19 AM
Last Post: scidam

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020