Cleaning my code to make it more efficient

[deanhystad]

I'm not asking for a description of the code. I want to know what the program does. What is it going to be used for. I am more interested in why you need to filter the dataframe than how you filter the dataframe.

This part of your description is useful:

I have csv file containing various data and it contains many columns, but i choose 12 columns to filter what i need.
So i have some pie charts, line bars, histograms, bar charts and thats how i analyze my data also.

What is this filtering meant to accomplish? How do the filters relate to the charts? Is that related to this?

Why im saying like this: i have Market column - market column contains 152 unique areas ( Atlanta, Houston, Los Angeles etc .. some major us cities ) so to write for each - i dont think its worth it as it would be 152 extra lines of code .... and to use my logic would be way much less but longer lines as i choose only Column MARKET instead of each from column MARKET or im completely wrong

I am still stymied by this:

and to use my logic would be way much less but longer lines as i choose only Column MARKET instead of each from column MARKET or im completely wrong ?

Read that last quoted line. Does it make sense to you? What am I missing that prevents it from making sense to me? Before pressing the "post" button, read the post as if you had no other knowledge of the problem except the contents of the post. Your audience is vast in Python knowledge and completely ignorant about your program.

These are equivalent. Your approach:

if not any ([regionas, valstija, marketas]):
    filtered_df = df
elif not any ([valstija, regionas]):
    filtered_df = df[df["Market"].isin(marketas)]
elif not any ([valstija, marketas]):
    filtered_df = df[df["Region"].isin(regionas)]
elif not any ([regionas, marketas]):
    filtered_df = df[df["State"].isin(valstija)]
elif valstija and marketas:
    filtered_df = df12[df["State"].isin(valstija) & df12["Marketas"].isin(marketas)]
elif valstija and marketas:
    filtered_df = df12[df["State"].isin(valstija) & df12["Market"].isin(marketas)]
elif regionas and marketas:
    filtered_df = df12[df["Region"].isin(regionas) & df12["Marketas"].isin(marketas)]
else:
    filtered_df = df12[df12["Market"].isin(marketas) & df12["Region"].isin(regionas) & df12["State"].isin(valstija) 

Using if statements:

filtered_df = df.copy()
if marketas:
    filtered_df = df[df["Market"].isin(marketas)]
if regionas:
    filtered_df = df[df["Region"].isin(regionas)]
if valstija:
    filtered_df = df[df["State"].isin(valstija)]

Using a dictionary and a for loop.

filters = {"Market": marketas, "Region": regionas, "State": valstija}
filtered_df = df.copy()
for column, values in filters.items():
    if values:
        filtered_df = filtered_df[filtered_df.isin(values)]

Just using a for loop.

for column, values in zip(("Market", "Region", "State"), (marketas, regionas, valstija)):
    if values:
        filtered_df = filtered_df[filtered_df.isin(values)]

I like the dictionary approach myself as it makes it really easy to add new filters to your code. Referring back to your initial post:

filters = {}

def add_filter(df, column):
    """Add filter to filters dictionary.  Return dataframe with all filters applied."""
    options = sorted(set(df[column].values))
    filters[column] = st.sidebar.multiselect(f"Pick you {column}", options)

    filtered_df = df.copy()
    for column, values in filters:
        filtered_df = filtered_df[filtered_df[column].isin(values)]
    return filtered_df

df2 = add_filter(df, "Regionas")

That may look like a lot of extra code, but to add Marketas all you add is:

d2 = add_filter(df, "Marketas")

So having 12 filters all you need is one function and 12 lines of code that call that function

If you are worried that updating df_filtered over and over is inefficient, you can do this.

def add_filter(df, column):
    """Add filter to filters dictionary.  Return dataframe with all filters applied."""

    options = sorted(set(df[column].values))
    filters[column] = st.sidebar.multiselect(f"Pick you {column}", options)

    selected = pd.Series([True] * len(df)):
    for column, values in filters:
        selected &= df[column].isin(values)
    return df.copy()[selected]