Find duplicates in a pandas dataframe list column on other rows

[Calab]

I have a pandas dataframe where one of the columns is a list of values. I need to identify if any item in the list occurs in that list in other rows.

in other words, on the first row, take the first value in the list in the "items" column and check the "items" column in all the other rows to see if that value occurs. If it does, I want to know the "name" from the first row, the "name" from the row where the match occurs, and the value we were looking for. Repeat for the each value in the "items" list in the first row. Repeat for all the rows.

df = pd.DataFrame([{"name": "abc", "items": ["1234", "5678", "9012"]},
                   {"name": "def", "items": ["3456", "7890"]},
                   {"name": "ghi", "items": ["9876", "1234"]},
                   {"name": "jkl", "items": ["5678", "7890", "2468"]}
                  ])

I want:
    [["abc", "ghi", "1234"],
     ["abc", "jkl", "5678"],
     ["def", "jkl", "7890"]
    ]

I could iterate through the lists, iterating over all the rows, but I feel that pandas should be able to handle at least some of the work here.

What should I be doing to achieve the result I'm looking for?

[snippsat]

I could iterate through the lists, iterating over all the rows, but I feel that pandas should be able to handle at least some of the work here.

Yes there is usually better way(think in a vectorized way),as loop/iterating can solve it but performance will usually not good at all in Pandas.
Something like this.

import pandas as pd

df = pd.DataFrame([
    {"name": "abc", "items": ["1234", "5678", "9012"]},
    {"name": "def", "items": ["3456", "7890"]},
    {"name": "ghi", "items": ["9876", "1234"]},
    {"name": "jkl", "items": ["5678", "7890", "2468"]}
])

# Explode the 'items' column
df_exploded = df.explode('items')
df_merged = pd.merge(df_exploded, df_exploded, on='items')
# Exclude rows where the names are the same
df_merged = df_merged[df_merged['name_x'] != df_merged['name_y']]
# Remove duplicate pairs by sorting the names and dropping duplicates
df_merged['sorted_names'] = df_merged.apply(
    lambda row: tuple(sorted([row['name_x'], row['name_y']])),
    axis=1
)
df_unique = df_merged.drop_duplicates(subset=['items', 'sorted_names'])
result = df_unique[['name_x', 'name_y', 'items']].values.tolist()
print(result)

explode function in pandas is not strictly a vectorized method,but is a efficient and practical approach.
Can also mention Polars worth looking into if in future need performance.

[Calab]

Yes there is usually better way(think in a vectorized way),as loop/iterating can solve it but performance will usually not good at all in Pandas.
Something like this.

import pandas as pd

df = pd.DataFrame([
    {"name": "abc", "items": ["1234", "5678", "9012"]},
    {"name": "def", "items": ["3456", "7890"]},
    {"name": "ghi", "items": ["9876", "1234"]},
    {"name": "jkl", "items": ["5678", "7890", "2468"]}
])

# Explode the 'items' column
df_exploded = df.explode('items')
df_merged = pd.merge(df_exploded, df_exploded, on='items')
# Exclude rows where the names are the same
df_merged = df_merged[df_merged['name_x'] != df_merged['name_y']]
# Remove duplicate pairs by sorting the names and dropping duplicates
df_merged['sorted_names'] = df_merged.apply(
    lambda row: tuple(sorted([row['name_x'], row['name_y']])),
    axis=1
)
df_unique = df_merged.drop_duplicates(subset=['items', 'sorted_names'])
result = df_unique[['name_x', 'name_y', 'items']].values.tolist()
print(result)

explode function in pandas is not strictly a vectorized method,but is a efficient and practical approach.
Can also mention Polars worth looking into if in future need performance.

Thanks... looks good, but I just found out that we only have Pandas v 0.24.2 as this machine is running Python2. explode() isn't an option, but you given me some ideas.