Python Forum
Thread Rating:
  • 1 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
csv filtering with python.
#1
I am attempting to take a csv file containing test results for products and filtering the results so that I know what product was tested and its serial number.

Each product has 5-10 tests run on it so i have to first eliminate all duplicates.

@upload_csv_blueprint.route('/upload_csv', methods=['GET','POST'])
@login_required
def upload_file():
    if request.method == 'POST':
        csvfile = request.files['file']
        s_n_bank = []
        main_bank = []
        p_n_bank = []
        reader = csv.DictReader(csvfile)

        for row in reader:
            temp_s_n=str(row['product_serial_number'])
            if temp_s_n not in s_n_bank:        # Filters the serial
                s_n_bank.append(temp_s_n)       # numbers so no 
                sorted_bank= s_n_bank.sort()    # duplicates. 
                single_entry= sorted_bank.pop() # Does not recognise .pop()

 # attempting to use the refined serial number list to extract the relevant part number. 
 #( UNTESTED ->)  
                for row in reader:
                    selected_entry= row.index(single_entry)    

                    temp_p_n = str(selected_entry['part_number'])
                    if temp_p_n not in p_n_bank:
                        p_n_bank.append(temp_p_n)
                        main_bank.append(p_n_bank)
                        main_bank.append(single_entry)
                        p_n_bank.remove(temp_p_n)

                        print main_bank
I have the error message - AttributeError: 'NoneType' object has no attribute 'pop'.
Any idea what my pop error is? Is the function obsolete ?
I am also aware that this is probably not the best way of conducting this search and am not certain that it will work ( as have not tested fully due to the pop error.) Any prompts on future errors i might find would be appreciated.
Reply
#2
First of all - always post the full traceback in error tags.
now, the error
.sort() method will sort in place and will return None. That is why you get the error.
you want to use sorted function, e.g. sorted_bank= sorted(s_n_bank)
or replace lines#15-16 with

s_n_bank.sort()    # duplicates. 
single_entry= s_n_bank.pop() 
all that said, there are way better ways to remove duplicates, e.g. see set

can you provide sample data as well as desired output.

Finally, it would be good to put the filtering in separate function
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
A simplified version of the csv would be something along the lines of:

date_time test_location part_number product_serial_number test_detail test_result
2017/08/23 12:09:10 London 04X-1101-0116 1633-0001 Overall Fail
2017/08/23 12:12:20 Glasgow 04X-1101-0115 1633-0020 Overall Pass
2017/08/23 12:09:55 London 04X-1101-0114 1633-0002 Overall Pass
2017/08/23 12:09:55 Somewhere Magical 04X-1101-0113 1633-0050 Overall Pass
2017/08/23 12:12:50 Glasgow 04X-1101-0112 1633-0021 Overall Pass
2017/08/23 12:09:10 London 04X-1101-0116 1633-0001 Self Test Fail
2017/08/23 12:12:20 Glasgow 04X-1101-0115 1633-0020 Self Test Pass
2017/08/23 12:09:55 London 04X-1101-0114 1633-0002 Self Test Pass
2017/08/23 12:09:55 Somewhere Magical 04X-1101-0113 1633-0050 Self Test Pass
2017/08/23 12:12:50 Glasgow 04X-1101-0112 1633-0021 Self Test Pass

I would like an output similar to:
04X-1101-0116 1633-0001
04X-1101-0114 1633-0002
04X-1101-0115 1633-0020
04X-1101-0112 1633-0021
04X-1101-0113 1633-0050
Reply
#4
is it comma delimited or no delimited at all? your example has no commas and the default dialect is 'excel', so it should have commas...
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#5
Sorry, did not realise i had opened in libreofice and not textedit.

It will be comma delimited . and in csv format not excel.
Reply
#6
csv file:


import csv

def filter_csv(csv_file):
    reader = csv.DictReader(csv_file)
    return set((row['part_number'], row['product_serial_number']) for row in reader)


@upload_csv_blueprint.route('/upload_csv', methods=['GET','POST'])
@login_required
def upload_file():
    if request.method == 'POST':
        csvfile = request.files['file']
        for product in filter_csv(csv_file=csvfile):
            print(','.join(product))
tested with
with open('products.csv') as csvfile:
    for product in filter_csv(csv_file=csvfile):
        print(','.join(product))
Output:
04X-1101-0116,1633-0001 04X-1101-0114,1633-0002 04X-1101-0112,1633-0021 04X-1101-0113,1633-0050 04X-1101-0115,1633-0020

by the way, 'excel' is just name of one of the available dialects in csv module. I didn't think it's excel file
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#7
Thanks, works perfectly and a lot more condense then my original :)

Pretty new to this so was not aware that excel is another dialect :P. Thanks for your help,

Seem to know your stuff when it comes to python / flask / sql. Do you know of any good online resources for a beginner looking to expand knowledge ?
Reply
#8
you can read about dialects here: https://docs.python.org/3/library/csv.ht...parameters

(Jun-26-2018, 10:30 AM)KirkmanJ Wrote: Seem to know your stuff when it comes to python / flask / sql. Do you know of any good online resources for a beginner looking to expand knowledge ?
it's hard to point out something in particular. I prefer reading the docs. for flask - https://blog.miguelgrinberg.com/post/the...ello-world

check our list of free python resources
https://python-forum.io/Thread-A-List-of...-Resources
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#9
Thank you :)
Reply
#10
if you prefer you can use set comprehension
{(row['part_number'], row['product_serial_number']) for row in reader}
instead of
set((row['part_number'], row['product_serial_number']) for row in reader)

because it is shorter :-)
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Unexpected Output - Python Dataframes: Filtering based on Overlapping Dates Xensor 5 655 Nov-15-2023, 06:54 PM
Last Post: deanhystad
  Help with filtering in Python Junes786 2 81,466 Jun-05-2019, 08:10 AM
Last Post: Junes786

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020