csv filtering with python. - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: csv filtering with python. (/thread-11168.html) Pages:
1
2
|
csv filtering with python. - KirkmanJ - Jun-26-2018 I am attempting to take a csv file containing test results for products and filtering the results so that I know what product was tested and its serial number. Each product has 5-10 tests run on it so i have to first eliminate all duplicates. @upload_csv_blueprint.route('/upload_csv', methods=['GET','POST']) @login_required def upload_file(): if request.method == 'POST': csvfile = request.files['file'] s_n_bank = [] main_bank = [] p_n_bank = [] reader = csv.DictReader(csvfile) for row in reader: temp_s_n=str(row['product_serial_number']) if temp_s_n not in s_n_bank: # Filters the serial s_n_bank.append(temp_s_n) # numbers so no sorted_bank= s_n_bank.sort() # duplicates. single_entry= sorted_bank.pop() # Does not recognise .pop() # attempting to use the refined serial number list to extract the relevant part number. #( UNTESTED ->) for row in reader: selected_entry= row.index(single_entry) temp_p_n = str(selected_entry['part_number']) if temp_p_n not in p_n_bank: p_n_bank.append(temp_p_n) main_bank.append(p_n_bank) main_bank.append(single_entry) p_n_bank.remove(temp_p_n) print main_bankI have the error message - AttributeError: 'NoneType' object has no attribute 'pop'. Any idea what my pop error is? Is the function obsolete ? I am also aware that this is probably not the best way of conducting this search and am not certain that it will work ( as have not tested fully due to the pop error.) Any prompts on future errors i might find would be appreciated. RE: csv filtering with python. - buran - Jun-26-2018 First of all - always post the full traceback in error tags. now, the error .sort() method will sort in place and will return None . That is why you get the error.you want to use sorted function, e.g. sorted_bank= sorted(s_n_bank) or replace lines#15-16 with s_n_bank.sort() # duplicates. single_entry= s_n_bank.pop()all that said, there are way better ways to remove duplicates, e.g. see set can you provide sample data as well as desired output. Finally, it would be good to put the filtering in separate function RE: csv filtering with python. - KirkmanJ - Jun-26-2018 A simplified version of the csv would be something along the lines of: date_time test_location part_number product_serial_number test_detail test_result 2017/08/23 12:09:10 London 04X-1101-0116 1633-0001 Overall Fail 2017/08/23 12:12:20 Glasgow 04X-1101-0115 1633-0020 Overall Pass 2017/08/23 12:09:55 London 04X-1101-0114 1633-0002 Overall Pass 2017/08/23 12:09:55 Somewhere Magical 04X-1101-0113 1633-0050 Overall Pass 2017/08/23 12:12:50 Glasgow 04X-1101-0112 1633-0021 Overall Pass 2017/08/23 12:09:10 London 04X-1101-0116 1633-0001 Self Test Fail 2017/08/23 12:12:20 Glasgow 04X-1101-0115 1633-0020 Self Test Pass 2017/08/23 12:09:55 London 04X-1101-0114 1633-0002 Self Test Pass 2017/08/23 12:09:55 Somewhere Magical 04X-1101-0113 1633-0050 Self Test Pass 2017/08/23 12:12:50 Glasgow 04X-1101-0112 1633-0021 Self Test Pass I would like an output similar to: 04X-1101-0116 1633-0001 04X-1101-0114 1633-0002 04X-1101-0115 1633-0020 04X-1101-0112 1633-0021 04X-1101-0113 1633-0050 RE: csv filtering with python. - buran - Jun-26-2018 is it comma delimited or no delimited at all? your example has no commas and the default dialect is 'excel', so it should have commas... RE: csv filtering with python. - KirkmanJ - Jun-26-2018 Sorry, did not realise i had opened in libreofice and not textedit. It will be comma delimited . and in csv format not excel. RE: csv filtering with python. - buran - Jun-26-2018 csv file: import csv def filter_csv(csv_file): reader = csv.DictReader(csv_file) return set((row['part_number'], row['product_serial_number']) for row in reader) @upload_csv_blueprint.route('/upload_csv', methods=['GET','POST']) @login_required def upload_file(): if request.method == 'POST': csvfile = request.files['file'] for product in filter_csv(csv_file=csvfile): print(','.join(product))tested with with open('products.csv') as csvfile: for product in filter_csv(csv_file=csvfile): print(','.join(product))
by the way, 'excel' is just name of one of the available dialects in csv module. I didn't think it's excel file RE: csv filtering with python. - KirkmanJ - Jun-26-2018 Thanks, works perfectly and a lot more condense then my original :) Pretty new to this so was not aware that excel is another dialect :P. Thanks for your help, Seem to know your stuff when it comes to python / flask / sql. Do you know of any good online resources for a beginner looking to expand knowledge ? RE: csv filtering with python. - buran - Jun-26-2018 you can read about dialects here: https://docs.python.org/3/library/csv.html#dialects-and-formatting-parameters (Jun-26-2018, 10:30 AM)KirkmanJ Wrote: Seem to know your stuff when it comes to python / flask / sql. Do you know of any good online resources for a beginner looking to expand knowledge ?it's hard to point out something in particular. I prefer reading the docs. for flask - https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world check our list of free python resources https://python-forum.io/Thread-A-List-of-Free-Python-Resources RE: csv filtering with python. - KirkmanJ - Jun-26-2018 Thank you :) RE: csv filtering with python. - buran - Jun-26-2018 if you prefer you can use set comprehension {(row['part_number'], row['product_serial_number']) for row in reader} instead of set((row['part_number'], row['product_serial_number']) for row in reader) because it is shorter :-) |