Python Forum
Looping Through Large Data Sets
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Looping Through Large Data Sets
#11
I timeit and there is some performance gain, but not as much as I expected

import pandas
import timeit

raw_data = pandas.DataFrame(columns=["Column_0", "Column_1", "Column_2", "Column_3", "Column_4", "Column_5", "Column_6", "Column_7", "Column_8"])
raw_data.loc[0,:] = (315589, "CHZ3", 1100, 218, 694.63, None, -1.589, 0, 1.3694)
raw_data.loc[1,:] = (364048, "CHZ3", 1100, 320, 12.09, None, -7.216, 0, 59.89)

print(timeit.timeit("raw_data['Column_5'] = raw_data['Column_3'] * 8950 + raw_data['Column_4']", setup='from __main__ import raw_data', number=20000))
print(timeit.timeit("raw_data.iloc[:,5] = raw_data.iloc[:,3] * 8950 + raw_data.iloc[:,4]", setup='from __main__ import raw_data', number=20000))
Output:
9.739402721000033 11.304438857000036
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Peculiar pattern from printing of sets SahandJ 7 1,634 Dec-29-2021, 06:31 PM
Last Post: bowlofred
  How does one combine 2 data sets ? detlefschmitt 2 1,675 Sep-03-2021, 03:38 AM
Last Post: detlefschmitt
  Looping to read data in database CEC68 1 1,700 Sep-24-2020, 08:54 PM
Last Post: scidam
  comprehension for sets Skaperen 2 1,856 Aug-07-2020, 10:12 PM
Last Post: Skaperen
  Extract data from large string pzig98 1 2,121 Jul-20-2020, 12:39 AM
Last Post: Larz60+
  Moving large amount of data between MySql and Sql Server using Python ste80adr 4 3,379 Apr-24-2020, 01:24 PM
Last Post: Jeff900
  alternative to nested loops for large data set JonnyEnglish 2 2,552 Feb-19-2020, 11:26 PM
Last Post: JonnyEnglish
  Looping JSON data graham23s 1 2,054 Jul-01-2019, 09:37 PM
Last Post: nilamo
  Sort sets by item values Sergey 4 66,215 Apr-19-2019, 10:50 AM
Last Post: Sergey
  Problem with character sets Pedroski55 4 3,701 Mar-04-2019, 02:35 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020