Python Forum
How to insert different types of data into a function
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to insert different types of data into a function
#1
I'm trying to design a function that will insert both a different "oldTable", string, and column name for each iteration. The "withColumn" calculation below works fine, but "withColumnRenamed" and the "where" line do not.

What I want, for example with newTable1, is "oldVar2" renamed to "string1_newVar2" and any rows with null values in the "oldVar_dropNull" variable dropped.

import pyspark.sql.functions as F

def functionName(x,y,z):
    return x.withColumn("newVar1", F.when(F.col("oldVar1") > 0, x.oldVar1*100/x.oldVar1)\
                                                    .otherwise(0)) \
               .withColumnRenamed("oldVar2", (y,"_newVar2")) \
               .where(F.col(z).isNotNull())
        
newTable1 = functionName(oldTable1,"string1","oldVar_dropNull")
newTable2 = functionName(oldTable2,"string2","oldVar_dropNull")
Some sample data:

import pandas as pd

df = {'oldVar1':['18.50', '649.27', '523.52'],
      'oldVar2':['24.56', '4564.56', '34.45'],
      'oldVar_dropNull':['12.54', '656.89', '0']
     }
 
oldTable1 = pd.DataFrame(df)
print(oldTable1)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  What data types can I use for default values? Mark17 1 571 Oct-09-2023, 02:07 PM
Last Post: buran
  In SQL Server, mix data types. shiv11 0 908 Sep-21-2022, 12:50 PM
Last Post: shiv11
  I need to add data types to cython conversion python to c Good_AI_User 1 1,051 Aug-19-2022, 07:52 AM
Last Post: Gribouillis
  How to read rainfall time series and insert missing data points MadsM 4 2,248 Jan-06-2022, 10:39 AM
Last Post: amdi40
  insert more data at once in MongoDB Leon79 2 2,377 Jul-05-2020, 01:34 PM
Last Post: Leon79
  List of mixed data types to bytes medatib531 1 2,374 Mar-16-2020, 11:53 AM
Last Post: scidam
  Insert data into sql after joining two excel data from python NeerajYadav 2 2,090 Jan-29-2020, 02:29 PM
Last Post: parthi1705
  How to insert data if not exists in mysql? farah97 0 2,851 Dec-29-2019, 08:32 AM
Last Post: farah97
  Changing Data Types BallisticSwami 2 2,443 Jun-27-2019, 01:17 PM
Last Post: BallisticSwami
  Type error when reading in different data types on an __init__ method Dylanmull 3 2,834 May-09-2019, 02:05 PM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020