Feb-18-2017, 04:15 PM
Hi
I am using pyspark, and whenever I attempt to use a comprehension it generates an error. At first I thought the issue was pyspark specific, but I think I am not coding the comprehension correctly.
The following works as expected in pyspark:
Any suggestions would be helpful.
Thanks
I am using pyspark, and whenever I attempt to use a comprehension it generates an error. At first I thought the issue was pyspark specific, but I think I am not coding the comprehension correctly.
The following works as expected in pyspark:
def reGrpLst(fw_c): fw,c = fw_c f,w = fw return (f,[(w,c)]) f_wcL_RDD = fw_c_RDD.map(reGrpLst)but when I attempt to rewrite using a comprehension as either
f_wcL_RDD = fw_c_RDD.map(lambda fw_c: [ (fw[0], (fw[1] ,fw_c[1]) ) for fw in fw_c[0] ] )or
f_wcL_RDD = fw_c_RDD.map(lambda fw_c: [ (f, (c ,fw_c[1]) ) for f,c in fw_c[0] ] )A pyspark error is generated. Is it because the comprehension, in both of the above cases, are incorrect?
Any suggestions would be helpful.
Thanks