Python Forum
get year information from a timestamp data frame
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
get year information from a timestamp data frame
#1
Hi all,
I am new to python.
I am reading a datafile where there is timestamp values as string.
I want to ger distinct years from this dataframe and keep them in an array.
I have some trial below that don't work.
Could you give a help about how to do it

import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('pyspark-by-examples').getOrCreate()
from pyspark.sql.types import StructType,StructField, StringType, IntegerType,ArrayType
from pyspark.sql.functions import split, explode
import pyspark.sql.types 

import calendar
import datetime
import pandas as pd
from pyspark.sql import functions as F
from pyspark.sql import types as T
import datetime as dt 

arrayData = spark.read.format("delta").load("/mnt/datalake/....something")
#arraySchema = StructType([ \
   # StructField("repair_year",StringType(),True), \
  #])

arrayData['repair_year']= arrayData.select('repair_date').withColumn("repair_date", F.col("repair_date").cast(T.TimestampType()))



#df = arraySchema
#df.printSchema()
#df.show()

arraySchema.show()
buran write Jan-08-2021, 04:52 PM:
Please, use proper tags when post code, traceback, output, etc. This time I have added tags for you.
See BBcode help for more info.
Reply
#2
import datetime
import time

# create a timestamp -- you won't have to do this as you already have timestamp
timestamp = time.time()
print(f"\n\ntimestamp: {timestamp}")

year = datetime.date.fromtimestamp(timestamp).year
print(f"Year: {year}")
Output:
timestamp: 1610140280.269422 Year: 2021
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how do you style data frame that has empty rows. gsaray101 0 500 Sep-08-2023, 05:20 PM
Last Post: gsaray101
  googletrans library to translate text language for using data frame is not running gcozba2023 0 1,165 Mar-06-2023, 09:50 AM
Last Post: gcozba2023
  Trying to get year not the entire year & time mbrown009 2 862 Jan-09-2023, 01:46 PM
Last Post: snippsat
  Load multiple Jason data in one Data Frame vijays3 6 1,500 Aug-12-2022, 05:17 PM
Last Post: vijays3
  conditionals based on data frame mbrown009 1 873 Aug-12-2022, 08:18 AM
Last Post: Larz60+
  error in timestamp Led_Zeppelin 3 3,106 Jun-15-2022, 08:28 PM
Last Post: deanhystad
  error in timestamp Led_Zeppelin 0 980 Jun-10-2022, 07:59 PM
Last Post: Led_Zeppelin
  Merging two Data Frame on a special case piku9290dgp 0 1,069 Mar-02-2022, 10:43 AM
Last Post: piku9290dgp
  Save data frame to .csv df.to.csv() mcva 1 1,497 Feb-03-2022, 07:05 PM
Last Post: mcva
  Move a particular row in pandas data frame to last row klllmmm 0 3,647 Dec-27-2021, 09:11 AM
Last Post: klllmmm

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020