Python Forum
get year information from a timestamp data frame
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
get year information from a timestamp data frame
#1
Hi all,
I am new to python.
I am reading a datafile where there is timestamp values as string.
I want to ger distinct years from this dataframe and keep them in an array.
I have some trial below that don't work.
Could you give a help about how to do it

import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('pyspark-by-examples').getOrCreate()
from pyspark.sql.types import StructType,StructField, StringType, IntegerType,ArrayType
from pyspark.sql.functions import split, explode
import pyspark.sql.types 

import calendar
import datetime
import pandas as pd
from pyspark.sql import functions as F
from pyspark.sql import types as T
import datetime as dt 

arrayData = spark.read.format("delta").load("/mnt/datalake/....something")
#arraySchema = StructType([ \
   # StructField("repair_year",StringType(),True), \
  #])

arrayData['repair_year']= arrayData.select('repair_date').withColumn("repair_date", F.col("repair_date").cast(T.TimestampType()))



#df = arraySchema
#df.printSchema()
#df.show()

arraySchema.show()
buran write Jan-08-2021, 04:52 PM:
Please, use proper tags when post code, traceback, output, etc. This time I have added tags for you.
See BBcode help for more info.
Reply
#2
import datetime
import time

# create a timestamp -- you won't have to do this as you already have timestamp
timestamp = time.time()
print(f"\n\ntimestamp: {timestamp}")

year = datetime.date.fromtimestamp(timestamp).year
print(f"Year: {year}")
Output:
timestamp: 1610140280.269422 Year: 2021
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Iterating Through Data Frame Rows JoeDainton123 3 1,020 Aug-09-2021, 07:01 AM
Last Post: Pedroski55
  Filtering a data frame according to number of occurences of an observation Menthix 1 409 May-31-2021, 10:50 PM
Last Post: supuflounder
  Working with Timestamp in Mulitindex krischanb 0 460 Jan-22-2021, 12:52 PM
Last Post: krischanb
  update column in one data frame with value of column from another data frame flexer 0 519 Dec-04-2020, 03:29 PM
Last Post: flexer
  Extracting Rows From Data Frame and Understanding The Code JoeDainton123 0 585 Aug-03-2020, 04:08 PM
Last Post: JoeDainton123
  data frame excel cell calulation buunaanaa 1 841 Jul-04-2020, 06:00 PM
Last Post: buunaanaa
  Knowing the index of a data frame Ivannovix 1 813 May-01-2020, 02:51 PM
Last Post: klllmmm
  Conditional Cumsum in pandas data-frame klllmmm 1 2,176 May-01-2020, 02:21 PM
Last Post: klllmmm
  Replace last 4 bytes of a timestamp mPlummers 18 3,115 Sep-17-2019, 12:47 AM
Last Post: Larz60+
  loop through python pandas data frame Johnse 4 1,241 Sep-02-2019, 01:45 AM
Last Post: Johnse

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020