Pyspark : Window Function

Problem: Need to find out the time till next action date in a table which has columns debtorid and action date. 

solution:

Define a window specification over 'debtorid' and order by 'date' 

window_spec = Window.partitionBy('debtorId').orderBy(f.col('date').desc())

Add a column with the difference in days between consecutive dates. The lag function is used to get the value of the 'date' column from the previous row within the specified window.

df = df.withColumn('days_till_next_action', f.datediff(F.lag('date').over(window_spec), 'date'))