dumbdatascientist

UPDATE <TABLENAME> SET COLNAME = "VALUE" WHERE 1 = 1 Gave the error message: Invalid column name 'VALUE' which was quite ambiguous took a while to figure out that string should be in single quote. The correct syntax is: UPDATE <TABLENAME> SET COLNAME = ' VALUE' WHERE 1 = 1

First set properties  ALTER TABLE < tableName > SET TBLPROPERTIES (   'delta.minReaderVersion' = '2' ,   'delta.minWriterVersion' = '5' ,   'delta.columnMapping.mode' = 'name' ) Then it is possible to rename the column alter table < tableName > rename column < colName > to < newcolName >

  # Delete all versions of a model: client = mlflow.tracking.MlflowClient() model_name = " model_name " # get the range manually for i in range( 35 , 43 ):     # convert first to archived stage else can't be deleted     client.transition_model_version_stage(         name = model_name,         version = i,         stage = " archived "     )     # delete the version     client.delete_model_version(model_name, i)

 Problem: Attempting to convert a string of digits representing an ID into an integer resulted in all values being null.  Solution: The ID values exceeded the range for integer casting. Casting them as bigint resolved the issue.

Problem: Need to find out the time till next action date in a table which has columns debtorid and action date.  solution: Define a window specification over 'debtorid' and order by 'date'  window_spec = Window.partitionBy(' debtorId ').orderBy(f.col(' date ').desc()) Add a column with the difference in days between consecutive dates. The lag function is used to get the value of the 'date' column from the previous row within the specified window. df = df.withColumn(' days_till_next_action ', f.datediff(F.lag(' date ').over(window_spec), ' date '))