Issues with Querying and writing into a External Storage in a new databricks workspace

[dipanjannet]

Hello All,

We have recently created a databricks with UC enabled. All the VNET Pairing is completed. Now, while I am trying to create a table [Managed] under a catalog / schema - I am not able to query the table. I am getting below error 

org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient  

Also, while I am trying to write a Dataframe into an External location, I am seeing the same issue as mentioned. Any idea what is wrong here? Please note - as I will be using Unity catalog, nothing has been done (configuration) with respect to hive_metastore. Is this some hive_metastore related configuration issue ?

my write code

# Write the DataFrame to the specified Delta table

df_out.write.format("delta").mode("overwrite").option(

    "path", "abfss://[email protected]/test_data/tbd/"

).saveAsTable("schema_name.table_name")

[Shua42]

Hi @dipanjannet ,

This could be due to the code falling back to the hive metastore, which could be due to a number of reasons, but the first thing that caught my eye was that the table name in your saveAsTable() call isn't fully qualified. Can you add the catalog name to the function call like saveAsTable("catalog_name.schema_name.table_name"), and if you haven't created a catalog yet I would start there.

[dipanjannet]

Hello @Shua42 - Actually I was using catalog only. The code is little modified one. Can you please help me understand - what do you mean by this - "This could be due to the code falling back to the hive metastore, "

[Shua42]

The legacy Databricks Hive metastore is used as a fallback for queries when Unity Catalog is unavailable or when specific external locations are configured with fallback mode enabled, so it could be something in the configuration here. Were you able to create a catalog and schema in UC before writing the table?