Databricks Community

BigRoux

Data Engineers, Share Your Experiences with Delta Lake Schema Evolution! We're calling on all data engineers to share their experiences with the powerful schema evolution feature in Delta Lake. This feature allows for seamless adaptation to changin...

BigRoux

Here are some considerations/ideas you might be interested in: Automatic Caching Mechanism Interference: Yes, there is a possibility that an automatic caching mechanism interferes with your dataset. Spark employs several caching mechanisms: The Data...

BigRoux

Let me dig in and see if I can find anything for you on this. Cheers, BigRoux.

BigRoux

Serverless is management free which means you cannot choose the image. Hope this helps. Lou.

BigRoux

We will get back to you shortly.

BigRoux

The OutOfMemory (OOM) issue you're experiencing in your PySpark job could stem from several factors. Here's a breakdown of potential causes and mitigation strategies: Skew in Data Partitions: Based on your calculation, the data size per partition is...

Databricks Community

User Stats

User Activity

What are your most impactful use cases for schema evolution in Databricks?

Re: Dataframe is getting empty during execution of daily job with random pattern

Re: Lakehouse federation for SQL server: database name with spaces

Re: Spark connect client and server versions should be same for executing UDFs

Re: Smaller dataset causing OOM on large cluster

Re: Smaller dataset causing OOM on large cluster