cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DQ anomaly detection : _quality_monitoring_summary table DDL

noorbasha534
Contributor III

Dears

Does anyone have the DDL for _quality_monitoring_summary table?

This is created by the DQ anomaly detector. Since the detector was trying to create a managed table which is not allowed in the environment I work, I am attempting to create this on my own from the documentation https://learn.microsoft.com/en-us/azure/databricks/lakehouse-monitoring/data-quality-monitoring#debu...

But i run into data types mismatches errors.

appreciate if anyone can share this table. Thank you...

4 REPLIES 4

dataoculus_app
New Contributor II

here is the schema i can see on my workspace. but honestly its not really covering much and fires lot of queries. check it out.

image.png

noorbasha534
Contributor III

Hey, thanks a lot. Can you please share the DDL here...thanks again in advance..

@dataoculus_app Can you please share the DDL here...thanks again in advance..

Yogesh_378691
New Contributor III

Hi,

The _quality_monitoring_summary table is an internal table created by the Data Quality Anomaly Detector in Databricks Lakehouse Monitoring. Unfortunately, the full DDL is not publicly documented in detail, and trying to manually create it can lead to data type mismatches, as you've encountered.

However, you can try the following workaround:

Enable the anomaly detector in a test environment where managed tables are allowed.

Let it create the _quality_monitoring_summary table automatically.

Use the following command to extract the DDL:


SHOW CREATE TABLE <catalog>.<schema>._quality_monitoring_summary;

This will give you the exact DDL with the correct data types.

If setting up a test environment isn't feasible, an alternative is to create an external table and configure the Lakehouse Monitoring to use it by specifying a different target location using Unity Catalog or your desired catalog pathโ€”ensuring the system has permissions to write to that location.

 

 

 

 

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now
OSZAR »