Azure DataBricks

Run a DQ check on any file in Azure Blob

Read the File by setting up the azure key.
1
spark.conf.set("fs.azure.account.key.abcCompany.blob.core.windows.net","GBB6Upzj4AxQld7cFv7wBYNoJzIp/WEv/5NslqszY3nAAlsalBNQ==")
2
3
val df = spark.read.parquet("wasbs://[email protected]/FILE_NAME/20190201_FILE_NAME.parquet")
4
Copied!
Process the file using Owl
1
// register in Owl Catalog, Optional
2
val owl = new Owl(df).register
3
4
// run a full DQ Check
5
owl.owlCheck()
Copied!
Additional imports and input options
1
import com.owl.core._
2
import com.owl.common._
3
4
val props = new Props()
5
props.dataset = datasetName
6
props.runId = 2019-03-02
7
props..... // look at the many input options
Copied!
Data Quality | Collibra
Collibra
Last modified 20d ago
Copy link