Command Line

Scale + Data Science
Where Scale meets Data Science.  Scale linearly with your data by adding executors and/or memory
1
-f "file:///Users/home/salary_data.csv" \
2
-d "," \
3
-rd "2018-01-08" \
4
-ds "salary_data"
5
-numexecutors 2 \
6
-executormemory 2g
Copied!
Yarn Master
If Owl is run on an edge node on a popular hadoop distribution such as HDP, CDH, EMR it will automatically register the jobs with Yarn Resource Manager.
Spark Master
Owl can also run using spark master by using the -master input and passing in spark:url
Spark Standalone
Owl can run in standalone most but naturally will not distribute the processing beyond the hardware it was activated on.
Options
Description
deploymode
spark deploymode option
drivermemory
driver memory example 3G for local space
executorcores
spark executor cores
executormemory
spark executor memory option example 3G
master
overrides local[*], i.e. spark://myhost:7077, yarn-client, yarn-cluster
sparkprinc
kerberos principal name ex: [email protected]
Use Spark-Submit directly bypassing OwlCheck
1
spark-submit \
2
--driver-class-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \
3
--driver-library-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \
4
--driver-memory 3g --num-executors 2 --executor-memory 1g \
5
--master spark://Kirks-MBP.home:7077 \
6
--class com.owl.core.cli.OwlCheck /opt/owl/bin/owl-core-trunk-jar-with-dependencies.jar \
7
-u user -p pass -c jdbc:postgresql://xyz.chzid9w0hpyi.us-east-1.rds.amazonaws.com/postgres \
8
-ds accounts -rd 2019-05-05 -dssafeoff -q "select * from accounts"
9
-driver org.postgresql.Driver -lib /opt/owl/drivers/postgres42/  
Copied!
Parallel JDBC Spark-Submit
1
spark-submit \
2
--driver-class-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \
3
--driver-library-path /opt/owl/drivers/postgres42/postgresql-42.2.4.jar \
4
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///opt/owl/config/log4j-TRACE.properties \
5
--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///opt/owl/config/log4j-TRACE.properties \
6
--files /opt/owl/config/log4j-TRACE.properties \
7
--driver-memory 2g --num-executors 2 --executor-memory 1g --master spark://Kirks-MBP.home:7077  \
8
--class com.owl.core.cli.OwlCheck /opt/owl/bin/owl-core-trunk-jar-with-dependencies.jar \
9
-u us -p pass -c jdbc:postgresql://xyz.chzid9w0hpyi.us-east-1.rds.amazonaws.com/postgres \
10
-ds aumdt -rd 2019-05-05 -dssafeoff -q "select * from aum_dt" \
11
-driver org.postgresql.Driver -lib /opt/owl/drivers/postgres42/  \
12
-connectionprops fetchsize=6000 -master spark://Kirks-MBP.home:7077 \
13
-corroff -histoff -statsoff \
14
-columnname updt_ts -numpartitions 4 -lowerbound 1557597987353 -upperbound 1557597999947
Copied!

FAQs

DQ Job JDBC

Last modified 18d ago

Copy link

Edit on GitHub

Contents

Use Spark-Submit directly bypassing OwlCheck

Parallel JDBC Spark-Submit