2021.10
Collibra DIC Integration
Powered By GitBook
File Look Back
As of 2021.11 this option is exposed in the Explorer under the DQ Job section. Users can persist (save) this option by clicking the Union Lookback checkbox.
Could not load image
Click the checkbox to enable -fllb flag
File look back (-fllb) should only be used when a SQL layer is not available. This is considered for advanced use cases, but may not be suitable for all file types and folder structures. Best practice is to expose a date signature somewhere in the file or directory naming convention.

Example

1
-ds "demo_lookback" \
2
-rd "2017-07-29" \
3
-lib "/opt/owl/drivers/mysql" \
4
-cxn "mysql" \
5
-q "select * from lake.dateseries where DATE_COL = '2017-07-29' " \
6
-dc DATE_COL \
7
-dl \
8
-dlkey sym \
9
-dllb 4 \
10
-fllb
Copied!
This look back will load your past 4 runs as your historical training set
File look back is used with deep learning or pattern matching. In the example above it is used with deep learning.
1
-fllb
Copied!
This is often used with files and in conjunction with -adddc in cases where a date column is not in an ideal format or you do not have a date column on the given dataset
Despite the name, this can be used with file or database storage formats.
Last modified 8d ago
Copy link
Contents
Example