Intro
from getl.lift import lift
lift(
spark: SparkSession,
lift_def: str,
parameters: dict,
)
spark
:SparkSession
, usually the global variablespark
in databrickslift_def
:str
, can be a yaml formatted string as lift definition or an s3 path with the yaml fileparameters
:dict
, all${string}
formatted strings will be replaced with the parameters provided here
Examples
Example: Yaml formatted string
yaml_str = """
Parameters:
Path:
Description: Path to import data from
LiftJob:
TrustedFiles:
Type: load::batch_parquet
Properties:
Path: ${Path}
"""
lift(
spark=spark,
lift_def=yaml_str,
parameters={
"Path": "s3://bucket/with/data"
}
)
Example: s3 path
lift(
spark=spark,
lift_def="s3://bucket-name/path/to/file.yml",
parameters={
"Path": "s3://bucket/with/data"
}
)