Skip to main content

Dataform YAML generation for BigQuery

serge
Updated

SqlDBM can turn your BigQuery data models into Dataform-ready YAML, so you don't re-type structure, documentation, or data-quality checks by hand. Generate declarations for your existing source tables, or fully documented table and view actions — with descriptions, labels, partitioning, clustering, and assertions carried over from your model. Your engineers add only the SQL logic, and your model stays the single source of truth from design to pipeline.

Before you start

  • Dataform YAML generation is available for BigQuery projects only.
  • Decide which output you need:
    • Sources — declarations that register your existing BigQuery tables in Dataform. These carry names, descriptions, and columns, and work in Dataform as generated.
    • Actions — table and view definitions that Dataform builds. These carry descriptions, columns, tags, labels, partitioning, clustering, data-quality assertions, and key constraints. You add the SQL for each action before running Dataform.

Generate the YAML

  1. Open your BigQuery project in SqlDBM.
  2. Go to Forward Engineering.
  3. Select the objects you want to include, the same way you select them for DDL generation.
  4. In Generation options, set Format to Dataform YAML.
  5. Set Type to Sources or Actions.
  6. Generate. SqlDBM produces one combined YAML file containing all selected objects.

Object names, descriptions, and columns fill in automatically from your model, including nested STRUCT fields, which appear as multi-part column paths. Table or view type is inferred from each object, and physical properties like labels, partitioning, and clustering come from the BigQuery properties already defined on your objects.

 

Use the file in Dataform

  1. In your Dataform workspace, add the generated file to the definitions/ folder.
  2. Sources: no further edits needed — declarations compile and run as generated.
  3. Actions: for each table or view, add a filename: line pointing to a SQL file, then create that file with the transformation SQL. Use an explicit .sql extension — the Dataform editor defaults to .sqlx, which registers a duplicate action.
  4. Compile and run. Primary and foreign keys apply through post-operations using fully qualified table names, and foreign-key dependencies are declared so parent tables build first.