What is a data pipeline? | Getting Started + FAQs | Building a Data Agency | Products | CIFL

What is a data pipeline?

What is a data pipeline?

LINKS

CIFL post: WTF is a data pipeline?

We refer to a "data pipeline" as a combination of:

Raw data feeds (from Google Analytics or other APIs)
A data warehouse (we prefer BigQuery)
SQL data models (written in dbt) that run the necessarily analysis in your warehouse
Data visualizations, that read the modeled data in your warehouse

The pipeline encompasses this entire ecosystem, from raw data through to visualizations.

Building a Data Agency

Buy nowLearn more

Getting Started + FAQs

***ALL THE TEMPLATE LINKS***

**GETTING HELP**

The business of data, end-to-end

Why'd we republish this course?

Who is this for, and what will you learn?

What is a data pipeline?

[THROWBACK ALERT] WTF is ADP?

Additional (FREE) CIFL Courses

Getting Started with BigQuery SQL

Data Studio the Lazy Way

0.1 The Sales Flow

0.1.1 Finding your niche

0.1.2 Building an inbound content strategy

0.1.3 Why we arrived at the sprint pricing model

0.1.4 Why we publish pricing

0.1.5 The initial sales call1

0.1.6 The roadmapping process1

0.1.7 Deal closing + contracts

0.2 Staffing & Resourcing

0.2.1 The sprint flow + roles

0.2.2 Hiring reporting + data modeling analysts

COMING SOON - Making a staffing plan & budget

1.1 Planning Development Sprints

1.1.0 Meet the Tracking Plan

1.1.1 Breaking the roadmap into a Tracking Plan

1.1.2 Mapping our raw data source requirements

1.1.3 What you'll do with data source schemas

1.1.4 Mapping out data source schemas

1.1.5 Populating key starter questions for reporting

1.1.6 Scoping out each Site or Client

2.1 Data Feeds - Getting Started

2.1.1 Intro to data feeds

2.1.2 BigQuery initial setup

2.1.3 Setting up your BigQuery tables

2.1.4 Pushing data from Sheets to BigQuery

2.1.5 Supermetrics quickstart for beginners

2.1.6 Pulling data from unsupported APIs into the Tracking Plan

2.2 Data Feeds - Stitch

2.2.1 Stitch initial setup

2.2.2 Pulling GA data using Stitch

2.2.3 Pulling Adwords data using Stitch

2.2.4 Pulling FB Ads data using Stitch

3.1 Intro to dbt

3.1.1 Intro to dbt for SQL data modeling

3.1.2 Planning your Data Models

3.1.3 Creating your dbt project

3.1.4 Connect your BigQuery database to dbt

3.1.5 Managing your dbt project via Github

3.2 Data modeling with dbt

3.2.1 Writing your 'processing' level SQL queries

3.2.2 Writing your 'join' level SQL models

3.2.3 Sidenote: on debugging dbt models

3.2.4 Pro tip: standardizing URL structure

3.2.5 Pro tip: using dbt macros

3.2.6 Writing your 'admin' level SQL models

3.2.7 Writing your 'math' and 'visualization' level SQL models

COMING SOON - Data documentation in dbt

COMING SOON - Data + schema testing in dbt

3.3 Productionalizing your dbt project

3.3.1 Intro to productionalizing your pipeline

3.3.2 Using dbt cloud to run your SQL models on a schedule

3.3.3 Scheduling your data pipeline orchestrations

3.3.4 Testing changes to your data pipeline

3.3.5 QCing data using Supermetrics as a check

4.1 Visualizations in Data Studio

4.1.1 The "PDA" reporting design framework

4.1.2 Designing reports in the Tracking Plan

4.1.3 Executing the reporting build

4.1.4 Reviewing reporting

4.1.5 Pulling data from BigQuery into Sheets

5.1 Sprint wrapup + review

5.1.1 Conditions for closing out a sprint

5.1.2 Wiring up reporting with live data

5.1.3 Sharing draft models + visualizations with clients

5.1.4 Transitioning to support mode

Congrats!

Wapow! You made it.

Interested in working with CIFL?