What Is a Data Pipeline?
A data pipeline is a series of data processing steps. If the data is not currently loaded into the data platform, then it is ingested at the beginning of the pipeline. Then there are a series of steps in which each step delivers an output that is the input to the next step. This continues until the pipeline is complete. In some cases, independent steps may be run in parallel.
Data pipelines consist of three key elements: a source, a processing step or steps, and a destination. In some data pipelines, the destination may be called a sink. Data pipelines enable the flow of data from an application to a data warehouse, from a data lake to an analytics database, or into a payment processing system, for example. Data pipelines also may have the same source and sink, such that the pipeline is purely about modifying the data set. Any time data is processed between point A and point B (or points B, C, and D), there is a data pipeline between those points.
As organizations look to build applications with small code bases that serve a very specific purpose (these types of applications are called “microservices”), they are moving data between more and more applications, making the efficiency of data pipelines a critical consideration in their planning and development. Data generated in one source system or application may feed multiple data pipelines, and those pipelines may have multiple other pipelines or applications that are dependent on their outputs.
Consider a single comment on social media. This event could generate data to feed a real-time report counting social media mentions, a sentiment analysis application that outputs a positive, negative, or neutral result, or an application charting each mention on a world map. Though the data is from the same source in all cases, each of these applications are built on unique data pipelines that must smoothly complete before the end user sees the result.
Common steps in data pipelines include data transformation, augmentation, enrichment, filtering, grouping, aggregating, and the running of algorithms against that data.
Analysis Of Big Data Pipeline
As the volume, variety, and velocity of data have dramatically grown in recent years, architects and developers have had to adapt to “big data.” The term “big data” implies that there is a huge volume to deal with. This volume of data can open opportunities for use cases such as predictive analytics, real-time reporting, and alerting, among many examples.
Like many components of data architecture, data pipelines have evolved to support big data. Big data pipelines are data pipelines built to accommodate one or more of the three traits of big data. The velocity of big data makes it appealing to build streaming data pipelines for big data. Then data can be captured and processed in real time so some action can then occur. The volume of big data requires that data pipelines must be scalable, as the volume can be variable over time. In practice, there are likely to be many big data events that occur simultaneously or very close together, so the big data pipeline must be able to scale to process significant volumes of data concurrently. The variety of big data requires that big data pipelines be able to recognize and process data in many different formats—structured, unstructured, and semi-structured.
Data Pipeline vs. ETL
ETL refers to a specific type of data pipeline. ETL stands for “extract, transform, load.” It is the process of moving data from a source, such as an application, to a destination, usually a data warehouse. “Extract” refers to pulling data out of a source; “transform” is about modifying the data so that it can be loaded into the destination, and “load” is about inserting the data into the destination.
ETL has historically been used for batch workloads, especially on a large scale. But a new breed of streaming ETL tools are emerging as part of the pipeline for real time streaming event data
Data Pipeline Considerations
Data pipeline architectures require many considerations. For example, does your pipeline need to handle streaming data? What rate of data do you expect? How much and what types of processing need to happen in the data pipeline? Is the data being generated in the cloud or on-premises, and where does it need to go? Do you plan to build the pipeline with microservices Are there specific technologies in which your team is already well-versed in programming and maintaining?
Architecture Examples
Data pipelines may be architected in several different ways. One common example is a batch-based data pipeline. In that example, you may have an application such as a point-of-sale system that generates a large number of data points that you need to push to a data warehouse and an analytics database. Here is an example of what that would look like:
Another example is a streaming data pipeline. In a streaming data pipeline, data from the point of sales system would be processed as it is generated. The stream processing engine could feed outputs from the pipeline to data stores, marketing applications, and CRMs, among other applications, as well as back to the point of sale system itself.
A third example of a data pipeline is the Lambda architeccture which combines batch and streaming pipelines into one architecture. The Lambda Architecture is popular in big data environments because it enables developers to account for both real-time streaming use cases and historical batch analysis. One key aspect of this architecture is that it encourages storing data in raw format so that you can continually run new data pipelines to correct any code errors in prior pipelines, or to create new data destinations that enable new types of queries.
HOW TO DESIGN DATA PIPELINE IN GOOGLE CLOUD PLATFORM
Every data ingestion requires a data processing pipeline as a backbone. A data processing pipeline is fundamentally an Extract-Transform-Load (ETL) process where we read data from a source, apply certain transformations, and store it in a sink. For the article’s context, we will provision GCP resources using Google Cloud APIs. For the readers who are already familiar with various GCP services, this is what our architecture will look like in the end …
Let’s go through details of each component in the pipeline and the problem statements we faced while using them.
Problem 1: Persisting Streaming Data
A data stream is a set of events generated from different data sources at irregular intervals and with a sudden possible burst. The first challenge with such a data source is to give it a temporary persistence. The GCP component we chose to deal with this is Cloud Pub/Sub.
PubSub is GCP’s fully managed messaging service and can be understood as an alternative to RabbitMQ or Kafka. So, in layman terms, it’s a Queue.
In most of the streaming scenarios, the incoming traffic streams through an HTTP endpoint powered by a routing backend. We will persist all of the traffic in the PubSub from where it can be consumed subsequently. It provided us the following benefits,
- PubSub can store the messages for up to 7 days. So in the case of downstream consumer failure, we get the persistence guarantee and the traffic can be replayed again.
- Traffic can be fully routed to multiple consumers downstream with support for all the custom routing behavior just like RabbitMQ.
Publishing events to a PubSub topic (read Queue) is as simple as the code snippet given below
'''
Set GOOGLE_APPLICATION_CREDENTIALS to point to IAM credentials file before running this.
Run `pip install google-cloud-pubsub` to install client lib.
'''import time
from google.cloud import pubsub_v1# TODO project_id = "Your Google Cloud Project ID"
# TODO topic_name = "Your Pub/Sub topic name"# Initiate client and set topic name
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_name)# Send the data to PubSub
future = publisher.publish(topic_path, data="Test PubSub Message")# Prints a server-generated ID (unique within the topic) on success
print(future.result())
Our pipeline till this point is looking like this,
Problem 2: Streaming To Batch Conversion
Processing streaming data in realtime requires at least some infrastructure to be always up and running. In one of our major use cases, we decided to merge our streaming workload with the batch workload by converting this data stream into chunks and giving it a permanent persistence. Once persisted, the problem inherently becomes a batch ingestion problem that can be consumed and replayed at will. The second component we chose for this is Cloud Dataflow.
Dataflow is GCP’s fully managed service for streaming analytics based on the Apache Beam programming model which supports a wide variety of use-cases for ETL pipelines. You can visit the list of templated use-cases here. We used a custom version of the PubSub-To-CloudStorage-Text template (built inhouse).
As the documentation states, Apache Beam is an open-source model for defining both parallel streaming and batch processing pipelines with simplified mechanics at big data scale. In this model, the pipeline is defined as a sequence of steps to be executed in a program using the Beam SDK. The program can then be run on a highly scalable processing backend of choice. Some of the popular options available are Google Cloud Dataflow, Apache Spark, Apache Flink, etc. (full list). The model gives the developer an abstraction over low-level tasks like distributed processing, coordination, task queuing, disk/memory management and allows to concentrate on only writing logic behind the pipeline.
Let’s take a quick look at code for defining an Apache Beam pipeline,
The pipeline here defines 3 steps of processing.
Step 1: Read the input events from PubSub.
Step 2: By using the event timestamp attached by PubSub to the event, group the events in the fixed-sized intervals.
a) To understand the concepts of event-time, windowing, and watermarking in-depth, please refer to the official Apache Beam documentation.
Step 3: Write each group to the GCS bucket once the window duration is over.
Once run, all the low-level details of executing this pipeline in parallel and at scale will be taken care of by the Dataflow processing backend.
For understanding on how to run the pipeline demonstrated above or how to write your Dataflow pipeline (either completely from scratch or by reusing the source code of predefined templates), please refer to Template Source Code section of the documentation given below,Google-provided streaming templates | Cloud Dataflow | Google CloudEdit descriptioncloud.google.com
Among other benefits, while using Dataflow, these were the major ones we observed,
- No concerns for the availability of PubSub consumers as it is fully managed. A new instance will be immediately spawned if the previous one goes down.
- On-demand horizontal autoscaling based on workload with support for worker instance-type and max-workers customizations.
- Apache beam’s inbuilt support for windowing the streaming data to convert it into batches.
The only con we observed was,
- It is not possible to automatically scale down Dataflow workers to 0 for streaming data based on the workload. That means, there will be at least one compute instance always up and running to read from PubSub. It will cost a bare minimum of $50 per month (as per current GCP pricing for 1 instance with minimum compute power).
- A possible workaround to this problem is to programmatically kill and restart Dataflow jobs on a need basis. Refer to this.
On Google Cloud console, the Dataflow job looks like this,
Dataflow can be configured to write data into logical components, or windows. After grouping individual events of an unbounded collection by timestamp, these batches can be written to a Google Cloud Storage (GCS) bucket.
GCS is a managed object store service provided by GCP. Consider it as an alternative to Amazon’s S3. This provided our data a permanent persistence and from here all the batch processing concepts can be applied.
Our pipeline till this point in continuation is looking like this,
In this part of the blog, we saw how we converted our high-velocity streaming data to batch data using managed GCP services without developing anything from scratch. In the next part II of this blog, we will see how we can do slicing and dicing on this data and make it available for final consumption.
Security Considerations:
- Setting the Dataflow pipeline’s “usePublicIps” flag to true has severe security implications. This will create Compute instances in the worker pool with public IPs exposed to the internet, which can lead to DDoS attacks.
- Hence it is recommended to create a private subnet in the parent GCP project and set the “usePublicIps” flag to false while creating the pipeline. This will cause Dataflow service to fallback to the private subnet and use private IPs by default.
Drop your comment
