3 minute read
Data ingestion is of vital importance and can be achieved through different patterns. Understanding these patterns is a must to support the development of business critical applications.
Data ingestion is a critical technology for all classes of businesses and organizations.
As the world increasingly churns out data in various levels of complexity and volume, there has to be a way to extract more value out of it.
To achieve this aim, technology providers have to make use of various types of data ingestion patterns. Let’s take a look at 5 data ingestion patterns that are an absolute boon for modern app development.
Data ingestion pattern #1: Point-to-point
Point-to-point data ingestion establishes the direct connection between the data’s source and its target.
In this pattern, the changed data is moved without causing interference to the database workload. It is also referred to as a real-time data ingestion pattern.
Data is ingested in real-time.
Finds applicability in time-sensitive use cases.
The data ingestion pipeline in this pattern allows for rapid operational decision-making based on the current data. It is the most implemented pattern by the majority of enterprises because of the simple architecture.
Data ingestion pattern #2: Hub and spoke
The hub and spoke data ingestion pattern has the source and target decoupled. Here, data ingestions only travel within the hubs.
These are then responsible for managing connections and performing data transformations.
This pattern minimizes the frequency of data ingestion connection, allowing for a simplified operational environment.
It suits most applications where requirements frequently change in an operational environment. The downside of this pattern has a limit in terms of size.
A hub is logical, and therefore vast enterprises may face challenges with performance and latency using this approach.
Data ingestion pattern #3: Batching
The batching data ingestion pattern focuses on collecting data and then transferring it in batches.
Batches are sent according to set schedules but not immediately.
The ingestion layer of this pattern can be set to collect data as a response to trigger events, simple scheduling, or some other types of logical ordering.
When crafting such a system, it must be able to support scaling to process data at peak volumes.
Data ingestion pattern #4: Lambda
Lambda data ingestion pattern incorporates both the point-to-point and batch ingestion methodologies.
It is highly desirable for its ability to provide broad views of the batch data.
When you need time-sensitive data, this pattern can step in and provide you with the viewpoints you need.
It achieves its abilities by allowing 3 different configurations:
- Batch;
- Serving, and;
- The speed layer.
The first two index the available data in batches whereas the third indexes data that hasn’t been picked up by the first two configurations.
A continuous hands-off between the 3 configurations allows minimal latency.
Data ingestion pattern #5: Sharding
Sharding pattern relies heavily on the software architecture you have available. The pattern results in a datastore separated from single storage instances known as shards.
As data is ingested, it is divided up to fulfill some sort of sharing logic.
When attempting to look up data, queries are performed against these shards.
Sharding is a superb data ingestion pattern for large, distributed applications, as it permits quick execution of commands and questions.
Which data ingestion pattern to consider
Deciding appropriate data ingestion is critical for any enterprise to make sense of ever-increasing data.
A real-time data ingestion pattern is simple to implement and suitable when insights out of data are required almost in real-time.
When requirements are changing rapidly, the hub-and-spoke pattern should be given consideration.
The batch data ingestion pattern is the most appropriate for large enterprises where data sources comprise legacy systems.
To handle variety, velocity, and volume best approach is to leverage the lambda pattern. Considering sharding data ingestion pattern when ecosystem comprises of distributed applications.
A knowledge engine like Datavid Rover enables you to connect your data across different cloud providers such as AWS buckets, Microsoft Sharepoint, and more.
The best part is, that you can easily extract valuable insights from the data while ingesting it all through the program’s simple user interface.
Frequently asked questions
