Organizations often have an branch that needs to deal with a variety of digital input sources that need to be normalized, cleaned up and stored for further processing.

Examples include:

• Market research companies import and consolidate data reported by partners and public sources.
• Regulatory bodies import and consolidate mandatory data reports specific to the industries they supervise – such as health care, banking, or agriculture.
• BI and Analytics teams are continually tasked with importing and maintaining data in cross-department data repositories.
• For many companies, it’s just an operational challenge to process invoices, time sheets, claims, or measurement data that continuously arrive from various sources and in diverse formats.

In each of these cases data from various sources and input formats needs to be normalized and imported into an internal system that processes it further.

Tweakstreet is a great tool to automate the data ingestion process, especially when you have little influence over the data formats data suppliers provide you with, and you have to deal with a diverse pool of inputs.

Drive data ingestion with meta-data

Build a meta-data repository that contains information about the data sources, and prepare a handful of ingestion flows that are each parameterized with information from the meta-data store to handle any specific data import.

Tweakstreet is an ideal platform for automating data ingestion tasks using a meta-data driven approach.

• Tweakstreet flows allow a high degree of parameterization, which gives you increased ability to support meta-data driven solutions and dynamic data-format detection at runtime.
• At the data level, Tweakstreet allows you to adapt to data sources dynamically without hard-coding all details of individual data providers.
• At the orchestration level, Tweakstreet allows you to establish processing queues, priorisation, error handling policies, and monitoring strategies that fit your organization.