Data Preparation

Data sources often provide data on an as-is basis as opposed to what is needed. The shape, format, and encoding of information happens to be whatever is practical for the system that generated the data. This means that data extracts are often cryptic like this:

The information content is far from self-evident. The data uses internal mnemonics, identifiers which need to be looked up, and a non-uniform structure that implies various kinds of records and record relationships are present.

Tweakstreet allows you to automate the data preparation process that takes the data as it comes from the source, and transforms it into a usable asset. The process typically consists of several conceptual phases.

Transformation

The structure of the data set needs to be transformed to fit its intended use. Staying with our example from above, we learn that the data given encodes the following logical structure:

Tweakstreet enables you to form and shape such data structures from raw source records. You would then store them in a manner suitable for your usecase, such as a SQL database, JSON, XML, CSV files, Excel files, or online spreadsheets.

Enrichment

Data sets often contain internal mnemonics or ids that need to be resolved or looked up in a reference system. Who, after all, knows that sku BBL refers to a bucket of blue paint, from the Paints product category. That information has to go into the dataset in order for it to be useful.

Tweakstreet allows you to look up reference data from any data source such as databases, reference files or online APIs.

Cleanup

Most data sets need cleanup before being processed further. Invalid or incomplete records need to be identified – and then corrected or filtered out.

Tweakstreet makes it easy to:

• Identify data exchange format problems
• Guard against unexpected format changes
• Validate data against plausibility rules
• Fix or redirect problematic records
• Collect bad records for further inspection and discussion with data suppliers

Data is only useful when prepared

Whether you’re training a ML model, preparing a custom report, or loading data warehouse tables,
you’ll always need to take raw data as you find it – and make it usable.

With Tweakstreet you can interactively design and automate that process in a visual way.
Turning cryptic data sources into queryable information and therefore into insights.