Sharing and integrating product data typically involves considerable custom coding and/or manual effort facilitated through a patchwork of disparate technologies and processes — data integration, data quality, governance, profiling, de-duping, content enrichment, content correction, outsourcing, custom publishing and a host of other activities.
The DataLens® System incorporates all these functions in a single integrated environment that is designed from the ground-up to resolve the most common and complex problems associated with sharing product data.
Data Integration – Extract, transform, match, load structured and unstructured data
Traditional “Extract, Transform and Load” technology works for well-structured data that can be simply mapped into the target environment. However, many data integration problems — particularly those involving product data — require much more complex “Transformation”:
- Semantic technology recognizes, extracts, standardizes, matches and restructures key information out of highly variable — even freeform — input.
- Extracting meaning can range from simple to complex, such as character-level parsing that’s required for ‘decoding’ of IPNs (intelligent part numbers) where semantic meaning is embedded in the part number.
- In targeting and extracting meaning, it is important to recognize context (lack of ability to do this is the fundamental reason why custom coding rarely succeeds). Whether “HP” means “Hewlett-Packard” or “horse power” is a question of context — and aging context is critical. The DataLens System identifies all information in context to eliminate this type of ambiguity.
- “Extract” and “Load” functions are handled by a simple workflow process.
Data Quality – Standardize, validate, and manage exceptions
Most Data Quality systems were built for name and address data where the semantics and data rules are fairly well established. Product data has highly variable semantics and requires a different approach to determine and improve item quality:
- The DataLens System evaluates every item against a product-specific schema to determine completeness and validity. Exceptions are highlighted and can be resolved in associated business workflows.
- In addition to restructuring data, the data values must also be standardized e.g. “10 hp”, “10.0 HorsePower” are the same; “_” drill – no battery” is equivalent to “0.75-inch drill without battery”; “cyan” and “turquoise” both standardize to “blue”. Individually, these standardizations seem simple, but the sheer number required quickly becomes unmanageable with traditional approaches.
- The DataLens System is not only designed to handle the extreme challenges of unstructured product data, but can also perform the relatively simpler processes such as name standardization.
Data Governance – Guarantee reliability, capture metrics
A successful data governance program blends technology, business practices and organizational changes to ensure the ongoing quality and usability of data. The ability to score data quality at an item and aggregate level is essential to delivering a closed-loop system where quality can be monitored and improved over time.
- The DataLens System monitors quality, completeness and compliance at an item level and engages exception workflows to resolve problem data before it is passed on — effectively creating a “data quality firewall” for downstream systems.
- Aggregate level quality metrics can also be captured to assess the overall quality of a data source and enable closed-loop process improvement.
Content Enrichment – Enrich core content, fill gaps by exception
Often manual effort is used to ‘scrub’ whole data sets, when in reality, only a small proportion really requires the external effort.
- Since manual effort should be used only as a last resort, the DataLens System leverages all available electronic resources to enrich and correct item records.
- Where manual research is required, the DataLens System saves time and effort by pinpointing the individual fields that require attention.
Custom Publication – Custom descriptions, classifications, languages
Data not only comes in many different forms, but is also often required in many different forms. Globalization means that increasingly data is required in many languages.
- The DataLens System’s resident interface allows full control of output — from word order to line lengths and metric to imperial.
- The DataLens System can translate product data from any language to any language — including double-byte languages.
Content Profiling – Understand your data and how it can be used
Data profiling is not new, but usually only discloses simple structure or syntactic data patterns, which is sufficient for structured data with limited variability. But only a semantic (content) profile will reveal the full potential hidden within your product data.
- The DataLens System first uses semantic recognition to classify items into product categories — providing a great baseline for project planning and prioritization.
- Each category can then be profiled for ‘richness’— allowing decisions to be made about how the data can be used and which business processes it will support.
