Retail Data Conversions; Part Art, Part Science. Which is more important?
Sunday, June 28th, 2009
Data conversions must be exact, precise, methodical….WAIT, this all sounds very scientific and not creative at all. So what part of a data conversion can possibly have anything to do with ‘ART’? Let me explain…
Retail data conversions can be complex. I am sure this is true for most industries, but as retail is my industry domain, that is what I am focusing on. Converting critical data like UPCs, prices, customers, etc, can have a huge impact on your organization, if not done correctly! I have worked on numerous data conversions for retailers over the years, and I feel like I have encountered every problem imagineable. Yet, new problems are encountered each and every time. Each conversion is different. No matter the methodology followed, the testing prepared for, unique situations arise. In so doing, we must adapt to the situation and this is where the creativity, the ‘art’ (and fun) begins. Here are a few prime examples:
- When a default is not a simple default - by definition a default should be simple. Not necessarily so. Providing data for all fields in a new system is not always possible and therefore we set defaults. But simple defaults will not always provide the best results. Maybe the default can be based on other important data or dates and provide a more meaningful ‘complex’ default. In order to have a better result, it is preferable to use as precise a default as possible.
- Using complex data maps - how creative can you be in building data maps that result in better and more meaningful output. A data map can be a simple one-one to map (replace the old value with this new value) or it can be fairly creative and complex. I recently had to map an old GL code structure with multiple parts to a new GL code with a differnet set of parts, where each part had a diffent series of rules and data masks to build the new G/L code from the old to the new - now, that was fun!!
- Finding a home for old data - Sometimes the residing system just does not have a place to store a piece of information from the old system. When that old information is valuable and you want to carry it forward, you must find a place for it to live, where it can be accessed and continue to be meaningful. A couple of options may be; to use a user-defined field, or put in another field that is not being used and give that field a new meaning, or merge the information into another field while keeping its meaning. All viable and creative solutions to a common data conversion problem.
- Resolving errors - and this is where the most creative part of the process occurs. Integrity is king! You have built a complex set of data conversions and business rules that are all inter-connected and must work in concert to get a complete result with full integrity. Sure, you can say if you built the conversions correctly to begin with there will be no errors and no discrepancies, but my experience tells me otherwise. Rejects, errors, and discrepancies are a natural part of data conversions, but how you deal with it is what becomes the differentiating factor. Some rejects or discrepancies can be fine, if they are expected or can be explained/justified. In other cases, the source of an error needs to be identified and then corrected. Once the source of the problem is found a correction is needed - corrections can be as simple as a small business rule change or as complicated as requiring a conversion re-write or additional data/conversions that might not have been anticipated. The goal is to ‘create’ a resolution that will get the desired result without disrupting the entire flow and integrity of the entire conversion process.
So there you have it, data conversions are not simply scientific, and a creative component is almost always needed, sometimes equally, if not more important. This just highlights a few examples of this and I am sure there are others you may have experienced that are also relevant? So which do you think is more important? Art or science? What have your experiences showed?