Data Integration took off around the millenium with the adoption of ETL technologies (Extract, Transform and Load). Sherwin Lake Group consultants, already riding the Business Intelligence wave, were there and caught the train. We participated with a series of different ETL vendors as they seemed to sequentially rise to prominience and accumulate matching feature sets. Over that span we've watched many of our ETL Best Practices and tricks like ABC/ABaC, CDC, Data Profiling and MPP become further absorbed by the ETL and DW platform technoolgies: but not all. Many design principles and considerations like divide and conquer, transient vs. persistent staging, join vs lookup, and upsert vs. bulk load are as pertinent as ever.
Technology advancement is important as we all have and require more data each day. However the real challenge often comes from the one constant in that Data Integration equation: there are only 24 hours in each day. To properly handle that truth we believe wisdom to be the essential ingredient.