Data Integration took off around the millenium with the adoption of ETL technologies (Extract, Transform and Load). Sherwin Lake Group consultants, already riding the Business Intelligence wave, were there and caught the train. We participated with a series of different ETL vendors as they seemed to sequentially rise to prominience and accumulate matching feature sets. Over that span we've watched many of our ETL Best Practices and tricks like ABC/ABaC, CDC, Data Profiling and MPP become further absorbed by the ETL and DW platform technoolgies: but not all. Many design principles and considerations like divide and conquer, transient vs. persistent staging, join vs lookup, and upsert vs. bulk load are as pertinent as ever.
Technology advancement is important as we all have and require more data each day. However the real challenge often comes from the one constant in that Data Integration equation: there are only 24 hours in each day. To properly handle that truth we believe wisdom to be the essential ingredient.
Some hold a perception that Data Integration and analysis in the cloud requires the use of Big Data technologies. Sure you can do that: we often do; but you don't have to. True many of the traditional ETL technologies we know from before the cloud aren't well architected for SaaS, but not all. At Sherwin Lake Group we can transform and integrate your in-house data with external content to the cloud using a myriad of approaches.