Maja Ferle

Too often I meet data warehouse development teams who are implementing what they call »data quality« in order to help the business have cleaner data to work with. But what they are really implementing is »data gatekeeping« which hurts the business even more than if they didn't have any data quality in the first place. Let's look at an example. A data quality rule may say that the value for education level in the customer table must exist in the education level reference table. Fair enough, right? We must ensure that all reference data values in our tables match their corresponding values in the reference tables. What happens when we receive a customer record with a value for education level that doesn't exist in the education level reference table? Our data gatekeepers say: »No, this record is not allowed to enter the data warehouse because it violates a data quality rule.« Fair enough, right? But think about it. We are not loading a customer...

All of us who work with data could donate some of our expertise to charitable causes through non-profit organizations. The organization DataKind helps to bring together volunteers with organizations who need data related services. I decided to attend an event organized by DataKind's European operation DataKindUK based in London. The event, which was basically a hackathon, took place the last weekend of July and was named the "DataDive". The event was well attended, both by volunteers and by organizations who wanted our help. Four organizations were chosen to present their requirements, goals and expectations. They each brought their own data and made it available to the volunteers. I decided to help the organization HelpAge International who was looking for meaningful data visualizations to be used in their brochures and marketing materials. Their aim is to bring attention to the issues that older people face when trying to make a decent living in all the countries in the world. The event in London had a very hacker feel. Each...

A surprising revelation related to data warehousing was that Facebook uses a traditional data warehouse based on relational technology. The news is interesting because Facebook is one of the representatives of the new generation of companies who deal with very large amounts of data and therefore don't want relational technology because it is too cumbersome for their needs. They have been inventing new database technologies that can handle such large amounts of data, have fast response times and they typically don't support the widely used query language SQL. At the recent The Data Warehousing Institute conference, a representative from Facebook explained that they decided to use relational technology for their data warehouse because the right technology should be used for the right purpose. Newer database technologies, especially those that utilize the Hadoop platform like Facebook does, are better at handling large amounts of unstructured data. Relational databases, on the other hand, require a more strict database structure that typically includes constraints on data elements...