It’s no secret that data is the foundation of any functional organization. Given this fact, a couple of questions arise. First, is the business making the most optimal decisions possible with the data that is available? Second, how can a business ensure that its data is of the highest quality? It turns out that on an annual basis, the average organization loses a whopping $8.2 million due to bad data! *
However, attaining high quality data is an ambitious task. With millions of records to account for, it is no small feat to accomplish accurate data movement to a data warehouse. It is here that Data Quality Assurance (Data QA), which performs these very behind-the-scene validations, comes to our rescue.
Data warehouse projects can be demanding, hence proper resourcing is crucial, when undertaking such challenging implementation. While it is necessary to have well-seasoned ETL (Extract Transform Load) developers, it isn’t enough. As with any software development, quality assurance testers act as the driving force to ensure that when the product reaches the customer, it is fit for their immediate consumption. This fact stands just as true when implementing downstream systems like DataHub and InfoCenter. In short, your data warehousing team will absolutely require experienced Data QA testers to safeguard data quality.
Data QA needs to possess more than just knowledge of the application. Ability to test case design, test execution and gain mastery over data manipulation are key skills required for Data QA. Prowess in SQL is a must for the Data QA; having the skillset to query databases is a prerequisite to ensure that proper data validation is taking place. If a business wants to ensure that its data is clean on a granular level, right down to the record, then having a Data QA on the team, with a knack for scripting relatively complex SQL queries is the way to do it. The Data QA takes on the demanding but fruitful task of building a library of queries that serve to validate every ETL load that is run. With a Data QA performing these validations, you can be confident that your reporting is being done with clean, tested data.
On a functional level, the Data QA role exists to dispel the concern of data quality by testing the movement of data from source to target, even if that means testing ten thousand records. ETL developers can develop and run as many loads as required, but there isn’t much value if the incoming data lacks quality. Data QA brings to the table a skillset of data manipulation, which when applied, can verify whether the data is truly consumable for the business.
When data gets loaded into the enterprise data warehouse, namely, InfoCenter, QA will have done due diligence in running multiple testing scenarios not only around the ETL process but around report generation as well. Data management QA can answer questions such as, did all records come through as expected? Is the data accurate? Did the data transform exactly as the business requires it to? Is the BI reporting reflective of the data source?
When Data QA has completed their job, not only will you be able to access and report on all pertinent data from policy, claims, billing applications, but that data will be accurate, integrated, and trustworthy. After all, data of such kind is the backbone of a thriving enterprise.
*(source: Gartner, Inc.)