Your Report 2.0: A Data Engineer’s Guide to Migrating BI Reports from Old to New Systems: Part 3, QA

In the last post, we discussed actually building the data extract to be consumed by the reporting tool of choice.  If you haven’t read the previous posts, you can find them here:

  1. Phase 1:
  2. Phase 2:

The next phase is crucial: Unit testing and QA.  If these steps are glazed over, it will result in many iterations of back and forth with business users, which is frustrating to all involved.  A wise data leader once described the data mantra he lives: “Get it working, get it right, get it fast”.  If the extract is running and producing data, it is ‘working’.  So now is time to get it ‘right’.


Before even running the extract and loading data into the reporting tool, I like to validate the data first by simply looking at it and doing some simple unit tests.  The first step to validating the data is attempting to find a source of truth.  Hopefully, if you are migrating an existing report, you can be provided with a historic version of that report and a data extract to compare to.  If this is a brand new report, or you are sourcing from a brand new system, you might have to work with the business to manually validate your first pass.  Once you have you source of truth, pick one specific record, or if it is an aggregated report, drop to the lowest granularity possible, and just make sure it looks everything looks good.  Pick a specific record by ID, and ask the data some questions: Is the record there?  Do all the fields match your source of truth? Do the datatypes and data formats match up (dates in the same format?).  If you are looking at an aggregate, are your totals at the lowest granularity accurate?  If not, you will have to dive in and take a look at how you are aggregating the data and see if something is missing, a calculation is off, or if your ETL is losing records or data. After you are happy with your first record, extend it to other records, and if you have other base ‘types’ or records, try those out as well.

During this time, you might discover bugs in the old report.  I have seen this frequently.  The best thing you can do in this scenario is to write up a detailed ‘report’ of the discrepancy, with 1-3 detailed examples and screenshots, and meet with the business to ‘prove’ your version is actually accurate. This will require stepping through the entire ETL process, from source to extract, so it can be quite time consuming. This happens more often than you would expect, so getting good at explaining these technical discrepancies to the business is an invaluable skill.

Once you have unit tested at the record level, you should unit test within a specific time range.  Some things to make sure: Are record counts matching? Do aggregates across a specific range work?  Basically, this is a good time to look at the data across a larger span, to make sure you have all the data you need.

After you have completed your unit testing, and you are pretty confident in the quality of your data extract, it is time to integrate with the reporting tool.  Ideally, this would be the time for a QA person to jump in and test out the report itself. It is always good to have a third party test your work, and remember not to take bugs personally!  They are completely unavoidable.  But if you do have to be your own QA, you will want to spin through the entire report.  Are the charts populating properly? Do all the filters and functions work? If you have historical data, and you can compare an old version of the report to the new version, now is the time to do it! Another piece of integration testing you can do is making sure the extract in the report works properly when you are doing incrementals. So if you have dates as a parameter in your extract, try running it multiple times with overlapping dates.  If you are getting duplicates, there is likely an issue in your source query itself, or in your merge statements.  Read a prior blog post here about identifying issues with incremental merges.

The final step in the process should always be business validation (User acceptance testing).  Once you are confident that the data looks good in the report, have a business user confirm them same.  Have them test it over a period of time (perhaps days), so you can confirm updates are all working properly, and that no issues might arise over time.  Once the business gives the green light, you are ready to push the report to production, thus beginning phase 4: Support and Enhancements.



Your Report 2.0: A Data Engineer’s Guide to Migrating BI Reports from Old to New Systems: Part 2, Building the Extract

Requirements in hand, you are ready to start the fun of actually building the report extract to match the data mapping contract document you produced earlier!

If you haven’t yet, I highly recommend reading part one of this series here: Part 1: Requirements Gathering

We left off completing the gap analysis for the report.  More than likely, this activity resulted in needing to make several changes in your new system to support the report.  For example, if you are sourcing from a newly built Data Warehouse, it is possible that the Data Warehouse does not have all the data that you need.  Thus, there might be enhancements required to the Warehouse before you can source the data.  Or, if you are sourcing from a Datamart that feeds off a Data Warehouse, you might need to modify the ETL that moves data to the Datamart.  However you source your data, the first step before building the extract is to get the data you need loaded to location you are going to source the report off of.

Sample Data Flow

Some other considerations at this point:

  1. Are your ETL’s running frequently enough to support reporting refresh requirements? For example, if your report requires to be refreshed hourly, but your ETLS that load the Datamart only run overnight, you are likely going to need to update the schedule.
  2. Are there any aggregations that should be performed in the ETL rather than in the report itself? If you have a blazing fast Data Warehouse, you should use it!
  3. Do you have all the historic data you need? Does the historic data change? If not, perhaps consider hosting a historic view of the data somewhere for speedier consumption by the reporting tool.  Or, just keeping that historic data in the tools data store itself.  We will discuss incrementals further in the next sections.
  4. Your goal is to get the data into the easiest and most simple form as possible for consumption into the report. Try to avoid using the reporting tool as another layer of ETL, as it wasn’t built for that purpose.

Once you have all the data in the final location you are going to source the report from, you are ready to write the actual report extract. If you have completed all the previous steps, this step should be quick and easy! Your report extract will consist of writing some sort of a view or stored procedure that exposes the data in the exact format the reporting tool requires.  The more you can expose the data to exactly what the report needs, the faster the report will run, and the snappier it will be on the front end.  I always suggest decoupling the extract from the source tables in the Data Warehouse or Data Mart, such that if you have to make changes to the model, it doesn’t affect the report extracts. You can do this easily in the form of a view or a stored procedure.

If you are going to run the extract for just certain period of time, or if the report you are building takes a historic look at the data for all time, you are likely going to want to implement the extract as an incremental procedure.  Elsewise, it will take an extremely long time for the extract to run, and even more time for the data to cross the internet and land in the report.  The easiest way (but not the only way, and there are always exceptions) to set up incrementals is the following:

  1. Make sure you have metadata on the tables in your Data Warehouse or Datamart.  At the least, this should include a ‘created date’ and a ‘last updated date time’.  This is a best practice for data management in general, not just for writing report incrementals.
  2. Pick your ‘main’ table / unique identifier.  What is the main table you want to drive the incrementals? For example, if you have a report that reports on Emergency Room visits, it will likely be your main ‘Visit’ table.
  3. Have the reporting tool pass your extract procedure the MAX last updated date that it has been sent by previous runs.
  4. Your procedure should take the MAX last updated date as a parameter, and only return records that have a ‘last updated date’ > the MAX of the last update.
  5. The report will have to do a merge into whatever data store it uses one the back-end. By merge, I mean if the record exists, update it, if not, insert it. There is SQL syntax for this that can be found here.

If you are getting merge errors in your incrementals, you might want to read my blog post on troubleshooting this issue:

The benefits of incrementals are immense – it will result in much fast processing and snappier report refreshes. Note that if you don’t have the ability to do the merging on the reporting tool end, and you are not dealing with millions of records, you can do the merging into a permanent table on your Warehouse or Mart that will house this data in the format that the report requires.  Then you can have the report pull all of the data. This should only be done with reports that do not use large volumes of data.

As you are running through all of these considerations, be mindful of any changes you are making and how they will affect the reporting tool.  Sometimes the requirement is to not change the report code as much as possible, in which case you should strive to match the old process / extract as closely as possible.  This usually happens when you have a short time frame.  However, if there are performance improvements / bugs you can fix by using the above points, and you have the time, definitely consider them!

Also, ALWAYS include header comments / comment your code! This gives full visibility to what the code does, what changes were made, when, and why they were made.  I use something simple like this, but feel free to whatever fits your aesthetic:

--AUTHOR:  Ryan Kennedy
--CREATED: 02/16/2019
--DESC:    Exposes the data extract for the Very Important Business Report
--[Ryan K - 2/17/19] Descriptive comment of changes made and why

Once the extract is ready to go, you can move on the Phase 3: Unit testing and QA. (Coming soon)

Your Report 2.0: A Data Engineer’s Guide to Migrating BI Reports from Old to New Systems: Part 1

As more and more companies go to the cloud, build data warehouse solutions, and become more data-centric, we as BI Developers and Data Engineers will find ourselves migrating reports to pull data from new systems, rather than old, deprecated ones.  Take a healthcare example: Say the healthcare company you work for runs their electronic medical records on Epic software, but they recently purchased several other hospitals, some of which use different software.   To report at an enterprise level, they build a data warehouse solution to bring all the data from these systems to one place.  Now, instead of sourcing reports out of all the systems separately, we want to pull all of the data from the data warehouse.

In this series of blog posts, I will walk through the process / some tips and tricks that I use when migrating a report from an old system to a new system. The main phases of this process are:


The first phase of this process should always be requirements gathering.  The better your requirements, the smoother the rest of the process will go.  I always like to start this process with a kickoff session, where I invite several different people who are all stakeholders in the success of this report migration:

  1. Business Users – Who are the business users who are going to be using this report regularly?
  2. Product / Area owners – is there a product owner who specializes in this subject matter? This person is usually an intermediary between the business owner and the BI team.
  3. BI Developer – If you aren’t the BI Developer yourself, you should definitely include whoever is going to be doing the front end of the report.
  4. Source System Data Specialist – This is rare to have, but if you do have someone like this to invite, it will definitely help!

In the kickoff session, I always try to discuss and accomplish the following:

  1. Timeline – roughly what is the requirement from a timeline perspective? How long will it take? Are there any hard deadlines?
  2. Functional explanation – get the business to explain what the report is, why they use it, and what decisions they make based off of it.  The better you can understand the business function behind the report, the easier it will be to understand the data.
  3. What are the different views in the report? Are there multiple tabs? One Tab? Are there aggregations? Have the business walk through the report and explain it.
  4. What is the time period the report shows? Is it just today’s data? Is it a rolling year? Is it monthly? Yearly?
  5. What level is the data at? Is it aggregated at a month level? What is the lowest granularity of the report? For example: In a report about financial transactions, is the report at the item level (eg – each line will represent one item that rolls up to a single ‘order’)? Or is it at the order level?
  6. Parameters and filters: Are the parameters passed into the report? What filters are used in the report? These are key data points you need to make sure are accurate.
  7. Schedule: How often does the report need to refresh? Daily, hourly, weekly? This will determine the schedule of the ETL’s, and if there are dependencies to your ETL’s, you will also have to take these into consideration.

After the kickoff, you will have a pretty solid understanding of the report.  At this point, there are some other considerations I like to explore and bring up if they seem relevant:

  • Is the report definition still valid? For example, if the report was built 15 years ago, it is very likely that some things have majorly changed in the interim.  Business users are often hesitant about changing reports, but it can’t hurt to ask the question.
  • Is all the data in the old report still used? We don’t want to be sending / including anything extraneous. Use this as an opportunity to trim things down.

Next comes documenting all of your findings. By the end of the requirements gathering phase, I always like to have two documents in hand.

The first is a design document.  Essentially, this document should consist of all the main points you gathered in the kickoff session.  Keep in mind that this should be a working document – You can have the business ‘sign it off’ in the beginning, but I always keep a ‘key decisions’ section that I update if we make any changes along the way, with a description of why the changes were made. This will allow you to keep a clear history of how the report changed over time and can always be referred to if you get any questions.

Next is a data contract / mapping document. The data contract should be given to you by the business / product manager.  The contract should contain a list of all the required fields, their data types, the level of the data, and the definitions of the fields. If they have mappings to old systems (tables / columns / etc), these should be provided as well.  If you cannot get a data contract, you may have to revert to reverse engineering the legacy report.  If this is the case, make sure to tack on extra time to your estimates!

Here is an example of what your report mapping could look like:


Note: You can always switch the source system to be on the left, and the target system on the right, it is all a matter of preference.  The above is better if you might have multiple source systems going to one target system / report.  But at the end of the day, feel free to pick your poison!

With your data contract in hand, you can begin the crucial exercise of gap analysis, which is essentially mapping the fields in the data contract to the new source system. Do you have all the required fields at the required granularity? Are you missing anything? If so, you will need to work with the business to either find out how to source the data, or move forward without that data. Will you need to write / modify any of the ETL to support this new report?  The gap analysis is important, because if there is a lot of work involved in procuring the required data for the report, the whole process could take longer, and it is vital to set expectations with the business.

By the end of this phase, you should have your documents completed and agreed to by the business / product owner.  Of course things are likely to change, but it is always a good idea to have signed off documentation, so that if things do change, you can explain why.  Once you are set with your signed off documentation, you are ready for phase 2: Building the report!

I have included below some document templates to get started.  Enjoy!

Design Document Template


My Top 5 Features of Azure Data Studio

I was recently at Azure Data Fest at the Boston Microsoft MTC Campus, and one of the tools they highlighted was Azure Data Studio. I took a spin through the tool afterwards and wanted to share what I found to be 5 of the most exciting features in Data Studio.

Before diving into my 5 – I want to preface this by saying that at the moment, Data Studio is not a complete replacement of SSMS.  There are definitely missing features (profiling is in preview, SQL Server Agent is in preview, etc).  However, over the coming year Microsoft has indicated they will continue to update the tool.  So for now, it will act as an addition tool in your toolbox, rather than a complete replacement of SSMS.

You can download and view the documentation here:

Alright, let’s get into it!

1. ‘Peek’ at Definition:


Often when writing queries,  you need to see underlying table definitions in order to check out datatypes, see what columns exist, etc.  Normally this involves scrolling through the list of available tables to find the definition, which can be cumbersome in databases with thousands of tables.  However, in Data Studio, you can simply right click and select ‘View Definition’, and a modal pops up showing the table definition.  This is a nifty little feature that I will definitely use!

2. Quick translation of queries to charts

The limits with this feature are pretty extensive. There are many instances where I need to quickly get a some information to a business user, and have to first query the data, and then pop the data into another tool to create a graph.   Now you can do this pretty easily, all at once!  It is also useful when analyzing a dataset for the first time.  You can quickly see how many nulls do certain fields have, check out record distribution across certain categorical columns, etc.

3. Built in Source Control

I have HUNDREDS of queries that I use for day to day analysis and tasks, but are not technically part of a code base.  These are all saved on my desktop.  I do regular backups, but these queries are not saved in source control in any way, and  I would be pretty devastated if I lost all of them.  With built in Source Control, I can easily create a folder where I save all of these queries, and quickly check them in right through the UI. As a result, I will never lose these scripts!


4. Quick exporting to Json / Excel / Csv

Many times a business user will want an extract based on a query – with one click I can now easily extract this and send it along. Same goes with extracting to JSON, where you can quickly send JSON format data to any users / processes that require JSON formats.

5. Custom Insights and Dashboards

I think this feature has the most potential out of any of the features in Data Studio.  One use case that jumps out is a performance dashboard – where you can see number of active queries, any deadlocking issues, or any sort of custom query you have, and have those all on one customized Insight Dashboard.  Many of these features are available in the Azure portal, but Data Studio allows for full customization, which is paramount for custom systems.

Another case could be data quality checks – you can write some simple queries that do quality checks on your data, and display any bad records / results in a single place.  This is a quick and easy way to set up monitoring with little runway time.



One thing I hope to see in the future on this platform is SQL queries in a ‘notebook’ form, similar to how python works in Jupityr notebooks.  I often have to query cross servers, and end up with many windows open.  If I could have one Notebook with different tiles that point to different servers, that would be an awesome feature to organizing queries and presenting business cases to users.

Web Scraping Ironman Triathlon Results to a CSV using BeatifulSoup and Pandas

Based on the name of my blog (DataStrides), and the contents of this post, you might be able to guess that I am an avid endurance athlete in my spare time.  So I hope to have many crossover posts that bring these two worlds together, because if there is one thing I have learned being a part of the endurance community, it is that they LOVE data.

Note: This post assumes a basic understanding of Python, HTML, and CSS.

So when would you want to use data scraping? Simply put, in any case where you want a data set that does not already exist in a clean and downloadable form, or through an API.

Let’s get started.  The first thing you will want to do is import your packages.  For this script we will use html from lxml, requests, bs4 (BeautifulSoup), and Pandas.

from lxml import html
import requests
from bs4 import BeautifulSoup
import pandas as pd
#Scraper goes to starting URL, gathers all of the URLs for the different years IM results.
#Then, iterates through the pagination per race, and stores in a CSV / Pandas DF.

We will use html and requests to interact with the URLs / HTML code we are scraping from, BeautifulSoup is our scraping tool, and Pandas is used for data manipulation.

Next, we will pull in the starting link and transform the HTML from the site into a format that BeautifulSoup understands:

#Get starting URL in place.
url = ''
response = requests.get(url)
html = response.content

Commonly, you will not have all the data you want for your dataset on a single page. You will need to loop through pages. We can do this pretty easily with arrays, for loops, and studying how the URLs are built into the pagination of the website.

Lets take a look at our starting page. You can see on the right hand side (highlighted) all links to the different pages of race results we want. If we inspect the element (F12, or right click > ‘Inspect Element’), we can see that the links are stored in an Unordered List (UL) with an ID of “raceResults”. BeautifulSoup essentially works by using HTML and CSS elements to know where to look on the web page, and then grab whatever you want from the structure.


#Get all race links.
raceLinks = []
for ul in soup.find_all('ul', {'id': 'raceResults'}):
    for link in ul.find_all('a', href=True):

So essentially we go into the unordered list with an ID of raceResults, find all of the a tags (that store the links), and append the href value into our array.

There are some blank links we grab, and 2002 was giving us some issues, so lets remove them for now. We can always go get it manually later:

#get rid of blank links
raceLinks = [x for x in raceLinks if x]
#get rid of 2002 link, as results in bad format and break code.
raceLinksFin = raceLinks[0:14]  

Alright, so our URL list is clean. To add another layer of fun, each result set has N number of pages we will have to paginate through:


How will we get all of these links? You guessed it, more loops! So our outer loop will be to loop through the initial links we gathered, and our inner loop will be to loop through all of the pages. Note that all the code beyond this will be contained in this for loop.

#Loop through races to gather data
mDF = pd.DataFrame()
for race in raceLinksFin:

Then lets grab the date for the page we are looking for so we can tag the results in our pandas DataFrame later on.  You can do this by examining the URL and seeing where the date is in the string.

 #get part of URL that corresponds to date to identify which year results belong to
    date = race[92:]

Next we will use our packages to grab the number of pages or results for that specific race.

 #Get number of pages
    response = requests.get(race)
    html = response.content
    soup = BeautifulSoup(html, 'lxml')
    numberOfPages = []
    for div in soup.find_all('div', {'class': 'pagination'}):
        for span in div.find_all('span'):

Again, we have to clean up our result set to get rid of some unwanted data. You will always want to be printing out what you are scraping during the debugging of the script so you can see what is going on.

 #clean non-numerics from list of gathered data
    cleaned = [ x for x in numberOfPages if x.isdigit() ]
    #convert to int
    ints = [ int(x) for x in cleaned ]
    #get max page numbers
    maxPages = max(ints)

So now we have the number of pages associated with that given year of results. How can we now loop through these to scrape the data? Lets examine the links for the first few paginations:

We can see that the first URL is different, but for 2,3, and 4, the only thing that is changing is what I made bold above. So all we need to do is modify our base URL with the new page up to the max number of pages, and we have our URLS!

As we saw, the first URL is different, so we need to handle it differently.  Feel free to look at the last screenshot which shows the HTML we are scraping.

#Starting URL, as route is different for page 1
    for div in soup.find_all('div', {'class': 'pagination'}):
        for link in div.find_all('a', href=True):
            firstLink = link['href']
            #build link route to loop through paginated pages
            part1 = firstLink[0:91]
            part2 = firstLink[92:]

Alright so now we have the ‘base’ link that we can loop through by doing some string manipulations to the URL and cutting out the part that references the page number. Now we get to use a super useful Pandas function called read_html, which pulls HTML tables into a list if Dataframe objects. Simply put, it makes it very simple to pull tabular data from the internet! As with BeautifulSoup, we can use the HTML / CSS attributes to specify that we want the table with the id of ‘eventResults’.

#Get data from page 1
    df = pd.DataFrame()
    df_intial = pd.read_html(race, attrs = {'id': 'eventResults'})


Then we can append the data to the data frame we initialize:

#append data to dataframe, adding date to identify race year
    df = df.append(df_intial)
    df['Date'] = date
    mDF = mDF.append(df)

Finally, we loop through all the remaining racelinks we gathered for this specific year, append them to the dataset, and once all data is gathered, we go to our outer loop and move on to the next year.

#Loop through remaining pages
    df_2 = pd.DataFrame()
    i = 2
    while i < maxPages: #116
        securl = part1 + str(i) + part2  
        df_temp = pd.read_html(securl, attrs = {'id': 'eventResults'})
        df_2 = df_2.append(df_temp)
        i = i + 1
    df_2['Date'] = date
    mDF = mDF.append(df_2)

Once these loops finish, all of our data is in the mDF DataFrame. From here, we can start doing some analysis on the DataFrame, or we can use one quick line of code to write our DataFrame to a CSV.

#write to a csv, or output to     

So that’s that, you have your data! If you are trying to scrape some less dynamic web pages, the Pandas read_html can be a super quick way to scrape some data down.

As always, feel free to reach out with any comments or questions!

Here is the Github link to the code:

Azure Data Warehouse Workaround – VALUES constructor to grab MIN / MAX value across several columns.

After having worked with Azure Data Warehouse for over 6 months, I have come to learn a lot of tricks about how to make things more performant, how to work around limitations, etc. Thus, I will try to document these workarounds here as frequently as I can. Though I will note, with the recent announcement of SQL Server 2019, many of these limitations will no longer be an issue as Azure SQL DW will support all of the standard SQL Server features.

This is a good read:

Please note that due to the rapid Azure release cycle, certain things I mention may get fixed sometime in the near future.  I will always link the Microsoft Docs to whatever problem I am working around, and you can start by clicking there to see if Azure DW is now supported for whatever topic I am discussing.

For today’s blog, the VALUES constructor.  As you can see, this is currently NOT supported by Azure Data Warehouse:

values - 1

The most common use case for this constructor is inserting some records into a table.  Unfortunately, there is not really a great workaround for this, other than just having a whole bunch of insert statements, rather than just comma separating the values:

Values - 2

However, this is not the use case I am going to go into – we can save that for another blog post.  Rather, I want to discuss point C on the Docs page, which is something I find myself frequently using in SQL Server:

Values - 3

Here is an example for getting the minimum date from several different columns using this logic:

,(SELECT MIN(f) FROM (VALUES (DateTime1), 
                             (DateTime4) AS Datetimes(f)) AS  MinField

This logic is neat, easy to read, and the most efficient way to grab the MIN value spread across multiple columns.  I constantly use this when there is some logic where I need to grab the MIN amongst n column contenders.  However, this logic is not supported (currently) in Azure Data Warehouse. Instead, simply use the following in your select statement.

    ,(SELECT MIN(DateTime) FROM (
        SELECT DateTime1 as DateTime
        SELECT DateTime2 as DateTime
        SELECT DateTime3 as DateTime 
        SELECT DateTime4 as DateTime) subquery
     ) AS MinDateTime

You can use as many Unions as you want to add additional columns, and the logic is just a hair more expensive than the VALUES constructor you are used to on SQL Server.

Note that there are other methods of achieving this same result, but I have found this to be the most performant and  the easiest to read.

Even though SQL Server 2019 will be resolving this limitation, I hope you still find these few SQL snippets useful!

Feel free to leave a comment or question!

First Post – Identifying Duplicates and Fixing Incremental Merge Errors

Figured I would start small with my first post, and share a simple TSQL query that I often find myself using to identify duplicates. If you have ever written an incremental merge script, you have likely gotten an error like the following:


What this essentially means is that either in your target dataset or in your source dataset / query, you have a duplicate record on the key field you are merging on.  Thus, the code is trying to update the same record multiple times, which causes an error.  Take this simple table:


As you can see, there are duplicate OrderIDs in the above table. The code snippet below is one very quick way to identify duplicates, and will return any OrderIDs that are duplicated. You can run it on either the source or the target, just make sure whatever you use in the group by is the key you are merging on, or the field that you are trying to identify as duplicated.

FROM #Orders


You now know that OrderID 1 is duplicated in the table. If you want to find duplicates on more than one column, just add the additional columns to the select and Group By statements.

FROM #Orders


Now we know there are multiple records with OrderID of 1 and ItemID of 123.  Then, you can take one of the returned OrderIDs, and try to analyze why this row was duplicated. Run a simple select (below) on the location of the duplicates using one of the keys you identified above to check out the data.  This will return the duplicated data.

FROM #Orders
where OrderID = 1 --This is an identified duplicate. 


You can tell if the rows are exact duplicates by adding DISTINCT to the above query.  If only one row is returned, you know the records are exactly the same, not just two rows with the same key.

FROM #Orders
WHERE OrderID = 1


Since one row disappeared out of the 3, it means two of the rows were exact duplicates. Your next step is to figure out why you are getting them in your result set, and whether it is valid to have duplicates on this key.  If you are doing a simple select from a table, it means your raw table has duplicates.  This becomes a question of how the table was populated, and whether or not duplicates should be allowed in this table. The above example could either be that one person bought two helmets in a single order, or there could be some sort of bug in whatever code populates the table.  The resolution will all depend on the granularity, or ‘level’ of the data you are trying to get, which I will discuss more later.

If the rows are not exact duplicates, then it means that some field is changing over the result set.  Again, the first step is to try to figure out if these duplicates are valid.  If your ETL source is a complex query, it could be an N:1 join which is causing duplication. In order to find the culprit, your goal should be to identify which column is changing through the rows of your SELECT * query.  Once you find the field, it is likely the join to get that field, or just the nature of the raw table which is causing the duplication. Above, we can identify the ‘ItemID’ as the culprit.  It is likely that you have an N:1 relationship, where many items can be in one order. However, given the above data, we also see that ItemID and OrderID can also be duplicated, so simply adding the ItemID to the MERGE key wouldn’t solve the problem.  You would have to find some other field.

Long story short – if you are doing incremental ETL, Always make sure your MERGE Key is unique!  While you are doing your analysis, you should not just add fields to the merge key to make it unique.  Everything should be done purposefully, and you should always be mindful of what ‘Level’ or granularity your data is at. Lets take a real world healthcare scenario:

Say you are sourcing your data from an EMR system (Electronic Medical Record).  You might find a many to one relationship between procedures undertaken in a surgery and the surgery itself.  That is, there can be multiple procedures done in a single surgery.  When designing your ETL and your data model, you need to decide what ‘level’ you want the surgeries to exist at. Should your target table be at the ‘Surgery’ level, or the ‘Procedure’ level? Take this data set:


  • Do you want to roll them up into one record, and expand your columns to have additional fields for each procedure? This would be the surgery level, but likely won’t work unless you can only have a set X number of procedures associated to a surgery.  This is generally not a good idea.



  • Do you want to just insert the ‘primary’ procedure? Is there some bit flag on the source table so you can filter on just these procedures? This is again the surgery level.


  • Do you want to add the ProcedureID to the merge key / primary key so you can keep the records at the procedure level and avoid issues on the merge key? This would keep the data at the procedure level. Your code might look something like this:
MERGE [datawarehouse].[Surgeries] AS TARGET
	SELECT [SurgeryID]
	FROM [emr].[SurgeryActuals]
	) AS Source
	ON Target.[SurgeryID] = Source.[SurgeryID]
		AND Target.[ProcedureID] = Source.[ProcedureID]
		SET Target.[SurgeryID] = Source.[SurgeryID]
			,Target.[ProcedureID] = Source.[ProcedureID]
			,Target.[SurgeryDate] = Source.[SurgeryDate]
			,Target.[IsPrimary] = Source.[IsPrimary]

As always, it will likely be up to the data consumers for what information they want displayed.  But when designing a Data Warehouse or Data Mart, I always err on the side of caution and include more data rather than less.  It can always be filtered down later.

Feel free to comment with any other tips / tricks / experience you have had with the above!