Enforcing Data Limits/Requirements in Power Query

%3CLINGO-SUB%20id%3D%22lingo-sub-2616828%22%20slang%3D%22en-US%22%3EEnforcing%20Data%20Limits%2FRequirements%20in%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2616828%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20-%20I'm%20new%20to%20using%20Power%20Pivot%2FPower%20Query%2C%20but%20I%20have%20some%20files%20that%20are%20too%20large%20for%20standard%20Excel%20and%20I%20don't%20feel%20like%20diving%20straight%20into%20Access%2C%20so%20Power%20Query%20it%20is.%20With%20that%20out%20of%20the%20way%20-%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20trying%20to%20create%20a%20query%20that%20I%20can%20use%20repeatedly%20to%20load%2Ftransform%20data%20and%20identify%20'errors'%20that%20need%20to%20be%20fixed%2C%20and%20then%20ultimately%20saving%20the%20final%20data%20as%20a%20.CSV.%20Ultimately%20it%20would%20be%20nice%20to%20have%20a%20set%20of%20headers%20that%20I%20need%20to%20use%20in%20the%20final%20data%2C%20and%20then%20map%20whatever%20source%20file%20I%20have%20to%20those%20column%20headers.%20I'm%20fine%20just%20renaming%20the%20headers%20in%20the%20source%20data%20to%20match%20the%20final%20headers%2C%20but%20what%20I%20really%20would%20like%20is%20an%20ability%20to%20create%20data%20restrictions%20or%20requirements%20for%20the%20columns%20I'm%20using%20and%20then%20use%20Power%20Query%20to%20find%20outliers%20that%20don't%20match.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EE.g.%20let's%20say%20one%20of%20the%20columns%20is%20always%20going%20to%20be%20%22Service_Code_ID%22%20and%20it%20has%20a%20character%20limit%20of%2050%20characters.%20How%20can%20I%20write%20a%20repeatable%20query%20that%20find%20all%20values%20in%20the%20Service_Code_ID%20column%20that%20exceed%2050%20characters%20and%20display%20them%20so%20that%20I%20can%20update%20them%20appropriately%3F%20There%20are%20probably%2012-15%20columns%20total%20I'm%20going%20to%20be%20working%20with%20and%20each%20one%20has%20its%20own%20set%20of%20characteristics%20that%20I%20want%20to%20check%20and%20flag%20anything%20that%20doesn't%20qualify.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMy%20initial%20thought%20was%20to%20write%20some%20kind%20of%20formula%20for%20each%20column%20and%20then%20use%20conditional%20formatting%20to%20highlight%20the%20problem%20values.%20At%20that%20point%20I%20could%20either%20filter%20on%20highlights%20or%20query%20just%20rows%20that%20contain%20a%20highlighted%20value%20maybe%20-%20but%20as%20complicated%20as%20it%20seems%20like%20it%20will%20be%20no%20matter%20what%2C%20using%20conditional%20formatting%20seems%20more%20complicated%20and%20less%20easily%20repeatable.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20ideas%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2616828%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EBI%20%26amp%3B%20Data%20Analysis%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EPower%20BI%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2618241%22%20slang%3D%22en-US%22%3ERe%3A%20Enforcing%20Data%20Limits%2FRequirements%20in%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2618241%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1121602%22%20target%3D%22_blank%22%3E%40MicrosoftMacros2011%3C%2FA%3E%26nbsp%3BIn%20PowerQuery%20(PQ)%20you%20can%20select%20a%20column.%20Then%20on%20the%20%22Add%20column%22%20tab%2C%20choose%20%3CEM%3EExtract%3C%2FEM%3E%2C%20%3CEM%3ELength.%26nbsp%3B%3C%2FEM%3EThis%20will%20create%20a%20new%20column%20with%20the%20lengths%20of%20the%20texts%20in%20the%20original%20columns.%20Now%20you%20can%20easily%20filter%20out%20the%20ones%20that%20exceed%2050.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EYou%20don't%20mention%20anything%20about%20what%20the%20other%20columns%20characteristics%20are%2C%20so%20I%20can't%20say%20how%20to%20check%20those.%20But%20very%20likely%20you%20can%20do%20it%20in%20a%20similar%20way.%20Perhaps%20you%20have%20learn%20to%20write%20some%20M-code%20as%20well.%20%26nbsp%3BOnce%20you%20have%20set%20this%20up%20correctly%2C%20you%20can%20connect%20to%20new%20data%20and%20refresh%20the%20query%20(or%20queries)%20over%20and%20over%20again.%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

Hi - I'm new to using Power Pivot/Power Query, but I have some files that are too large for standard Excel and I don't feel like diving straight into Access, so Power Query it is. With that out of the way -

 

I'm trying to create a query that I can use repeatedly to load/transform data and identify 'errors' that need to be fixed, and then ultimately saving the final data as a .CSV. Ultimately it would be nice to have a set of headers that I need to use in the final data, and then map whatever source file I have to those column headers. I'm fine just renaming the headers in the source data to match the final headers, but what I really would like is an ability to create data restrictions or requirements for the columns I'm using and then use Power Query to find outliers that don't match.

 

E.g. let's say one of the columns is always going to be "Service_Code_ID" and it has a character limit of 50 characters. How can I write a repeatable query that find all values in the Service_Code_ID column that exceed 50 characters and display them so that I can update them appropriately? There are probably 12-15 columns total I'm going to be working with and each one has its own set of characteristics that I want to check and flag anything that doesn't qualify.

 

My initial thought was to write some kind of formula for each column and then use conditional formatting to highlight the problem values. At that point I could either filter on highlights or query just rows that contain a highlighted value maybe - but as complicated as it seems like it will be no matter what, using conditional formatting seems more complicated and less easily repeatable.

 

Any ideas?

2 Replies

@MicrosoftMacros2011 In PowerQuery (PQ) you can select a column. Then on the "Add column" tab, choose Extract, Length. This will create a new column with the lengths of the texts in the original columns. Now you can easily filter out the ones that exceed 50. 

 

You don't mention anything about what the other columns characteristics are, so I can't say how to check those. But very likely you can do it in a similar way. Perhaps you have learn to write some M-code as well.  Once you have set this up correctly, you can connect to new data and refresh the query (or queries) over and over again.

Hmm sounds like looking into some M-Code might be best eventually, but also sounds like Power Query doesn't have something out of the box. An additional column is what I would do in Excel and use the LEN function; but I usually only do that if I'm concerned something exceeds a length and I wouldn't want to do an extra column per actual column. Especially not if I eventually needed 1 column per "characteristic." I don't have all of the requirements in front of me, but it's your standard gamut of restrictions: specific lengths, some alpha/numeric, some only numeric, one column has specific text that each value needs to match and only 4 possible options, etc.