SOLVED

Flag specific date records via Power Query

%3CLINGO-SUB%20id%3D%22lingo-sub-1777750%22%20slang%3D%22en-US%22%3EFlag%20specific%20date%20records%20via%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1777750%22%20slang%3D%22en-US%22%3E%3CP%3EI%20have%20eagerly%20adopted%20Power%20Query%20as%20a%20new%20way%20of%20interacting%20with%20data%20as%20it%20is%20so%20much%20easier.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHowever%20I've%20run%20into%20a%20challenge%20and%20was%20hoping%20to%20get%20some%20guidance%20here.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EReferring%20to%20the%20screenshot%20below%20I%20have%20a%20table%20with%203%20columns%3A%20'Job%20No'%2C%20'Onsite%20time'%20and%20'Date_created'.%20This%20data%20essentially%20represents%20trips%20by%20a%20service%20technician%20to%20service%20machines.%20It's%20possible%20that%20there%20are%20multiple%20trips%20and%20the%20screenshot%20shows%20job%20no%20'89452'%20as%20an%20example%20for%20that.%20In%20these%20cases%20I%20need%20to%20select%20the%20%3CSTRONG%3Eearliest%20date%3C%2FSTRONG%3E%3B%20i.e.%204%2F8%2F2020%2011%3A09%20in%20the%20example.%20Given%20it%20is%20the%20same%20job%2C%20'Date_created'%20has%20the%20same%20timestamp.%20so%20I%20need%20help%20in%20identifying%20(e.g.%20set%20a%20flag%20in%20a%20new%20column%20etc.).%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWith%20the%20record%20identified%20I%20then%20need%20to%20look%20up%20the%20value%20in%20column%20'Response%20time'%20which%20is%20simply%20the%20difference%20between%20'Onsite%20Time'%20and%20'Date_created'.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThere's%20probably%20a%20few%20ways%20to%20address%20this%20but%20given%20I'm%20learning%20Power%20Query%20the%20most%20elegant%20way%20is%20probably%20to%20'add'%20a%20'conditional%20column'%20in%20the%20query%20to%20simply%20identify%20the%20record%20with%20a%20flag%20which%20we%20then%20can%20use%20to%20do%20a%20VLOOKUP()%20-%202.4hrs%20in%20the%20case%20of%20job%20no%2089452.%20This%20logic%20would%20have%20to%20identify%20even%20single%20records%20such%20as%20job%20No%2089491%20(49.3hrs).%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAnother%20approach%20may%20be%20to%20'add'%20a%20'conditional%20column'%20to%20directly%20create%20the%20'Response%20Time'%20rather%20than%20having%20to%20do%20the%20VLOOKUP()%20manually.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-left%22%20image-alt%3D%22image.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F226549iCB73BA69E1179D3D%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22image.png%22%20alt%3D%22image.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1777750%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EExcel%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EFormulas%20and%20Functions%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOffice%20365%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1779526%22%20slang%3D%22en-US%22%3ERe%3A%20Flag%20specific%20date%20records%20via%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1779526%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F831234%22%20target%3D%22_blank%22%3E%40tfmeier%3C%2FA%3E%26nbsp%3BYou%20mention%20to%20have%20a%20table%20with%20%3CSTRONG%3Ethree%3C%2FSTRONG%3E%20columns%20in%20your%20screenshot.%20But%2C%20in%20fact%2C%20at%20least%20(parts%20of)%20seven%20columns%20are%20visible.%20Not%20clear%20if%20only%20the%20three%20you%20mention%20are%20loaded%20by%20PQ%20and%20that%20the%20others%20are%20just%20added%20outside%20PQ.%20Would%20be%20helpful%20tif%20you%20could%20upload%20(part%20of)%20your%20workbook%2C%20without%20confidential%20information%2C%20including%20the%20query%20that%20you%20have%20created.%26nbsp%3BShouldn't%20be%20too%20difficult%20to%20filter%20out%20the%20earliest%20occurrence%20of%20a%20service%20call.%20But%2C%20it%20will%20be%20difficult%20without%20real%20data%20to%20play%20with.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1781945%22%20slang%3D%22en-US%22%3ERe%3A%20Flag%20specific%20date%20records%20via%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1781945%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%20for%20getting%20back%2C%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F570951%22%20target%3D%22_blank%22%3E%40mtarler%3C%2FA%3E%2C%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F403176%22%20target%3D%22_blank%22%3E%40Riny_van_Eekelen%3C%2FA%3E.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHere's%20a%20bit%20more%20info%20and%20attached%20sanitised%20files.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%203%20source%20data%20files%3A%3C%2FP%3E%3CUL%3E%3CLI%3E'Job_Activity_report'%3C%2FLI%3E%3CLI%3E'BI%20Report'%3C%2FLI%3E%3CLI%3E'Mobile_Status_Log'%3C%2FLI%3E%3C%2FUL%3E%3CP%3EAnd%20pull%20data%20via%20Power%20Query%20into%20'Breakdown%20Report'%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EOutcome%3C%2FSTRONG%3E%3A%20I%20need%20to%20create%20a%20histogram%20showing%20response%20time%20in%2024%20hr%20blocks%20(bin%20size%20of%2024)%20from%20column%20'Response%20Time'%20in%20sheet%20'Mobile%20Status%20Log'.%20This%20sheet%20contains%20both%20columns%20populated%20by%20Power%20Query%20and%20manual%20columns%20to%20identify%20valid%20records%20and%20for%20(selected%20columns)%20become%20the%20source%20for%20the%20graph.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EColumns%20of%20interest%20are%3C%2FSTRONG%3E%3A%20'Onsite%20Time'%2C%20'Date_created'%2C%20'Response%20time'.%20'Response%20time'%20%3D%20'Onsite%20Time'%20less%20'Date_created'.%20However%20not%20every%20record%20is%20valid.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EWhat's%20a%20valid%20record%3C%2FSTRONG%3E%3F%20It's%20possible%20to%20have%20a%20few%20service%20trips%20per%20job%20and%20the%20system%20creates%20a%20time%20stamp%20every%20time%20a%20technician%20is%20onsite%3B%2089452%20in%20the%20original%20post%20is%20a%20good%20example%20where%20we%20have%20multiple%20onsite%20times%20against%20the%20same%20date_created%20timestamp.%20In%20cases%20like%20this%20I%20need%20to%20select%20the%20earliest%20Onsite%20Time%3B%20i.e.%204%2F8%2F2020%2011%3A09%20in%20the%20example.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F570951%22%20target%3D%22_blank%22%3E%40mtarler%3C%2FA%3E's%20solution%20works%20and%20all%20valid%20records%20get%20tagged%20'0'%20and%20I%20can%20select%20these%20to%20create%20the%20histogram.%20Thanks%20for%20that!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHowever%20given%20I'm%20in%20the%20midst%20of%20learning%20Power%20Query%20I%20was%20wondering%20how%20to%20do%20this%20by%20adding%20a%20conditional%20column%20'Valid%20Record'%20with%20values%20of%20'yes'%20%2F%20'no'%20to%20then%20allow%20me%20to%20only%20pull%20records%20tagged%20'yes'.%20So%20in%20other%20words%20all%20that's%20left%20is%20to%20pull%20selected%20columns%20to%20create%20the%20presentation%20(histogram%20in%20this%20case).%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIs%20that%20possible%3F%20Is%20there%20a%20smarter%20way%20of%20doing%20this%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1782387%22%20slang%3D%22en-US%22%3ERe%3A%20Flag%20specific%20date%20records%20via%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1782387%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F831234%22%20target%3D%22_blank%22%3E%40tfmeier%3C%2FA%3E%26nbsp%3BPlayed%20around%20with%20your%20files%20a%20bit%20and%20revised%20parts%20of%20the%20queries%20to%20arrive%20at%20a%20table%20%22Merge1%22.%20Hopefully%2C%20it%20helps%20you%20exploring%20PQ%20further.%20You%20will%20notice%20that%20merging%20queries%20(the%20correct%20way)%20eliminates%20the%20need%20for%20VLOOKUP%20outside%20the%20PQ%20tables.%3C%2FP%3E%3CP%3EJust%20point%20the%20source%20for%20each%20of%20the%20queries%20to%20the%20files%20on%20your%20own%20system%20and%20you%20should%20be%20able%20to%20see%20the%20end%20result%20and%20follow%20through%20all%20the%20steps.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1783162%22%20slang%3D%22en-US%22%3ERe%3A%20Flag%20specific%20date%20records%20via%20Power%20Query%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1783162%22%20slang%3D%22en-US%22%3E%3CP%3EThank%20you%20for%20that%2C%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F403176%22%20target%3D%22_blank%22%3E%40Riny_van_Eekelen%3C%2FA%3E.%20So%20merged%20queries%20can%20combine%20data%20from%20different%20tables%20via%20a%20common%20key%2C%20Job%20no%20(Job%20id)%20in%20this%20case%3F%20Do%20the%20common%20columns%20always%20have%20the%20be%20in%20the%20first%20%2F%20most%20left%20column%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAlso%20what's%20the%20purpose%20of%20%22expand%22%20in%20power%20query%20in%20layman%20terms%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20again%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
New Contributor

I have eagerly adopted Power Query as a new way of interacting with data as it is so much easier.

 

However I've run into a challenge and was hoping to get some guidance here.

 

Referring to the screenshot below I have a table with 3 columns: 'Job No', 'Onsite time' and 'Date_created'. This data essentially represents trips by a service technician to service machines. It's possible that there are multiple trips and the screenshot shows job no '89452' as an example for that. In these cases I need to select the earliest date; i.e. 4/8/2020 11:09 in the example. Given it is the same job, 'Date_created' has the same timestamp. so I need help in identifying (e.g. set a flag in a new column etc.).

 

With the record identified I then need to look up the value in column 'Response time' which is simply the difference between 'Onsite Time' and 'Date_created'.

 

There's probably a few ways to address this but given I'm learning Power Query the most elegant way is probably to 'add' a 'conditional column' in the query to simply identify the record with a flag which we then can use to do a VLOOKUP() - 2.4hrs in the case of job no 89452. This logic would have to identify even single records such as job No 89491 (49.3hrs).

 

Another approach may be to 'add' a 'conditional column' to directly create the 'Response Time' rather than having to do the VLOOKUP() manually.

 

image.png

6 Replies
Highlighted

@tfmeier You mention to have a table with three columns in your screenshot. But, in fact, at least (parts of) seven columns are visible. Not clear if only the three you mention are loaded by PQ and that the others are just added outside PQ. Would be helpful tif you could upload (part of) your workbook, without confidential information, including the query that you have created. Shouldn't be too difficult to filter out the earliest occurrence of a service call. But, it will be difficult without real data to play with.

 

Highlighted

@tfmeier  I agree with @Riny_van_Eekelen  that a sample sheet and more info on exactly what you want would be helpful.  That said and based on you looking for only the earliest case I will put out a thought.  Apparently you already have a column "Duplicates" that appears to show a count of how many times that job no appears in the list (something like =countif(a:a,a1) but obviously not column A and maybe you are using Table structure references, which would be good.  But let me ask why you want to show the total count?  What if you changed it to =countif($A$1:$A1,A1) which would then count the duplicates (i.e. if there are 8 instances the first would be 1, the second 2, etc...  Then your query can selectively pull only the 1s from that column.  You could also check the above =1 to return Yes or No.  If you want to know how many duplicate you could   =countif($A$1:$A1,A1) & "of" & countif(A:A,A1) and it would display "1 of 8" but then the query would have to search for "1 of" because if you only look for starting with "1" then "10 of 12" would get pulled also.

Highlighted

Thanks for getting back, @mtarler, @Riny_van_Eekelen.

 

Here's a bit more info and attached sanitised files.

 

I have 3 source data files:

  • 'Job_Activity_report'
  • 'BI Report'
  • 'Mobile_Status_Log'

And pull data via Power Query into 'Breakdown Report'

 

Outcome: I need to create a histogram showing response time in 24 hr blocks (bin size of 24) from column 'Response Time' in sheet 'Mobile Status Log'. This sheet contains both columns populated by Power Query and manual columns to identify valid records and for (selected columns) become the source for the graph.

 

Columns of interest are: 'Onsite Time', 'Date_created', 'Response time'. 'Response time' = 'Onsite Time' less 'Date_created'. However not every record is valid.

 

What's a valid record? It's possible to have a few service trips per job and the system creates a time stamp every time a technician is onsite; 89452 in the original post is a good example where we have multiple onsite times against the same date_created timestamp. In cases like this I need to select the earliest Onsite Time; i.e. 4/8/2020 11:09 in the example.

 

@mtarler's solution works and all valid records get tagged '0' and I can select these to create the histogram. Thanks for that!

 

However given I'm in the midst of learning Power Query I was wondering how to do this by adding a conditional column 'Valid Record' with values of 'yes' / 'no' to then allow me to only pull records tagged 'yes'. So in other words all that's left is to pull selected columns to create the presentation (histogram in this case).

 

Is that possible? Is there a smarter way of doing this?

Highlighted
Best Response confirmed by tfmeier (New Contributor)
Solution

@tfmeier Played around with your files a bit and revised parts of the queries to arrive at a table "Merge1". Hopefully, it helps you exploring PQ further. You will notice that merging queries (the correct way) eliminates the need for VLOOKUP outside the PQ tables.

Just point the source for each of the queries to the files on your own system and you should be able to see the end result and follow through all the steps.

Highlighted

Thank you for that, @Riny_van_Eekelen. So merged queries can combine data from different tables via a common key, Job no (Job id) in this case? Do the common columns always have the be in the first / most left column?

 

Also what's the purpose of "expand" in power query in layman terms?

 

Thanks again

Highlighted

@tfmeier You can merge on the basis of any column as long as you expect them to have common values. So, you can merge the third column from one table called "JobNo" with e.g. the 10th column from another called "JobCode", as long as these columns use the same codes. Otherwise, you'll never find a match. You can also merge on the basis of multiple columns.

 

Merging can be done in several ways (outer, inner, full and anti). In simple terms, it adds a new column to the first table that in itself is a table of matching records. These can then be expanded so that matching records are "joined" side by side. Similar to what you would do with VLOOKUP.

 

There are many PQ tutorials on line. This particular one I found very useful myself. 

https://exceloffthegrid.com/power-query-introduction/