Forum Discussion

borntobemild's avatar
borntobemild
Copper Contributor
Apr 03, 2025
Solved

Why does Remove Duplicates Excel online Automated Script work when created but not when re-run?

BACKGROUND:

I have an Excel Online workbook that is populated by employees completing a Microsfot Online Form. Employees will submit multiple forms with updates, and each new submission creates a new row in the workbook in the "all responses" worksheet including their Full Name, the date the Form was submitted and the submission ID number. I want to create a second worksheet called "current responses" that only shows the most recent Form response (row) from each unique employee. I want to create an Automated Script to do this so my colleagues can quickly and repeatedly filter for current data without having to manually process the data (which will get messy with many users of varying Excel skill levels). I have written the Script that works when I create it, but when I try to test and re-run it, it spits out the wrong data. Why is this and how do I fix it?

 

DATA PROCESSING:

  1. clear data in range A3-S1003 on current sheet ("current responses")
  2. refresh data
  3. copy data in range A1-S1000 on "all responses" sheet
  4. paste Values into current sheet A3
  5. paste Formatting into current sheet A3
  6. delete columns K-O on current sheet
  7. turn on column filtering
  8. custom sort rows by Form response date with most recent at the top (descending sort on column H) (standard Sort function is not recorded by Automate)
  9. Remove Duplicates on column "Full Name"
  10. custom sort rows by Form "Last Name" with most recent at the top (ascending sort on column B) (standard Sort function is not recorded by Automate)

 

OUTCOME:

Intended: gives a formatted table of 48 rows, one for each employee's most recent Form, ordered alphabetically by Last Name

Actual: works when created, but when tested and re-run it spits out 6 rows of data. These are neither the duplicated nor the unique rows, so I'm not sure why those 6 are being returned.

 

NOTES:

- the paste function is done separately for Values and Format as the original data contains some formulae that I don't want copied over.

- I can't use table.buffer in PowerQuery as PQ is not available with Excel Online

 

CODE:

function main(workbook: ExcelScript.Workbook) {
  let selectedSheet = workbook.getActiveWorksheet();

  // Clear ExcelScript.ClearApplyTo.contents from range A3:S1003 on selectedSheet
  selectedSheet.getRange("A3:S1003").clear(ExcelScript.ClearApplyTo.contents);

  // Refresh all data connections
  workbook.refreshAllDataConnections();
  let all_responses = workbook.getWorksheet("all responses");

  // Paste to range A3 on selectedSheet from range A1:S1000 on all_responses
  selectedSheet.getRange("A3").copyFrom(all_responses.getRange("A1:S1000"), ExcelScript.RangeCopyType.values, false, false);

  // Paste to range A3 on selectedSheet from range A1:S1000 on all_responses
  selectedSheet.getRange("A3").copyFrom(all_responses.getRange("A1:S1000"), ExcelScript.RangeCopyType.formats, false, false);

  // Delete range K:O on selectedSheet
  selectedSheet.getRange("K:O").delete(ExcelScript.DeleteShiftDirection.left);

  // Toggle auto filter on selectedSheet
  selectedSheet.getAutoFilter().apply(selectedSheet.getRange("3:3"));

  // Custom sort on range range A4:S1002 on selectedSheet
  selectedSheet.getRange("A4:S1002").getSort().apply([{key: 7, ascending: false}], false, true, ExcelScript.SortOrientation.columns);

  // Remove duplicates from range A3:N1002 on selectedSheet
  selectedSheet.getRange("A3:N1002").removeDuplicates([2], true);

  // Custom sort on range range A4:S51 on selectedSheet
  selectedSheet.getRange("A4:S51").getSort().apply([{key: 1, ascending: true}], false, true, ExcelScript.SortOrientation.columns);
}
  • Without sample it's hard to say is sort orientation using correctly or not. Plus if you have responses in structured table it's easier and more reliable to manipulate with table rather than range.

    IMHO, main point you shall to wait while data refresh will be done before continue with the next step. Below for that is sleep() function added. Set it on few seconds which will be enough to refresh with some gap. It depend on concrete data and query.

    function main(workbook: ExcelScript.Workbook) {
        
        const sheet = workbook.getActiveWorksheet()
        const range =sheet.getRange("A3:S1003")
        const all_responses = workbook.getWorksheet("all responses").getRange("A1:S1000")
    
        range.clear(ExcelScript.ClearApplyTo.contents);
    
        // Refresh all data connections
        //workbook.refreshAllDataConnections();
        sleep(2) // Sleeps for 2 seconds
    
        range.getCell(0,0).copyFrom(
            all_responses,
            ExcelScript.RangeCopyType.all, false, false
        )
    
        range.getColumn(10).getResizedRange(0, 4)
            .delete(ExcelScript.DeleteShiftDirection.left)
    
        const rangeUsed = range.getUsedRange()
        sheet.getAutoFilter().apply( rangeUsed.getRow(0) )
    
        const rangeSort = rangeUsed
            .getRow(0)
            .getRowsBelow(rangeUsed.getRowCount() - 1)
    
        rangeSort.getSort()
            .apply(
                [
                    { key: 7, ascending: false }
                ],
                false, false, ExcelScript.SortOrientation.rows  //.columns
            )
    
        rangeUsed.removeDuplicates([2], true)
    
        rangeUsed.getSort()
            .apply(
                [{ key: 1, ascending: true }],
                false, true, ExcelScript.SortOrientation.rows)
    }
    
    // Delay in seconds
    function sleep(seconds: number) {
        const waitUntil = new Date().getTime() + seconds * 1000
        while (new Date().getTime() < waitUntil) { }
    }

     

  • Without sample it's hard to say is sort orientation using correctly or not. Plus if you have responses in structured table it's easier and more reliable to manipulate with table rather than range.

    IMHO, main point you shall to wait while data refresh will be done before continue with the next step. Below for that is sleep() function added. Set it on few seconds which will be enough to refresh with some gap. It depend on concrete data and query.

    function main(workbook: ExcelScript.Workbook) {
        
        const sheet = workbook.getActiveWorksheet()
        const range =sheet.getRange("A3:S1003")
        const all_responses = workbook.getWorksheet("all responses").getRange("A1:S1000")
    
        range.clear(ExcelScript.ClearApplyTo.contents);
    
        // Refresh all data connections
        //workbook.refreshAllDataConnections();
        sleep(2) // Sleeps for 2 seconds
    
        range.getCell(0,0).copyFrom(
            all_responses,
            ExcelScript.RangeCopyType.all, false, false
        )
    
        range.getColumn(10).getResizedRange(0, 4)
            .delete(ExcelScript.DeleteShiftDirection.left)
    
        const rangeUsed = range.getUsedRange()
        sheet.getAutoFilter().apply( rangeUsed.getRow(0) )
    
        const rangeSort = rangeUsed
            .getRow(0)
            .getRowsBelow(rangeUsed.getRowCount() - 1)
    
        rangeSort.getSort()
            .apply(
                [
                    { key: 7, ascending: false }
                ],
                false, false, ExcelScript.SortOrientation.rows  //.columns
            )
    
        rangeUsed.removeDuplicates([2], true)
    
        rangeUsed.getSort()
            .apply(
                [{ key: 1, ascending: true }],
                false, true, ExcelScript.SortOrientation.rows)
    }
    
    // Delay in seconds
    function sleep(seconds: number) {
        const waitUntil = new Date().getTime() + seconds * 1000
        while (new Date().getTime() < waitUntil) { }
    }

     

Resources