Blog Post

Azure Data Factory Blog
2 MIN READ

ADF Monitoring Improvements

JoshuhaOwen's avatar
JoshuhaOwen
Icon for Microsoft rankMicrosoft
Apr 26, 2022

The Azure Data Factory team is happy to announce several new improvements to the monitoring experience. These improvements and new features are based off community feedback and votes, social media feedback, and survey results.

 

1. First up is the ability to export the data you see to a CSV file. This will be whatever data is on the screen so will obey any filters you have in place!

 

 


2. You can now clear all filters with an easy button press, this will make it easier to get back to your default filter state.

 


3. You now have the ability to open the activity run details in a new tab. This will make it easier to look at multiple runs without losing context of where you were. There are two different ways to perform this action.

a) First, you select one or more pipeline runs and click the “open new tabs” button at the top. Please note you may need to allow popup windows if you are unable to open more than 1 tab at a time.

 


b) The second method is for a single pipeline run you can right-click to get a context menu that allow you to open in a new tab, window, or copy a link to the pipeline run.

 


4. Any column changes you make via “Edit columns” will be preserved for that browser profile. So, if you add or remove columns your column list should be remembered!
5. Similarly, if you navigate from the pipeline runs into the activity details of an individual run and navigate back via either the browser back button or via the breadcrumb your previous state should be still there. No more waiting on the monitoring data to refresh when you go back!

 

We hope these bundle of monitoring improvements helps to make your monitoring experience smoother, quicker and more productive. We are always open for feedback on what’s next for monitoring in ADF and you can provide suggestions or vote via our Ideas forum Data factory · Community (azure.com).

Updated Apr 26, 2022
Version 1.0
  • malcolndandaro's avatar
    malcolndandaro
    Copper Contributor

    Those updates helped, but I still find the currently monitoring part of ADF/Synapse really bad. What I'm doing is getting data from all the pipelines from the API and saving it in cosmos, we can do way better analysis this way.

  • valnate's avatar
    valnate
    Copper Contributor

    The most pressing need here is exporting all the csvs with one click instead of going page by page.

     

    Or even better creating  a way of sending the csv to a synapse table, isn't that the benefit of integrated azure services ?

  • ThucNguyen's avatar
    ThucNguyen
    Copper Contributor

    CSV export is greatly helpful. However, can you shed some light on the Duration field in CSV file? It is a numeric value while on web it shows 'hh:mm:ss' format.

  • Jude_Renaudin's avatar
    Jude_Renaudin
    Copper Contributor

    That s a very good feature, now we finished with the back and forth in the run view of the pipeline 

  • Rajiv_Ramanjani's avatar
    Rajiv_Ramanjani
    Copper Contributor

    A few more improvements on the metrics page that would be of BIG HELP (truly!!) are:

     

    a. Have the 'Export to CSV' export the data that is on all the pages of the view ..................not just the one that is visible currently. 

    b. Including some more performance metrics of the pipeline in the main view itself.  This like Data Movement Activities, External Activities, Pipeline Activities and Pipeline Orchestration Activity Runs.  Right now to get this information - we have to go into each pipeline and pull the details.  This is very painful.  Please see the diagram below:

     

    If these details are available as part of the main background view columns - that would be extremely helpful for us to export the details into a csv - for analysis purposes. 

  • Syedhamjath's avatar
    Syedhamjath
    Copper Contributor

    Atlast some useful features. Our Ops engineer spend most of the time in navigating back and forth in the pipeline run view. Opening the pipeline run in new tab is what we usually do 🙂

  • HebertBanes's avatar
    HebertBanes
    Copper Contributor

    No more waiting on the monitoring data to refresh when you go back!

     

    This will be awesome.

  • KevinSames's avatar
    KevinSames
    Copper Contributor

    Thank you for sharing this Update. Happy to see the one click button!

    cheers