Not able to get logs related to azure data factory mapping data flows from log analytics

%3CLINGO-SUB%20id%3D%22lingo-sub-1541652%22%20slang%3D%22en-US%22%3ENot%20able%20to%20get%20logs%20related%20to%20azure%20data%20factory%20mapping%20data%20flows%20from%20log%20analytics%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1541652%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20are%20working%20on%20implementing%20a%20custom%20logging%20solution.%20Most%20of%20the%20information%20what%20we%20need%20is%20already%20present%20in%20log%20analytics%20from%20data%20factory%20analytics%20solution%20but%20for%20getting%20log%20info%20on%20data%20flows%2C%26nbsp%3B%20there%20is%20a%20challenge.%20When%20querying%2C%20we%20get%20this%20error%20in%20output.%20%22Too%20large%20to%20parse%22.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESince%20data%20flows%20are%20complex%20and%20critical%20piece%20in%20a%20pipeline%2C%20we%20are%20in%20desperate%20need%20to%20get%20data%20like%20rows%20copied%2C%20skipped%2C%20read%20etc%20of%20each%20activities%20with%20in%20data%20flow.%20can%20you%20pls%20help%20how%20to%20get%20those%20info%3F%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Bharani365_0-1595521135831.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F207470i9B8C7D0CBCCD5FA3%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Bharani365_0-1595521135831.png%22%20alt%3D%22Bharani365_0-1595521135831.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F377021%22%20target%3D%22_blank%22%3E%40IGaurav%3C%2FA%3E%26nbsp%3B%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F63403%22%20target%3D%22_blank%22%3E%40scott%3C%2FA%3E%26nbsp%3BNot%20sure%20if%20this%20mention%20refers%20to%20Gaurav%20malhotra%20of%20data%20factory%20team.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1610967%22%20slang%3D%22en-US%22%3ERe%3A%20Not%20able%20to%20get%20logs%20related%20to%20azure%20data%20factory%20mapping%20data%20flows%20from%20log%20analytics%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1610967%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F545767%22%20target%3D%22_blank%22%3E%40Bharani365%3C%2FA%3E%26nbsp%3BA%20fix%20was%20recently%20deployed%20to%20reduce%20the%20impact%20of%20this%20error.%20Can%20you%20try%20again%20and%20let%20us%20know%20if%20problem%20persists%3F%20Furthermore%2C%20if%20your%20dataflow%20is%20too%20complex%20or%20using%20too%20many%20partitions%2C%20you%20can%20simplify%20your%20dataflow%20to%20further%20improve%20this%20situation.%26nbsp%3BIf%20you%20could%20share%20a%20run%20id%2C%20we%20could%20analyze%20it%20and%20have%20a%20better%20understanding%20of%20the%20issue.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Senior Member

We are working on implementing a custom logging solution. Most of the information what we need is already present in log analytics from data factory analytics solution but for getting log info on data flows,  there is a challenge. When querying, we get this error in output. "Too large to parse". 

 

Since data flows are complex and critical piece in a pipeline, we are in desperate need to get data like rows copied, skipped, read etc of each activities with in data flow. can you pls help how to get those info?

Bharani365_0-1595521135831.png

@IGaurav  @scott Not sure if this mention refers to Gaurav malhotra of data factory team.

 

1 Reply

@Bharani365 A fix was recently deployed to reduce the impact of this error. Can you try again and let us know if problem persists? Furthermore, if your dataflow is too complex or using too many partitions, you can simplify your dataflow to further improve this situation. If you could share a run id, we could analyze it and have a better understanding of the issue.