Explainability of Bonsai Prediction

%3CLINGO-SUB%20id%3D%22lingo-sub-2118717%22%20slang%3D%22en-US%22%3EExplainability%20of%20Bonsai%20Prediction%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2118717%22%20slang%3D%22en-US%22%3E%3CP%3EDoes%20Bonsai%20have%20local%20explainability%20%3F%20For%20example%20in%20the%20scenario%20of%20Cartpole%2C%20Bonsai%20continuously%20predicts%20action%2C%20right%20or%20left.%20Local%20explainability%20shows%20the%20contribution%20of%20input%20features%20contribute%20to%20a%20each%20prediction.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2124323%22%20slang%3D%22en-US%22%3ERe%3A%20Explainability%20of%20Bonsai%20Prediction%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2124323%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F209043%22%20target%3D%22_blank%22%3E%40Keita%20Onabuta%3C%2FA%3E%26nbsp%3BThe%20explainability%20benefits%20of%20machine%20teaching%20come%20from%20decomposing%20a%20problem%20into%20concepts%20%E2%80%94%20skills%20or%20strategies%20that%20can%20be%20learned%20and%20applied%20separately.%20Given%20such%20a%20decomposition%2C%20the%20brain%20can%20output%20%22here's%20what%20skill%20or%20strategy%20I'm%20applying%20now%22.%20This%20is%20a%20complement%20to%20local%20explainability%2C%20which%20would%20explain%20the%20actions%20chosen%20by%20a%20particular%20concept.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2119700%22%20slang%3D%22en-US%22%3ERe%3A%20Explainability%20of%20Bonsai%20Prediction%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2119700%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F662081%22%20target%3D%22_blank%22%3E%40VictorShnayder%3C%2FA%3E%26nbsp%3BOkay.%20I%20found%20this%20sentence%20in%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Farchitecture%2Fsolution-ideas%2Farticles%2Fmachine-teaching%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3E%22Machine%20Teaching%22%20of%20Azure%20Architecture%20center%3C%2FA%3E.%26nbsp%3B%20What%20does%20it%20mean%20for%20Bonsai%20%3F%20How%20about%20Global%20Explainability%20%3F%20Thanks.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Screen%20Shot%200003-02-10%20at%209.38.47.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F253755i37B2AC1E23529B54%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Screen%20Shot%200003-02-10%20at%209.38.47.png%22%20alt%3D%22Screen%20Shot%200003-02-10%20at%209.38.47.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2119186%22%20slang%3D%22en-US%22%3ERe%3A%20Explainability%20of%20Bonsai%20Prediction%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2119186%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F209043%22%20target%3D%22_blank%22%3E%40Keita%20Onabuta%3C%2FA%3E%26nbsp%3Bwe%20do%20not%20have%20a%20feature%20for%20local%20explainability%20today.%20Are%20you%20interested%20in%20them%20to%20help%20debug%20during%20training%2C%20or%20when%20using%20a%20trained%20brain%3F%20If%20you%20have%20use%20cases%20in%20mind%2C%20please%20add%20them%20as%20a%20suggestion%20at%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ffeedback.azure.com%2Fforums%2F928846-project-bonsai%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Ffeedback.azure.com%2Fforums%2F928846-project-bonsai%3C%2FA%3E%26nbsp%3Bto%20help%20us%20prioritize.%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Does Bonsai have local explainability ? For example in the scenario of Cartpole, Bonsai continuously predicts action, right or left. Local explainability shows the contribution of input features contribute to a each prediction.

3 Replies

@Keita Onabuta we do not have a feature for local explainability today. Are you interested in them to help debug during training, or when using a trained brain? If you have use cases in mind, please add them as a suggestion at https://feedback.azure.com/forums/928846-project-bonsai to help us prioritize. 

 

@VictorShnayder Okay. I found this sentence in "Machine Teaching" of Azure Architecture center.  What does it mean for Bonsai ? How about Global Explainability ? Thanks.

Screen Shot 0003-02-10 at 9.38.47.png

@Keita Onabuta The explainability benefits of machine teaching come from decomposing a problem into concepts — skills or strategies that can be learned and applied separately. Given such a decomposition, the brain can output "here's what skill or strategy I'm applying now". This is a complement to local explainability, which would explain the actions chosen by a particular concept.