How to identify Shadow AI traffic: the signals we've found actually useful
How to identify Shadow AI traffic: the signals we've found actually useful Shadow AI detection sounds simple until you actually try to build it. At first, the obvious approach is: Maintain a list of ChatGPT, Claude, Gemini, Perplexity, etc. -> detect traffic -> classify it as AI. But that breaks down surprisingly quickly. Take GitHub as an example. An employee visiting GitHub is not necessarily using Shadow AI. The same employee using GitHub Copilot potentially is. The same problem exists with Gmail/Gemini, Microsoft 365/Copilot, Slack AI, Notion AI, IDE extensions, browser extensions, and hundreds of SaaS products embedding AI into otherwise legitimate applications. So the real problem is not: "Is this domain an AI domain?" It is: "Does this particular interaction involve a user sending data to an AI system?" We have been working on this problem while building Shadow AI detection at Walled AI, and these are some of the signals I have found most useful. 1. Known AI domains and endpoints This is the easiest layer. Maintain intelligence for known: AI domains inference endpoints API paths model providers AI-specific subdomains SaaS AI endpoints For something like `api.openai.com`, confidence is obviously high. But domain-level detection becomes much weaker when AI functionality is embedded inside a larger application. So domain intelligence should be the first signal, not the final classifier. 2. Request characteristics The request itself can provide surprisingly useful signals. For example: POST requests relatively large text payloads JSON structures containing message- or prompt-like fields conversation history model identifiers generation parameters document or image uploads associated with an inference request None of these individually means "AI." A large POST request could simply be someone submitting a form. But several of these appearing together can increase confidence considerably. 3. Response characteristics The response can provide another independent signal. LLM applications often exhibit patterns such as: streamed responses Server-Sent Events token- or chunk-style incremental output long generated-text responses a request followed by sustained incremental response traffic Again, streaming does not automatically mean AI. Plenty of legitimate applications stream data. But: prompt-like request + AI-looking endpoint + streamed generated response is substantially stronger than any one of those signals alone. 4. Browser context matters This becomes particularly useful for embedded AI. The browser often knows more than the network does. It can potentially tell you: which page the employee is on what UI element initiated the interaction whether a prompt box exists whether text or files are being submitted whether the employee switched into an AI feature whether the user is on a personal or enterprise AI account This helps distinguish: GitHub -> normal development activity from: GitHub -> Copilot interaction without simply classifying all GitHub traffic as AI. This is one reason I do not think Shadow AI discovery can ultimately be solved through network intelligence alone. We have documented some of the multi-layer detection architecture we are building here: https://walled.ai/platform/shadow-ai-detection 5. Identity is another signal Finding an AI service is not necessarily enough. Consider: ChatGPT Enterprise -> approved versus Personal ChatGPT account -> unapproved The destination may effectively be the same while the governance implications are very different. Ideally, AI detection should therefore be correlated with: user device department account or tenant application approved or unapproved status At that point, you are moving beyond traffic classification into actual Shadow AI discovery. 6. Use confidence scoring instead of binary rules This is probably the biggest architectural lesson. Instead of: IF domain == X -> AI I would rather think about: Domain signal + endpoint signal + request signal + response signal + browser signal + identity signal -> AI confidence score High-confidence traffic can be classified deterministically. Low-confidence traffic can remain unclassified. The interesting part is the middle. 7. Use an LLM classifier only for ambiguous traffic One tempting solution is to send everything to an AI classifier. Technically, that is possible. Economically and operationally, it is probably not what you want. If you are inspecting endpoint traffic at enterprise scale, pushing every ambiguous HTTP interaction through an LLM adds: cost latency compute requirements another potential privacy consideration another dependency in the enforcement path We have therefore been experimenting with a different philosophy: Rules and patterns first -> confidence scoring -> LLM judge as fallback As the deterministic detection library improves, fewer requests should require expensive classification. In other words, the system should ideally become **cheaper and faster as its detection intelligence improves**. 8. Detection is only half the problem Once you have classified something as AI traffic, you can start answering more useful governance questions: Which AI applications are employees using? Which are approved versus unapproved? Which departments are adopting Shadow AI? Is sensitive information being submitted? Which models and services are receiving organizational data? What was allowed, redacted, or blocked? This is where we are feeding discovery and enforcement telemetry into the Walled AI Governance Dashboard, so security teams can get a consolidated view of AI usage rather than a large collection of individual network events: https://walled.ai/platform/governance-dashboard <a href="www.walled.ai">Walled AI</a> A practical architecture Known intelligence -> Network signals -> Request + response patterns -> Browser context -> Identity -> Confidence scoring -> LLM fallback rather than: List of AI domains -> Block The hardest part is not identifying ChatGPT. It is identifying AI functionality embedded inside otherwise legitimate applications without creating an unacceptable number of false positives. Curious how others working on Shadow AI are approaching this. What signals have you found reliable for distinguishing actual AI interactions from normal application traffic?Welcome to the Machine Learning and AI .NET space!
Hello and welcome to the Machine Learning & AI space! Here are some resources you might find useful: ML.NET Documentation: https://docs.microsoft.com/dotnet/machine-learning Samples: dotnet/machinelearning-samples: Samples for ML.NET, an open source and cross-platform machine learning framework for .NET. (github.com) Repository: dotnet/machinelearning: ML.NET is an open source and cross-platform machine learning framework for .NET. (github.com) Model Builder Repository: dotnet/machinelearning-modelbuilder: Simple UI tool to build custom machine learning models. (github.com) Machine Learning Community Standup (filter for Machine Learning in dropdown): .NET Community Standups | .NET Live TV (microsoft.com) .NET for Apache Spark Documentation: http://docs.microsoft.com/dotnet/spark Repository: dotnet/spark: .NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers. (github.com) Tools .NET Interactive Notebooks: dotnet/interactive: .NET Interactive takes the power of .NET and embeds it into your interactive experiences. Share code, explore data, write, and learn across your apps in ways you couldn't before. (github.com) .NET Interactive Notebooks VS Code extension: .NET Interactive Notebooks - Visual Studio Marketplace Visual Studio Notebook Editor extension: Notebook Editor - Visual Studio Marketplace Numerical & Statistical Libraries Math.NET Numerics: Math.NET Numerics (mathdotnet.com) FSharp.Stats: FSharp.Stats (fslab.org) Plotting / Graphic Libraries Plotly.NET: Plotly.NET Deep Learning Libraries TensorFlow.NET: SciSharp/TensorFlow.NET: .NET Standard bindings for Google's TensorFlow for developing, training and deploying Machine Learning models in C# and F#. (github.com) TensorFlow.Keras: NuGet Gallery | TensorFlow.Keras 0.6.4 TorchSharp: dotnet/TorchSharp: .NET bindings for the Pytorch engine (github.com) DiffSharp: DiffSharp: Differentiable Tensor Programming Made Simple OnnxRuntime: microsoft/onnxruntime: ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator (github.com)How GitHub Copilot Helps with Test-Driven Development (TDD)
Test-Driven Development (TDD) has been a cornerstone of modern software engineering for decades. By writing tests before implementing functionality, developers ensure better design, maintainability, and fewer bugs. But while TDD is powerful, it can sometimes feel slow or cumbersome, especially when setting up repetitive test structures or boilerplate code. This is where GitHub Copilot, the AI-powered coding assistant, becomes a valuable partner. It doesn’t replace the discipline of TDD, but it can accelerate the process and help developers stay in the flow. https://dellenny.com/how-github-copilot-helps-with-test-driven-development-tdd/Consume Azure Custom Vision ONNX Models with ML.NET
Azure Custom Vision is an image recognition service that lets you build, deploy, and improve your own computer vision models. Custom Vision allows you to specify your own labels and train custom machine models using your data. Custom Vision supports training models for the following tasks: Image classification Object Detection Once you've trained a model, Custom Vision provides you with a variety of formats to export and deploy your model to. One of those formats is the Open Neural Network Exchange (ONNX). ONNX is an open-source format to represent AI models. ONNX models can be used to make predictions with the help of the ONNX Runtime (ORT). The ORT is a runtime for ONNX models which provides an interface for accelerating the consumption / inferencing of machine learning models, integrating with hardware-specific libraries, and sharing models across programming languages and frameworks like PyTorch, Tensorflow / Keras, scikit-learn, Windows ML, ML.NET, and others. ML.NET is an open-source, cross-platform machine learning framework for .NET developers. ML.NET provides a set of APIs that build on the ONNX Runtime. That means you can take models you've exported from Custom Vision and use them inside your .NET applications. Get started with Custom Vision and ML.NET To get started using Custom Vision models with ML.NET Train an image classification or object detection model. Export your model to ONNX Use and reference these sample applications which show how to build an ML.NET pipeline that consumes a Custom Vision ONNX model. Resources What is Custom Vision? ONNX website ONNX Runtime website What is ML.NET?Machine Learning Community Standup - January 18, 2023
Topic: New Year, New Releases Recording Community Links Announcements Announcing ML.NET 2.0 Accelerate ML.NET Training with Intel OneDAL Videos .NET Conf 2022 - Announcing ML.NET 2.0 .NET Conf 2022 - Machine Learning Models with ONNX and .NET .NET Conf 2022 - Deep Learning in .NET Getting Practical with ML.NET Series Pt. 1 - Getting Started with ML.NET Getting Practical with ML.NET Series Pt. 2 - Text Classification with Model Builder Getting Practical with ML.NET Series Pt. 3 - Deploying & Consuming ML Models YOLO v7 .NET sample Jon Wood - What's new in ML.NET 2.0 Livestream - How well can ChatGPT write ML.NET Code? Document Extraction Using ML Blog Posts ODSC: Getting Started with ML.NET New Docs / Updates How to use ML.NET AutoML API Deep Learning Overview Razor Pages Sentiment Analysis Deploy to Azure Functions New Samples / Updates ML.NET 2.0 Samples DirectML ONNX Sample Resources ML.NET Website ML.NET Documentation Machine Learning Notebooks Machine Learning Samples ML.NET Case Studies Feedback We want to hear from you. Do you have topics you'd like to hear more about or have ideas on how we can improve the show? Take a few minutes to fill out our feedback form. Stay Connected Join the Virtual ML.NET Community Discord and the #machine-learning channel on the .NET Development Discord. Find recordings from previous shows on .NET Live TVMachine Learning Community Standup | August 17, 2022
Topic: Introducing SynapseML Recording Community Links .NET Data Hub Machine Learning with ML.NET for Absolute Beginners Plotly C# Bindings NER Announcing SynapseML for .NET Synapse ML Synapse ML GitHub Repo Paper: Large-Scale Intelligent Microservices Mosaic: find artistic connections with deep learning Resources ML.NET Website ML.NET Documentation Machine Learning Notebooks Machine Learning Samples Feedback We want to hear from you. Do you have topics you'd like to hear more about or have ideas on how we can improve the show? Take a few minutes to fill out our feedback form. Stay Connected Join the Virtual ML.NET Community Discord and the #machine-learning channel on the .NET Development Discord. Find recordings from previous shows on .NET Live TVMachine Learning Community Standup | May 11, 2022
Topic: Office Hours YouTube Recording Community Links Normalization - Machine Learning Glossary (ML.NET) Introducing Channel Suggestions (Kliptok) Can ML.NET Save Dr. Who? Probably not, but let's give it a go! ( YouTube) Deep Learning with ML.NET (YouTube) Sound Classification using ML.NET - Practical ML.NET User Group 04/14/2022 (YouTube) Serverless Deep Neural Network(DNN) with Azure Functions and ML.Net (YouTube) Feedback We want to hear from you. Do you have topics you'd like to hear more about or have ideas on how we can improve the show? Take a few minutes to fill out our feedback form. Stay Connected Join the Virtual ML.NET Community Discord and the #machine-learning channel on the .NET Development Discord. Find recordings from previous shows on .NET Live TV.
