MicrosoftMicrosoftHi, I’m finding this a bit confusing and would appreciate some clarification.
In the “official” Microsoft repository (Microsoft), there appears to be a Ruby version of the plugin (v2.1.0):
https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/microsoft-sentinel-log-analytics-logstash-output-plugin
However, the README there points to the RubyGems package: https://rubygems.org/gems/microsoft-sentinel-log-analytics-logstash-output-plugin.
At the same time, the documentation states:
To ensure a seamless transition, the new implementation is still packaged and distributed as a standard Logstash Ruby gem. This means the installation and usage experience remains unchanged for customers, while benefiting from a more secure and maintainable foundation.
In practice, this hasn’t been seamless. For example:
- The DCR configuration has changed from dcr_immutable_id to dcr_id, which breaks existing pipeline configurations.
- In testing the new Java-based implementation, there are data compatibility issues (e.g., arrays being sent to Sentinel as “ConvertedLists”).
- There are also noticeable performance differences: previously we could process ~700k events per minute, whereas now it requires around 8GB of JVM heap to approach similar throughput.
Given these differences, could you clarify:
- Which plugin is the one that will be actively maintained going forward?
- Will the Ruby version continue to be supported?
MicrosoftHi cacalacho,
Thank you for your detailed feedback, we really appreciate you taking the time to test the new plugin and share your findings.
To answer your questions directly:
Which plugin will be actively maintained? The new Java-based implementation (v2.x of the microsoft-sentinel-log-analytics-logstash-output-plugin gem) is the version that will be actively maintained going forward.
Will the Ruby version continue to be supported? No. The move to a Java-based implementation was driven by security and compliance requirements. All future development and support will be focused on the Java-based plugin.
Regarding the issues you raised:
- GitHub README linking to older version: Good catch, we have already submitted a fix to update the README so it points directly to the latest v2.1.0 plugin.This is currently under review and will be published shortly.
- DCR configuration field rename (dcr_immutable_id → dcr_id): This change was part of the v2 major version update, along with the simplification of other field names. We acknowledge this wasn't as seamless as documented and we are exploring the feasibility of accepting both configuration keys to ease migration.
- Data compatibility (ConvertedLists) and performance: We take these concerns seriously. Our engineering team is actively investigating the performance differences
you've described, and we are treating this as a high-priority item.
To help us investigate further and address issues specific to your environment, we encourage you to open a case on the Issues · Azure/Azure-Sentinel page with additional details (e.g., your prior resource/memory configuration with the Ruby version). This will allow us to work with you directly and ensure the transition is as seamless as possible. Kindly keep me updated whenever you do so.
We'll continue to share updates as our investigation progresses. Thank you again for helping us improve this plugin.
