Monitoring Azure Service Fabric with Azure Managed Grafana
1. Why integrate Grafana with Service Fabric? Scope: this walkthrough covers a classic, VMSS-backed Service Fabric cluster running Windows. Service Fabric managed clusters (SFMC) use managed node types instead of VM scale sets, but the same AMA + DCR + Grafana pattern applies once AMA is installed on the managed node type. Linux node types use a different AMA data collection path (syslog instead of Windows Event Log channels) and aren't covered here. Grafana is valuable for Service Fabric customers who want a single operational view across cluster nodes, VM infrastructure, and Service Fabric platform events. Position Grafana as the visualization layer, while Azure Monitor and Log Analytics are the telemetry backbone. Grafana does not “monitor Service Fabric” by itself but it visualizes telemetry collected by Azure Monitor Agent (AMA) and stored in Log Analytics. Customer need Why Grafana helps Unified monitoring One dashboard for VM health, guest-OS counters, and Service Fabric platform events. Operations wallboard Easier to display and share with NOC/SRE teams than SFX that needs certificates and the team can mistakenly take actions from if they have admin permissions. Cross-service observability Teams already using Grafana for AKS/VMs/SQL can add Service Fabric beside them. Reusable dashboards Dashboard JSON can be exported, versioned, and reused across environments. Note: Small clusters with basic operations may be fine with Service Fabric Explorer (SFX) + the Azure portal. Also, Grafana is not a replacement for SFX, SF PowerShell, or the SF REST APIs, it's for observability, not cluster management. 2. Reference architecture The telemetry flow used in this repro: Service Fabric cluster nodes run on a Windows Virtual Machine scaleset (VMSS). Azure Monitor Agent (AMA) on the VMSS collects Windows performance counters and Service Fabric event channels. A Data Collection Rule (DCR) routes that telemetry to a Log Analytics workspace. Azure Managed Grafana queries Azure Monitor / Log Analytics through its managed identity (MI). A single dashboard combines infrastructure, guest-OS, and Service Fabric event views. 3. Prerequisites Install add-ons to your local Azure CLI so it understands commands specific to Grafana and Azure Monitor. Add the Azure Managed Grafana (AMG) extension. This allows you to run az grafana commands to create and configure your Grafana workspace. az extension add --name amg --only-show-errors 2. Add the extension for Data Collection Rules (DCRs) and endpoints. You need this to configure how telemetry data flows from your Service Fabric cluster to Azure Monitor. az extension add --name monitor-control-service --only-show-errors 3. Registering Resource Providers By default, Azure subscriptions do not have every possible service enabled. The az provider register commands tell Azure: "I am going to use these specific services in this subscription, so enable them." az provider register --namespace Microsoft.ServiceFabric --wait az provider register --namespace Microsoft.Insights --wait az provider register --namespace Microsoft.OperationalInsights --wait az provider register --namespace Microsoft.Dashboard --wait For this repro, I created a small Service Fabric cluster with 3 nodes that has a running app: 3 instances Running / provisioning Succeeded 4. Enable telemetry collection (AMA + DCR) Install Azure Monitor Agent on the VM scale set, then create and associate a Data Collection Rule. az vmss extension set -g rg-sf-grafana-repro --vmss-name nt1vm --name AzureMonitorWindowsAgent --publisher Microsoft.Azure.Monitor --enable-auto-upgrade true Important Note: the VMSS MUST have a managed identity (MI), or the AMA won't authenticate and no data flows. Note: both system-assigned and user-assigned managed identities are supported – assign the identity before installing the AMA extension, as done below. See the Azure Monitor Agent requirements. az vmss identity assign -g rg-sf-grafana-repro -n nt1vm The DCR collects CPU/memory/disk/network counters, Windows Application/System warnings and errors, and the Service Fabric Admin + Operational event channels. This next block is where the actual monitoring pipeline gets wired up. It configures AMA to start capturing the specific metrics and logs you care about from your SF nodes. az monitor data-collection rule create -g rg-sf-grafana-repro --location westus2 --name dcr-sfg08120632 --rule-file dcr-servicefabric-windows.json The full dcr-servicefabric-windows.json used in this repro, along with the dashboard JSON, ARM template, and all deployment scripts, is available in this GitHub repo so this is reproducible: https://github.com/AhmedKhaledAbdalla/Azure-Service-Fabric-Grafana-Integration --rule-file dcr-servicefabric-windows.json: This is the most critical part. Azure CLI reads this local JSON file, which contains the actual configuration you mentioned. It defines exactly what to collect (the CPU/memory counters, Windows Event logs, and Service Fabric Admin/Operational channels) and where to send that data (your Log Analytics Workspace). az monitor data-collection rule association create --resource <vmss-resource-id> --name assoc-dcr --rule-id <dcr-resource-id> Creating a rule doesn't do anything on its own until you apply it to a resource. This command creates a Data Collection Rule Association (DCRA). What happens behind the scenes: When you create this association, the Azure Monitor Agent running on your Service Fabric VMSS nodes detects the new rule. The agents download the dcr-servicefabric-windows.json instructions, begin collecting the specified performance counters and the selected Windows Event Log channels, and start streaming that data into Azure Monitor. The event channels collected (XPath queries in the DCR). These XPath queries are the exact filtering rules your Data Collection Rule (DCR) uses to tell the Azure Monitor Agent which Windows Event Logs to capture and which ones to ignore. By filtering at the source (on the VM itself), you save on ingestion costs and reduce noise in your Log Analytics workspace. Application!*[System[(Level=1 or Level=2 or Level=3)]] System!*[System[(Level=1 or Level=2 or Level=3)]] Microsoft-ServiceFabric/Admin!*[System[(Level=1 or Level=2 or Level=3 or Level=4)]] Microsoft-ServiceFabric/Operational!*[System[(Level=1 or Level=2 or Level=3 or Level=4)]] Here is a breakdown of what those specific queries are collecting: Standard Windows Logs: Application!*[System[(Level=1 or Level=2 or Level=3)]]: Captures events from the Windows Application log, but only if they are Critical (Level 1), Error (Level 2), or Warning (Level 3). System!*[System[(Level=1 or Level=2 or Level=3)]]: Captures events from the Windows System log, applying the same filter for Critical, Error, and Warning events. Notice that these exclude Level 4 (Information) and Level 5 (Verbose) events, which happen constantly and would unnecessarily bloat your log storage. Service Fabric Specific Logs: Microsoft-ServiceFabric/Admin!*[System[(Level=1 or Level=2 or Level=3 or Level=4)]]: Captures Admin channel events for Service Fabric. Because these are critical to understanding cluster health, it includes Information (Level 4) events alongside Warnings, Errors, and Criticals. Microsoft-ServiceFabric/Operational!*[System[(Level=1 or Level=2 or Level=3 or Level=4)]]: Captures Operational channel events (like node up/down, application deployment success/failure) using the same Level 1-4 filter. 5. Deploy Grafana and grant access This step represents the final infrastructure deployment for your repro: provisioning the visualization layer (AMG) and securely connecting it to the data we started collecting in the previous steps. 1- Provisioning Azure Managed Grafana (AMG): az grafana create -g rg-sf-grafana-repro --location westus2 --name amg-sfg08120632 --zone-redundancy Disabled 2- Wiring Up Permissions (The Manual Workaround) When you create an Azure Managed Grafana instance, Azure automatically creates a System-Assigned Managed Identity for it. Think of this as a "service account" that Grafana uses to talk to other Azure resources securely without needing passwords. The next commands grant the necessary permissions for the data to flow and for you to log in. Role 1: Monitoring Reader (Grafana's Identity) az role assignment create --assignee <grafana-mi-principal-id> --role "Monitoring Reader" --scope <resource-group-id> Note: by default, Azure Managed Grafana already gets Monitoring Reader on creation, and that single role is sufficient to query both Azure Monitor metrics and Log Analytics logs (Grafana permissions docs). A separate Log Analytics Reader assignment is normally redundant; we recreated Monitoring Reader manually above because the automatic role assignment failed in this repro (see below), scoped to the resource group. For teammates who only need to view the dashboard, assign the built-in Grafana Viewer role instead of Grafana Admin. Role 2: Grafana Admin (Your User Identity) az role assignment create --assignee <your-user-id> --role "Grafana Admin" --scope <grafana-resource-id> This one is for you, not the Grafana service. Even if you are the Subscription Owner who created the resource, Azure Managed Grafana has its own internal data plane authorization. This command guarantees that when you navigate to the Grafana URL in your web browser, you have full administrative rights to build dashboards, add data sources, and manage the workspace. 6. Import the dashboard az grafana dashboard import -g rg-sf-grafana-repro -n amg-sfg08120632 --definition dashboard-servicefabric-grafana.json --overwrite true Now that the data is being collected and Grafana has the permissions to read it, this command actually builds the visualization layer. Instead of manually clicking through the Grafana UI to create graphs, panels, and queries one by one, this command automates the deployment of a pre-built dashboard. --definition @dashboard-servicefabric-grafana.json: This is the payload. The @ symbol is critical here. it tells the Azure CLI, "Do not treat this as a string; open this local file and read its contents." This JSON file contains the entire blueprint for your dashboard, including the layout of the panels, the KQL queries used to fetch those Service Fabric Admin/Operational events, and the formatting rules. See the same GitHub repo for dashboard-servicefabric-grafana.json . Dashboard panels: Panel Source Purpose CPU by node Perf Detect overloaded nodes. Available memory by node Perf Spot memory pressure affecting replicas/services. Disk free space by node Perf Watch capacity before disks fill. SF warning/error events Event Trend platform warnings and errors. Recent SF events Event Troubleshoot recent cluster/node issues. SF Operational events Event Application/service lifecycle activity. First capture immediately after import, panels show ‘No data’ because AMA telemetry takes ~10–15 mins to appear (and only after the managed-identity fix) Once telemetry flows, the same dashboard populates with data across all three nodes: Dashboard with live data: CPU, memory, disk, and SF events per node If you have a deployed app in your cluster, the application appears in the portal and its lifecycle events flow through the Operational channel into Log Analytics and onto the dashboard: 7. Useful KQL queries 1. Tracking CPU Utilization per Node Perf | where ObjectName == "Processor" and CounterName == "% Processor Time" and InstanceName == "_Total" | summarize AvgCpu = avg(CounterValue) by bin(TimeGenerated, 5m), Computer This query targets the standard Windows performance counters collected by the Azure Monitor Agent. 2. Auditing Service Fabric Lifecycle Events Event | where EventLog == "Microsoft-ServiceFabric/Operational" | project TimeGenerated, Computer, EventID, RenderedDescription | order by TimeGenerated desc 3. Visualizing Cluster Health and Error Spikes Event | where EventLog startswith "Microsoft-ServiceFabric" | summarize count() by bin(TimeGenerated, 5m), EventLevelName 8. Validate the pipeline Before trusting the dashboard, confirm each stage of the pipeline in order as a broken step upstream will silently make the dashboard look "empty" for the wrong reason. 1- AMA extension is installed on the VMSS. In the portal, open the VM scale set → Settings (Extensions + applications), and confirm AzureMonitorWindowsAgent shows Provisioning succeeded. Or from CLI: az vmss extension list -g rg-sf-grafana-repro --vmss-name nt1vm --query "[].{Name:name, State:provisioningState}" 2- DCR is associated with the VMSS. The Data Collection Rule must have exactly one connected resource (the VMSS). In the portal, open the DCR → Resources: az monitor data-collection rule association list --resource <vmss-resource-id> 3- Telemetry is actually landing in Log Analytics. Open the workspace → Logs, and run each of these and you should see rows within the last hour: Heartbeat | where TimeGenerated > ago(1h) | summarize count() by Computer Perf | where TimeGenerated > ago(1h) | summarize count() by ObjectName Event | where TimeGenerated > ago(1h) | summarize count() by EventLog If Heartbeat is empty, AMA isn't authenticating (usually the missing managed identity). If Heartbeat is fine but Perf/Event is empty, the DCR isn't matching what you expect. 4- Grafana's managed identity has the right role. In the portal, open the Managed Grafana instance → Access control (IAM) → Role assignments, and confirm the system-assigned identity has Monitoring Reader at the subscription/RG/workspace scope you expect. 5- Dashboard time range is wide enough. Set the dashboard to Last 1 hour (or wider), the default "Last 5 minutes" will look empty even when everything is working, since AMA batches data. The ~10–15-minute figure elsewhere in this article is latency observed in this repro after fixing the managed-identity issue, not a guaranteed SLA. In practice, first data lands anywhere from a couple of minutes to ~15 minutes after AMA authenticates. 9. Real failures encountered (and fixes) Empty Grafana panels: missing VMSS managed identity After AMA + DCR were correctly configured, panels stayed empty and no Perf/Event/Heartbeat rows appeared in Log Analytics. Root cause: the template-created VM scale set had NO managed identity, so Azure Monitor Agent could not authenticate to Azure Monitor. Fix: az vmss identity assign -g <rg> -n <vmss> Data started flowing ~10–15 minutes later. Grafana CLI role-assignment API error 'az grafana create' created the instance but then crashed with 'APIVersion 2022-04-01 is not available' while auto-creating role assignments. The instance is fine; create the role assignments manually (Monitoring Reader, Log Analytics Reader, Grafana Admin). We hit this with Azure CLI 2.40.0 and amg extension version 1.3.6, run: az extension update --name amg first, as this may already be fixed in a newer release. So, when you run the manual az role assignment create commands, you are bypassing that specific auto-assignment code path in the amg extension. Instead, you are using the core Azure CLI's role assignment engine directly, which is a reliable workaround regardless of the root cause. 10. Cost and cleanup This repro uses paid Azure services, and leaving it running has an ongoing cost. Two components dominate: Log Analytics bills for data ingestion (GB ingested per day) and data retention (GB stored beyond the free 30-day window). Verbose event channels and short bin intervals both push ingestion up quickly. See Log Analytics pricing. Azure Managed Grafana has two tiers: Essential (lower cost, fewer active users, no zone redundancy) and Standard (per-instance hourly rate + per-active-user fee). See Azure Managed Grafana pricing. Check the current pricing in your region before enabling this in production, and delete the whole repro resource group when you're done: az group delete --name rg-sf-grafana-repro --yes --no-wait References: Service Fabric diagnostics with AMA and DCRs Azure Monitor Agent requirements Azure Managed Grafana permissions Import a Grafana dashboard
Installing AzureMonitoringAgent and linking it to your Log Analytics Workspace
The current Service Fabric clusters are currently equipped with the MicrosoftMonitoringAgent (MMA) as the default installation. However, it is essential to note that MMA will be deprecated in August 2024, for more details refer- We're retiring the Log Analytics agent in Azure Monitor on 31 August 2024 | Azure updates | Microsoft Azure. Therefore, if you are currently utilizing MMA, it is imperative to initiate the migration process to AzureMonitoringAgent (AMA). Installation and Linking of AzureMonitoringAgent to a Log Analytics Workspace: Create a Log Analytics Workspace (if not already established): Access the Azure portal and search for "Log Analytics Workspace." Proceed to create a new Log Analytics Workspace. Ensure that you select the identical resource group and geographical region where your cluster is located. Detailed explanation: Create Log Analytics workspaces - Azure Monitor | Microsoft Learn Create Data Collection Rules: Access the Azure portal and search for "Data Collection Rules (DCR)”. Select the same resource group and region as of your cluster. In Platform type, select the type of instance you have like Windows, Linux or both. You can leave data collection endpoint as blank. In the resources section, add the Virtual machine Scale Set (VMSS) resource which is attached to the Service fabric cluster. In the "Collect and deliver" section, click on Add data source and add both Performance Counters and Windows Event Logs one by one. Choose the destination for both the data sources as Azure Monitor Logs and in the Account or namespace dropdown, select the name of the Log Analytics workspace that we have created in step 1 and click on Add data source. Next click on review and create. Note: - For more detailed explanation on how to create DCR and various ways of creating it, you can follow - Collect events and performance counters from virtual machines with Azure Monitor Agent - Azure Monitor | Microsoft Learn Adding the VMSS instances resource with DCR: Once the DCR is created, in the left panel, click on Resources. Check if you can see the VMSS resource that we have added while creating DCR or not. If not then, click on "Add" and navigate to the VMSS attached to service fabric cluster and click on Apply. Refresh the resources tab to see whether you can see VMSS in the resources section or not. If not, try adding a couple of times if needed. Querying Logs and Verifying AzureMonitoringAgent Setup: Please allow for 10-15 minutes waiting period before proceeding. After this time has elapsed, navigate to your Log Analytics workspace, and access the 'Logs' section by scrolling through the left panel. Run your queries to see the logs. For example, query to check the heartbeat of all instances:Heartbeat | where Category contains "Azure Monitor Agent" | where OSType contains "Windows" You will see the logs there in the bottom panel as shown in the above screenshot. Also, you can modify the query as per your requirement. For more details related to Log Analytics queries, you can refer- Log Analytics tutorial - Azure Monitor | Microsoft Learn Perform the uninstallation of the MicrosoftMonitoringAgent (MMA): Once you have verified that the logs are getting generated, you can go to Virtual Machine Scale Set and then to the "Extensions + applications" section and delete the old MMA extension from VMSS.Preserve Disk space in ImageStore for Service Fabric Managed Clusters
As mentioned in this article: Service Fabric: Best Practices to preserve disk space in Image Store ImageStore keeps copied packages and provisioned packages. In this article, we will discuss how can you configure cleaning up the copied application package for Service Fabric Managed Cluster 'SFMC'. The mitigation is to set "AllowRuntimeCleanupUnusedApplicationTypesPolicy": "true". For properties, specify the following tags: ... "applicationTypeVersionsCleanupPolicy": { "maxUnusedVersionsToKeep": 3 } Let me show you a step-by-step guide to automatically remove the unwanted application versions in your Service Fabric Managed cluster below: Scenario: - I have deployed 4 versions of my app (1 InUse - 3 UnUsed) to my managed cluster as the following: Symptom: I need Service Fabric to do automatic cleanup for the Application Unused Versions and keep only the last 3 so as not to fill the disk space. Mitigation Steps: From https://resources.azure.com/ open your managed cluster resource and open the read/write mode. Add the tag "AllowRuntimeCleanupUnusedApplicationTypesPolicy": "true" Under fabricsettings, add the param "name": "CleanupUnusedApplicationTypes", "value": "true" under fabricsettings and set the "maxUnusedVersionsToKeep": 3 Click on PUT to save the changes, and I deployed the 5th version (1.0.4) to the cluster, which should make the cleaning happens for the oldest version (1.0.0) Note: The automatic clean-up should be effective after 24 hours of making those changes. Then I tried to deploy a new version, and I could see that the oldest version was also cleaned up. For manual cleanup of the ImageStoreService: You can use PowerShell commands to delete copied packages and unregister application types as needed. This includes using Get-ServiceFabricImageStoreContent to retrieve content and Remove-ServiceFabricApplicationPackage to delete it, as well as Unregister-ServiceFabricApplicationType to remove application packages from the image store and image cache on nodes.
Not enough disk space issue in Service Fabric cluster
Time by time, when you use Service Fabric cluster, the cluster may meet different issues and in the reported error/warning message, it’s marked that one or some specific nodes do not have enough disk space. This may be caused by different reasons. This blog will talk about the common solution of this issue. Possible root causes: There will be lots of possible root causes for the not enough disk space issue. In this blog, we'll mainly talk about following five: Diagnostic log files (.trace and .etl) consumes too much space Paging file consumes too much space Too many application packages existed in node Too many registered versions of application type Too many images existed (Only for cluster used with container) To better identify which one is matching your own scenario, please kindly check the following description of them: For the log files, we can RDP into the node reporting not enough disk space and check the size of the folder D:\SvcFab\Log. If the size of this folder is bigger than expected, then we can try to reconfigure the cluster to decrease the size limit of the diagnostic log files. For the paging files, it's a built-in feature of Windows system. For detailed introduction, please check this document. To verify if we got this issue, we can RDP into the node and check whether we can find the hidden file D:\pagefile.sys. If we can find it, that means your service fabric cluster is consuming some disk space as RAM. We can consider about configuring the Paging file to be saved in disk C instead of disk D. For too many application packages existed in node which consume too much disk space, we can verify it in Service Fabric Explorer (SFX). By visiting SFX from Azure Portal Service Fabric overview page, we can turn to the Image Store page of the cluster and verify whether there is any record with name different from Store and WindowsFabricStore. If yes, then please click on Load Size button to check its size. Similar to point 3, for too many registered versions of application type, we can check same page, but pay attention to size of Store and see if it consumes lots of disk space. When a version of application type is registered in SF cluster, SF will save some files used for deploy the services included by this new version into each node. More versions are registered, more disk space it will consume. For the too many images existed cause, this will only happen when our service fabric cluster is running with Container feature. We can RDP into the node and use command docker image ls to list all used images on this node. If there are some images which were used before but not removed/pruned even after no longer being used, it will consume a lot of disk space since the image file is normally with a huge size. For example, the size of image for windows server core is more than 10 GB. Possible solutions: Then let's talk about the solutions of the above four kinds of issue. 1. To reconfigure the size limit of the diagnostic log files, we need to open a PowerShell command window with Az module installed. Please refer to the official document for how to install. After login successfully, we can use the following command to set the expected size limit. Set-AzServiceFabricSetting -ResourceGroupName SF-normal -Name sfhttpjerry -Section Diagnostics -Parameter MaxDiskQuotaInMB -Value 25600 Please remember to replace the resource group name, service fabric cluster name and the number for size limit by yourself before running command. Once this command is run successfully, it may not take effect immediately. The Service Fabric cluster will scan size of the diagnostic log periodically. We need to wait until next scan is triggered. Once it's triggered and if the size of the diagnostic log files are bigger than your setting number (25600MB = 20 GB in my example), cluster will automatically delete some log files to release more disk space. 2. To change the path of paging file, we can follow these steps to switch. Check the status of our Service Fabric cluster in Service Fabric Explorer to make sure every node, service and application is healthy. RDP into the VMSS node In the Search bar, type in "Advanced System Setting". Then choose Advanced -> Advanced -> Change -> Next is to set D drive to No Paging file and set C drive to System Managed Size. This setting change will need user to reboot the VMSS node to take effect. Please reboot the node and wait until everything is back to healthy status in Service Fabric Explorer before RDP into next node. Repeat above steps for all nodes. 3. To clean up the application package, this is easy to do in SFX. Once we visit SFX, go to the same Image Store page as how we check the issue for this kind of root cause. Then on the left side, there will be a menu to delete the unneeded package. After typing in the name in confirmation window and select Delete Image Store content, cluster will automatically delete the unneeded application on every node. 4. For the issue caused by too many registered versions of application type, we need to manually unregister the not needed versions. In the Service Fabric Explorer, we can click on the Application/Application type to see the currently existing versions. If there is any not currently used and no more needed versions, please use following command to unregister: Unregister-ServiceFabricApplicationType -ApplicationTypeName "application type name" -ApplicationTypeVersion "version number" -Force 5. For the issue caused by too many images, we can configure the cluster to automatically deleted the unused image. The detailed configuration can be found in this document and the way about how to update cluster configuration will be as following: a. Visit Azure Resource Explorer with Read/Write mode, login and find the Service Fabric cluster. b. Click Edit button and modify the json format cluster configuration as expected. In this solution, it will be to add some configuration into fabricSettings part. c. Send out the request to save the new configuration by clicking the green PUT button and wait until the provisioning status of this cluster becomes Succeeded. To make this solution work, there is one more thing which we need to do is to unregister all unnecessary and unused applications. This can also be done by the command documented here. Since the parameter ApplicationTypeName and ApplicationTypeVersion are both required for this command, that means we can only unregister one version of an application type after running the command once. But since maybe you may have very many versions and many application types, here are 2 following possible ways: If there is/are actually some versions of some application types which you want to keep it registered for future use in this cluster, please unregister those unnecessary versions by running command Unregister-ServiceFabricApplicationType -ApplicationTypeName VotingType -ApplicationTypeVersion 1.0.1 (Remember to replace the ApplicationTypeName and ApplicationTypeVersion and Use step 2.e to connect to cluster at first.) If there isn't any version of any application type which you want to keep specially, which means we only need to keep the application type being used by running application, then we can use step 2.e to connect to cluster and then run the following script: $apptypes = Get-ServiceFabricApplicationType $apps = Get-ServiceFabricApplication $using = $false foreach ($apptype in $apptypes) { $using = $false foreach ($app in $apps) { if ($apptype.ApplicationTypeName -eq $app.ApplicationTypeName -and $apptype.ApplicationTypeVersion -eq $app.ApplicationTypeVersion) { $using = $true break } } if ($using -eq $false) { Unregister-ServiceFabricApplicationType -ApplicationTypeName $apptype.ApplicationTypeName -ApplicationTypeVersion $apptype.ApplicationTypeVersion -Force } } In additional to the above four possible causes and solutions, there are three more possible solutions for the "Not enough disk space" issue. The following is the explanation. Scale out VMSS: Sometimes scaling out, which means increasing the number of nodes of Service Fabric cluster will also help us to mitigate the disk full issue. This operation will not only be useful to improve the CPU and memory usage, but will also auto-balance the distribution of the services among nodes to improve the disk usage. When using the Silver or higher durability, to scale out the VMSS instances, we can scale out the nodes number in the VMSS directly. Scale up VMSS: It is easy to understand this point. Since the issue is about the full disk, we can simply change the VM sku to a bigger size to have bigger disk space. But please kindly check all above solutions at first to make sure everything is reasonable and we do really need more disk size to handle more data. For example, if our application is with stateful services and the full disk happens due to our stateful services save too much data, then we should consider about improving the code logic but not scaling out the VMSS at first. Otherwise, with bigger VM sku, the issue will still reproduce sooner or later. To scale up the VMSS, we can following two ways: We can use the command Update-AzVmss to update the state of a VMSS. This is the simple way however the solution is not recommended, because there is a little risk of data loss/instance down. When using the Silver or higher durability, the risk can be mitigated because they support repair tasks. The second way to upgrade the size of the SF primary node type is adding the new node type with the bigger SKU. The option is much more difficult than the option one but officially recommended and you can check the document for more information. Reconfigure the ReplicatorLog size: Please be careful that the ReplicatorLog is not saving any kind of log file. It's saving some important data of both Service Fabric cluster and application. Delete this folder will possibly cause data loss. And the size of this folder is fixed to the configured size, by default 8 GB. It will always be the same size no matter how much data is saved. It's NOT recommended to modify this setting. You should only do it if you absolute have to do so. It may run the risk of data loss. This should only be done if absolutely required. For the ReplicatorLog size, as mentioned above, the key point is to add a customized ktlLogger setting into the Service Fabric cluster. To do that, we need to: a. Visit Azure Resource Explorer with Read/Write mode, login and find the Service Fabric cluster. b. Add the ktlLogger setting into fabricSettings part. The expected expression will be such as following: { "name": "KtlLogger", "parameters": [{ "name": "SharedLogSizeInMB", "value": "4096" }] } c. Send out the request to save the new configuration by clicking the green PUT button and wait until the provisioning status of this cluster becomes Succeeded. d. Visit SFX and check the status to make sure everything is in healthy state. e. Open a PowerShell command window from a computer where the cluster certificate is installed. If the Service Fabric module is not installed yet, please refer to our document to install at first. Then run following command to connect to the Service Fabric cluster. Here the thumbprint is the one of the cluster certificate and also remember to replace the cluster name by correct URL. $ClusterName= "xxx.australiaeast.cloudapp.azure.com:19000" $CertThumbprint= "7279972D160AB4C3CBxxxxx34EA2BCFDFAC2B42" Connect-serviceFabricCluster -ConnectionEndpoint $ClusterName -KeepAliveIntervalInSec 10 -X509Credential -ServerCertThumbprint $CertThumbprint -FindType FindByThumbprint -FindValue $CertThumbprint -StoreLocation CurrentUser -StoreName My f. Use command to disable one node from Service Fabric cluster. (_nodetype1_0 in example code) Disable-ServiceFabricNode -NodeName "_nodetype1_0" -Intent RemoveData -Force g. Monitor in SFX until the node included in last command is with status disabled. h. RDP into this node and manually delete the D:\SvcFab\ReplicatorLog folder. Attention! This operation will remove all logs in ReplicatorLog. Please double-confirm whether any context there is still needed before deletion. i. Use following command to enable the disabled node. Monitor until the node is with status Up. Enable-ServiceFabricNode -NodeName "_nodetype1_0" j. Wait until everything is healthy in SFX and repeat step f to step i on every node. After that, the ReplicatorLog folder in node will be with new customized size.
SSL/TLS connection issue troubleshooting guide
You may experience exceptions or errors when establishing TLS connections with Azure services. Exceptions are vary dramatically depending on the client and server types. A typical ones such as "Could not create SSL/TLS secure channel." "SSL Handshake Failed", etc. In this article we will discuss common causes of TLS related issue and troubleshooting steps.Service Fabric Explorer (SFX) web client CVE-2023-23383 spoofing vulnerability
Service Fabric Explorer (SFX) is the web client used when accessing a Service Fabric (SF) cluster from a web browser. The version of SFX used is determined by the version of your SF cluster. We are providing this blog to make customers aware that running Service Fabric versions 9.1.1436.9590 and below are affected. These versions could potentially allow unwanted code execution in the cluster if an attacker can successfully convince a victim to click a malicious link and perform additional actions in the Service Fabric Explorer interface. This issue has been resolved in Service Fabric 9.1.1583.9589 released on March 14th, 2023, as CVE-2023-23383 which had a score of CVSS: 8.2 / 7.1. See the Technical Details section for more information.
Force delete application while the Application/Service stuck in deleting state
Few days back i was working on scenario where I was unable to delete the application and service from Azure Service fabric explorer and PowerShell. By looking at the application state in Service Fabric Explorer the application was in deleting state. Executing the PowerShell command left out with the following error: Remove-ServiceFabricApplication -ApplicationName fabric:/Voting -Force -TimeoutSec 350 Remove-ServiceFabricApplication : Operation timed out. At line:1 char:1 + Remove-ServiceFabricApplication -ApplicationName fabric:/Voting -Forc ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OperationTimeout: (Microsoft.Servi...usterConnection:ClusterConnection) [Remove-ServiceFabricApplication], TimeoutException + FullyQualifiedErrorId : RemoveApplicationInstanceErrorId,Microsoft.ServiceFabric.Powershell.RemoveApplication At this scenario, there are PowerShell cmdlet switches which play a vital role and can help to mitigate the issue. I have tried using -ForceRemove switch. The following cmdlet helps me to delete all application that are distributed across multiple nodes. $ClusterName= "clustername.cluster_region.cloudapp.azure.com:19000" $Certthumprint = "{replace_with_ClusterThumprint}" Connect-ServiceFabricCluster -ConnectionEndpoint $ClusterName -KeepAliveIntervalInSec 10 ` -X509Credential ` -ServerCertThumbprint $Certthumprint ` -FindType FindByThumbprint ` -FindValue $Certthumprint ` -StoreLocation CurrentUser ` -StoreName My $ApplicationName = "fabric:/voting" foreach($node in Get-ServiceFabricNode) { [void](Get-ServiceFabricDeployedReplica -NodeName $node.NodeName -ApplicationName $ApplicationName | Remove-ServiceFabricReplica -NodeName $node.NodeName -ForceRemove) } Remove-ServiceFabricApplication -ApplicationName $ApplicationName -Force There might be situation where -Force switch could also fail. In that scenario try with 1) Move the Cluster Manager primary to another node and try to remove the app again: a) Move-ServiceFabricPrimaryReplica -PartitionId [Cluster Manager System Service Partition Id] -ServiceName fabric:/System/ClusterManagerService b) Remove-ServiceFabricApplication -ApplicationName fabric:/voting -Force -ForceRemove 2) If it still doesn't work due to stuck services or PLB affinity, try to move the primary for each partition of NamingService and try again. There might also a scenario that the issue is happening with only one service distributed in multiple nodes instead of the whole application. Then use the below script which is like the one for application but with a little change which will only delete a specific Service. $ClusterName= "clustername.cluster_region.cloudapp.azure.com:19000" $Certthumprint = "{replace_with_ClusterThumprint}" Connect-ServiceFabricCluster -ConnectionEndpoint $ClusterName -KeepAliveIntervalInSec 10 ` -X509Credential ` -ServerCertThumbprint $Certthumprint ` -FindType FindByThumbprint ` -FindValue $Certthumprint ` -StoreLocation CurrentUser ` -StoreName My $ApplicationName = "fabric:/voting" $ServiceName = "fabric:/Voting/FEService" foreach($node in Get-ServiceFabricNode) { [void](Get-ServiceFabricDeployedReplica -NodeName $node.NodeName -ApplicationName $ApplicationName | Where-Object {$_.ServiceName -match $ServiceName} | Remove-ServiceFabricReplica -NodeName $node.NodeName -ForceRemove) } Remove-ServiceFabricService -ServiceName $ServiceName -Force Hope this helps.
