%3CLINGO-SUB%20id%3D%22lingo-sub-1978657%22%20slang%3D%22en-US%22%3EService%20Fabric%3A%20Best%20Practices%20to%20preserve%20disk%20space%20in%20Image%20Store%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1978657%22%20slang%3D%22en-US%22%3E%3CH1%20id%3D%22toc-hId-616227086%22%20id%3D%22toc-hId-616227116%22%3EImageStore%20keeps%20copied%20package%20and%20provisioned%20packages.%3C%2FH1%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1306788560%22%20id%3D%22toc-hId-1306788590%22%3ESequence%20of%20provision%20in%20Best%20Practice%3A%3C%2FH2%3E%0A%3COL%3E%0A%3CLI%3ECopy%20package%20to%20ImageStore%20with%20compress%20option%3C%2FLI%3E%0A%3CLI%3EProvision%20package%3C%2FLI%3E%0A%3CLI%3ERemove%20package%20in%20ImageStore%3C%2FLI%3E%0A%3CLI%3EUpgrade%20app%2Fcluster%3C%2FLI%3E%0A%3CLI%3EUnprovision%20old%20version%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EWhen%20step%203%20and%205%20are%20missing%2C%20ImageStore%20accumulates%20files.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--500665903%22%20id%3D%22toc-hId--500665873%22%3ESymptom%3A%3C%2FH2%3E%0A%3CUL%3E%0A%3CLI%3EThe%20ImageStoreService%20(fabric%3A%2FSystem%2FImageStoreService)%20could%20%3CSTRONG%3Efill%20up%20disk%3C%2FSTRONG%3E.%3C%2FLI%3E%0A%3CLI%3EImageStoreService%20replica%20may%20%3CSTRONG%3Etake%20long%20time%20in%20InBuild%3C%2FSTRONG%3E.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1986846930%22%20id%3D%22toc-hId-1986846960%22%3EOption%2FConfiguration%20for%20automatic%20cleanup%3A%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHow%20to%20configure%20cleaning%20up%20copied%20application%20package%20(%3CSTRONG%3Eautomatic%20Step%203%3C%2FSTRONG%3E)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CSTRONG%3ERegister-ServiceFabricApplicationType%3C%2FSTRONG%3E%20%3CSTRONG%3E-%3C%2FSTRONG%3E%3CSTRONG%3EApplicationPackageCleanupPolicy%3C%2FSTRONG%3E%3CSTRONG%3E%20Automatic%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EAt%20step%202%2C%20Register%20(a.k.a.%20Provision)%2C%20the%20application%20package%20is%20deleted%20after%20successfully%20registering%20the%20application%20type.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CSECTION%20name%3D%22%26quot%3BManagement%26quot%3B%22%3E%3C%2FSECTION%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%3CPARAMETER%20name%3D%22%26quot%3BCleanupApplicationPackageOnProvisionSuccess%26quot%3B%22%20value%3D%22%26quot%3BFalse%26quot%3B%22%3E%3C%2FPARAMETER%3E%3C%2FP%3E%0A%3CP%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CP%3E%3C%2FP%3E%0A%3CP%3EThis%20configuration%20enabled%20automatic%20cleanup%20of%20application%20package%20after%20successfully%20registering%20the%20application%20type.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHow%20to%20configure%20cleaning%20up%20automatically%20unused%20application%20type%20(%3CSTRONG%3Eautomatic%20Step%205%3C%2FSTRONG%3E)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%3CSECTION%20name%3D%22%26quot%3BManagement%26quot%3B%22%3E%3C%2FSECTION%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%3CPARAMETER%20name%3D%22%26quot%3BCleanupUnusedApplicationTypes%26quot%3B%22%20value%3D%22%26quot%3Btrue%26quot%3B%22%3E%3C%2FPARAMETER%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3CPARAMETER%20name%3D%22%26quot%3BPeriodicCleanupUnusedApplicationTypes%26quot%3B%22%20value%3D%22%26quot%3Btrue%26quot%3B%22%3E%3C%2FPARAMETER%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3CPARAMETER%20name%3D%22%26quot%3BTriggerAppTypeCleanupOnProvisionSuccess%26quot%3B%22%20value%3D%22%26quot%3Btrue%26quot%3B%22%3E%3C%2FPARAMETER%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%3CPARAMETER%20name%3D%22%26quot%3BMaxUnusedAppTypeVersionsToKeep%26quot%3B%22%20value%3D%22%26quot%3B3%26quot%3B%22%3E%3C%2FPARAMETER%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-179392467%22%20id%3D%22toc-hId-179392497%22%3EManual%20Cleanup%3A%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhen%20ImageStoreService%20must%20be%20cleaned%20up%20manually%2C%20you%20can%20follow%20this%20steps.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E1.%20Delete%20copied%20packages%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%23Delete%20Content%20from%20Image%20Store%3C%2FP%3E%0A%3CP%3E%24content%20%3D%3C%2FP%3E%0A%3CP%3EGet-ServiceFabricImageStoreContent%20-RemoteRelativePath%20%22%5C%22%20-ImageStoreConnectionString%20fabric%3AImageStore%3C%2FP%3E%0A%3CP%3Eforeach(%24folder%20in%20%24content)%3C%2FP%3E%0A%3CP%3E%7B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Write-Host%20Working%20on%20%24folder.StoreRelativePath%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20if%20((%24folder.StoreRelativePath%20-ne%20%22Store%22)%20-and%20(%24folder.StoreRelativePath%20-ne%20%22WindowsFabricStore%22))%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%7B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Write-Host%20Deleting%20%24folder.StoreRelativePath%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%3CSTRONG%3ERemove-ServiceFabricApplicationPackage%3C%2FSTRONG%3E%20-ApplicationPackagePathInImageStore%20%24folder.StoreRelativePath%20-ImageStoreConnectionString%20fabric%3AImageStore%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20%7D%3C%2FP%3E%0A%3CP%3E%7D%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E2.%20Unregister%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CLI-WRAPPER%3E%3C%2FLI-WRAPPER%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EUnregister-ServiceFabricApplicationType%3C%2FSTRONG%3E%20%3CSPAN%3Ewill%20remove%20application%20packages%20from%20image%20store%20as%20well%20as%20ImageCache%20on%20nodes%20(after%20a%20while).%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CU%3E%3CSTRONG%3EScenario1%26nbsp%3B%20%3A%3C%2FSTRONG%3E%3C%2FU%3E%26nbsp%3B%26nbsp%3B%20Seeing%20zombie%20application%20package%20in%20the%20store%2C%20which%20taking%20most%20of%20the%20disk%20space.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EVerify%20whether%20meta%20data%20for%20the%20older%20version%20%26nbsp%3Bis%20associated%20with%20ISS%20%26nbsp%3B%5B%3CSTRONG%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%20Get-SFImageStoreContent%20-remoterelativepath%26nbsp%3B%20%E2%80%98Store%5CIberFabric.App.PortalComercial.Services.AgendaType%E2%80%99%3C%2FSTRONG%3E%20%26nbsp%3Bverify%20whether%20older%20package%20type%20is%20in%20the%20list%20%5D%3C%2FLI%3E%0A%3CLI%3EIf%20the%20older%20file%20is%20not%20listed%20in%20the%20%231%2C%20you%20can%20rdp%20into%20each%20VM%20which%20hosting%20the%20ISS%20replica%2C%20and%20delete%20it%20manually%3C%2FLI%3E%0A%3CLI%3EIf%20meta%20data%20is%20present%20in%20%231%20output%2C%20please%20verify%20the%20version%20customer%20wanted%20to%20delete%20is%20already%20in%20the%20registry%20by%20running%2C%20%E2%80%98%3CSTRONG%3EGet-ServiceFabricApplicationType%E2%80%99%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3CUL%3E%0A%3CLI%3EIn%20the%20%233%20output%2C%20If%20the%20application%20type%20is%20already%20there%2C%20please%20un%20provision%20by%26nbsp%3B%20%E2%80%98%3CSTRONG%3EUnregister-ServiceFabricApplicationType%E2%80%99%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3CLI%3EIf%20the%20%233%20output%20application%20type%20detail%20is%20not%20present%2C%20you%20can%20run%20%26nbsp%3B%E2%80%98%20%3CSTRONG%3ERemove-ServiceFabricApplicationPackage%20-ApplicationPackagePathInImageStore%20%22Store%5CIberFabric.App.PortalComercial.Services.AgendaType%5C%26lt%3B%3CPACKAGE%20with%3D%22%22%20version%3D%22%22%20number%3D%22%22%3E%26gt%3B%3C%2FPACKAGE%3E%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3C%2FUL%3E%0A%3CP%3E%5B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-application-upgrade-tutorial-powershell%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-application-upgrade-tutorial-powershell%3C%2FA%3E%5D%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpowershell%2Fmodule%2Fservicefabric%2Fregister-servicefabricapplicationtype%3Fview%3Dazureservicefabricps%23optional-parameters%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpowershell%2Fmodule%2Fservicefabric%2Fregister-servicefabricapplicationtype%3Fview%3Dazureservicefabricps%23optional-parameters%20%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENote%20%3A%20For%206.5%2B%20Cluster%2C%20SFX%20had%20ImageStoreService%20tab%20in%20Cluster%2C%20where%20GetSize%20button%20to%20get%20the%20size%20of%20Store%20content.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1978657%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22image_store_service.png%22%20style%3D%22width%3A%20293px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F239770iDE51C771DBE6A6D8%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22image_store_service.png%22%20alt%3D%22image_store_service.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1978657%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Service%20Fabric%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Microsoft

ImageStore keeps copied package and provisioned packages.

 

Sequence of provision in Best Practice:

  1. Copy package to ImageStore with compress option
  2. Provision package
  3. Remove package in ImageStore
  4. Upgrade app/cluster
  5. Unprovision old version

When step 3 and 5 are missing, ImageStore accumulates files.

 

Symptom:

  • The ImageStoreService (fabric:/System/ImageStoreService) could fill up disk.
  • ImageStoreService replica may take long time in InBuild.

 

Option/Configuration for automatic cleanup:

 

How to configure cleaning up copied application package (automatic Step 3)

 

  • Register-ServiceFabricApplicationType -ApplicationPackageCleanupPolicy Automatic

At step 2, Register (a.k.a. Provision), the application package is deleted after successfully registering the application type.

 

  • <Section Name="Management">

    <Parameter Name="CleanupApplicationPackageOnProvisionSuccess" Value="False" />

</Section>

This configuration enabled automatic cleanup of application package after successfully registering the application type.

 

How to configure cleaning up automatically unused application type (automatic Step 5)

    <Section Name="Management">

      <Parameter Name="CleanupUnusedApplicationTypes" Value="true" />

      <Parameter Name="PeriodicCleanupUnusedApplicationTypes" Value="true" />     

      <Parameter Name="TriggerAppTypeCleanupOnProvisionSuccess" Value="true" />

      <Parameter Name="MaxUnusedAppTypeVersionsToKeep" Value="3" />

    </Section>

 

Manual Cleanup:

 

When ImageStoreService must be cleaned up manually, you can follow this steps.

 

1. Delete copied packages

#Delete Content from Image Store

$content =

Get-ServiceFabricImageStoreContent -RemoteRelativePath "\" -ImageStoreConnectionString fabric:ImageStore

foreach($folder in $content)

{

    Write-Host Working on $folder.StoreRelativePath

    if (($folder.StoreRelativePath -ne "Store") -and ($folder.StoreRelativePath -ne "WindowsFabricStore"))

    {

        Write-Host Deleting $folder.StoreRelativePath

        Remove-ServiceFabricApplicationPackage -ApplicationPackagePathInImageStore $folder.StoreRelativePath -ImageStoreConnectionString fabric:ImageStore

    }

}

 

2. Unregister

Unregister-ServiceFabricApplicationType will remove application packages from image store as well as ImageCache on nodes (after a while).

 

Scenario1  :   Seeing zombie application package in the store, which taking most of the disk space.

 

  • Verify whether meta data for the older version  is associated with ISS  [    Get-SFImageStoreContent -remoterelativepath  ‘Store\IberFabric.App.PortalComercial.Services.AgendaType’  verify whether older package type is in the list ]
  • If the older file is not listed in the #1, you can rdp into each VM which hosting the ISS replica, and delete it manually
  • If meta data is present in #1 output, please verify the version customer wanted to delete is already in the registry by running, ‘Get-ServiceFabricApplicationType’
    • In the #3 output, If the application type is already there, please un provision by  ‘Unregister-ServiceFabricApplicationType’
    • If the #3 output application type detail is not present, you can run  ‘ Remove-ServiceFabricApplicationPackage -ApplicationPackagePathInImageStore "Store\IberFabric.App.PortalComercial.Services.AgendaType\<<package with version number>>

[https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-application-upgrade-tutorial-po...]

https://docs.microsoft.com/en-us/powershell/module/servicefabric/register-servicefabricapplicationty...

 

Note : For 6.5+ Cluster, SFX had ImageStoreService tab in Cluster, where GetSize button to get the size of Store content.