Issue:
With reference to article https://docs.microsoft.com/hi-in/azure/service-fabric/service-fabric-reliable-actors-delete-actors , after Calling the DeleteActorMethod and enumerating the Actor list, the disk space is not getting cleaned up.
What can we analyze from our end:
We can check the snapshot of the partition size after RDP into the node. Below is an example of one the Partition folder (E.g.: a16a1c07-1468-4664-bf4f-483436dcbda0) size before Calling DeleteActorMethod:
After Calling DeleteActorMethod: Disk space remains same:
Points to Note:
- Actual usage on disk depends on numerous factors from the underlying store and we don’t see immediate reduction of disk usage right after Actor deletion.
- Deleting data does not shrink the physical size of the DB down; it only shrinks the logical size (the size of the data) of the DB. However, the remaining space is reused when more data is added.
For example, imagine the execution of the following operations:
- Inserting 10GB of data.
- Deleting 7GB of data.
- Inserting 3GB of data.
The physical size of the DB remains to be 10GB after the occurrence of all the above operations. This is because of physical size not going down after the data is deleted as stated above. However, during step 3, the existing available space is reused instead of creating additional physical space.
- If we are interested in bringing the physical size of the db down, we can perform compaction. Shrinking of db file size is not supported proactively as it impacts write latency.
The recommended way is to test on workloads with real data which is having huge size. Disk space is rarely a bottleneck is general workloads.
- For compacting the partitions, we have added settings under LocalEseStoreSettings:
CompactionThresholdInMB = set to the max_data_size that customer expects to add like 5 GB + delta
FreePageSizeThresholdInMB = some threshold for skipping compaction if bloating is less than this size.. e.g. 500 MB
CompactionProbabilityInPercent = 5 or 10 %
These settings will make sure that compaction of partitions happen at appropriate time to reduce bloating of db files.
These can be set in settings.xml under “<ActorName>LocalStoreConfig” like “GameActorServiceLocalStoreConfig”
- Minimum db file size with 0 data is 4 MB. After that as we write data, file size will grow. Deletion information is shared above. Generally, file space is reused.
- The current compaction steps are deprecated in favor of new automatic compaction feature that we are working on priority and will update the Release Notes when the same is public.
- There may be a question where a node in the cluster fails for some reason and the cluster will automatically reconfigure the service replicas to the available node to maintain availability.
During this scenario, does the disk space reclaims?
The answer is No, because db file gets copied from some other node (which is in UP state) to new node.
For completeness, Replica folder and files get deleted on old node where replica is not needed anymore. New replica/node will get db files from current primary.