%3CLINGO-SUB%20id%3D%22lingo-sub-2110648%22%20slang%3D%22en-US%22%3ETroubleshooting%20-%20Stateless%2FStateful%20Services%20Cannot%20Be%20Started%20In%20Service%20Fabric%20cluster%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2110648%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20blog%20is%20about%20the%20issue%20that%20stateless%2Fstateful%20service%20cannot%20be%20started%20in%20service%20fabric%20cluster.%3C%2FP%3E%0A%3CP%3EIt%20provides%20a%20overview%20about%20reliable%20service%20life%20cycle%2C%20details%20of%20some%20important%20events%20like%26nbsp%3BCreateServiceInstanceListener%2C%20RunAsync%20and%20troubleshooting%20steps%20to%20identify%20the%20issue%20event%20and%20exceptions%20details.%3C%2FP%3E%0A%3CHR%20%2F%3E%0A%3CP%3E%3CFONT%20size%3D%225%22%3E%3CSTRONG%3EStateless%20And%20Stateful%20Service%20Life%20Cycle%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EThe%20lifecycle%20of%20a%20stateless%20service%3C%2FSTRONG%3E%20is%20straightforward.%20Here's%20the%20order%20of%20events%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B1.%20The%20service%20is%20constructed.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B2.%20Then%2C%20in%20parallel%2C%20two%20things%20happen%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%20StatelessService.CreateServiceInstanceListeners()%26nbsp%3Bis%20invoked%20and%20any%20returned%20listeners%20are%20opened.%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3BICommunicationListener.OpenAsync()%26nbsp%3Bis%20called%20on%20each%20listener.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%20The%20service's%26nbsp%3BStatelessService.RunAsync()%26nbsp%3Bmethod%20is%20called.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B3.%20If%20present%2C%20the%20service's%26nbsp%3BStatelessService.OnOpenAsync()%26nbsp%3Bmethod%20is%20called.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3BThis%20call%20is%20an%20uncommon%20override%2C%20but%20it%20is%20available.%20Extended%20service%20initialization%20tasks%20can%20be%20started%20at%20this%20time.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EStateful%20services%20have%20a%20similar%20pattern%20to%20stateless%20services%3C%2FSTRONG%3E%2C%20with%20a%20few%20changes.%20For%20starting%20up%20a%20stateful%20service%2C%20the%20order%20of%20events%20is%20as%20follows%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B1.%20The%20service%20is%20constructed.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B2.%20StatefulServiceBase.OnOpenAsync()%26nbsp%3Bis%20called.%20This%20call%20is%20not%20commonly%20overridden%20in%20the%20service.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3BThe%20following%20things%20happen%20in%20parallel%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20StatefulServiceBase.CreateServiceReplicaListeners()%26nbsp%3Bis%20invoked.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20If%20the%20service%20is%20a%20Primary%20service%2C%20all%20returned%20listeners%20are%20opened.%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3BICommunicationListener.OpenAsync()%26nbsp%3Bis%20called%20on%20each%20listener.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20If%20the%20service%20is%20a%20Secondary%20service%2C%20only%20those%20listeners%20marked%20as%26nbsp%3BListenOnSecondary%20%3D%20true%26nbsp%3Bare%20opened.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3BHaving%20listeners%20that%20are%20open%20on%20secondaries%20is%20less%20common.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20If%20the%20service%20is%20currently%20a%20Primary%2C%20the%20service's%26nbsp%3BStatefulServiceBase.RunAsync()%26nbsp%3Bmethod%20is%20called.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B3.%20After%20all%20the%20replica%20listener's%26nbsp%3BOpenAsync()%26nbsp%3Bcalls%20finish%20and%26nbsp%3BRunAsync()%26nbsp%3Bis%20called%2C%26nbsp%3BStatefulServiceBase.OnChangeRoleAsync()%26nbsp%3Bis%20called.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20This%20call%20is%20not%20commonly%20overridden%20in%20the%20service.%3C%2FP%3E%0A%3CHR%20%2F%3E%0A%3CP%3E%3CSTRONG%3E%3CFONT%20size%3D%225%22%3EEvents%20and%26nbsp%3BCancellation%20Token%3C%2FFONT%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ECreateServiceInstanceListener%3C%2FSTRONG%3E%20is%20to%20supply%20the%20communication%20listeners%20for%20the%20service%20instance%2C%20it%20is%20normally%20override%20in%20stateless%20service%20like%20using%20Kestrel%20%2C%20https%20and%20so%20on.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ERunAsync()%3C%2FSTRONG%3E%20is%20executed%20in%20its%20own%20task.%20Note%20that%20in%20the%20code%20snippet%20above%2C%20we%20jumped%20right%20into%20a%20while%20loop.%20There%20is%20no%20need%20to%20schedule%20a%20separate%20task%20for%20your%20workload.%20Cancellation%20of%20your%20workload%20is%20a%20cooperative%20effort%20orchestrated%20by%20the%20provided%20cancellation%20token.%20The%20system%20will%20wait%20for%20your%20task%20to%20end%20(by%20successful%20completion%2C%20cancellation%2C%20or%20fault)%20before%20it%20moves%20on.%20It%20is%20important%20to%20honor%20the%20cancellation%20token%2C%20finish%20any%20work%2C%20and%20exit%20RunAsync()%20as%20quickly%20as%20possible%20when%20the%20system%20requests%20cancellation.%20It%20will%20be%20triggered%20for%20stateful%20primary%20replica%20or%20all%20stateless%20instances%20and%20normally%20override%20in%20stateful%20service.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ECancellation%20token%3C%2FSTRONG%3E%20is%20provided%20to%20coordinate%20when%20your%20service%20instance%20needs%20to%20be%20closed.%20In%20Service%20Fabric%2C%20this%20open%2Fclose%20cycle%20of%20a%20service%20instance%20can%20occur%20many%20times%20over%20the%20lifetime%20of%20the%20service%20as%20a%20whole.%20This%20can%20happen%20for%20various%20reasons%2C%20including%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20The%20system%20moves%20your%20service%20instances%20for%20resource%20balancing.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20Faults%20occur%20in%20your%20code%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20The%20application%20or%20system%20is%20upgraded.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20*%26nbsp%3B%20The%20underlying%20hardware%20experiences%20an%20outage.%3C%2FP%3E%0A%3CHR%20%2F%3E%0A%3CP%3E%3CFONT%20size%3D%225%22%3E%3CSTRONG%3ETroubleshooting%3C%2FSTRONG%3E%3C%2FFONT%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBelow%20are%20some%20ways%20to%20get%20more%20insight%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B1.%20RDP%20to%20service%20fabric%20node.%20(primary%20replica%20node%20if%20it%20is%20stateful%20service)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B2.%20Check%20Application%20event%20logs%20for%20any%20exception%20if%20no%20exceptions%20go%20to%20step%203.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B3.%20Check%20if%20the%20port%20is%20occupied%20by%20the%20other%20services.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CEM%3E%3CFONT%20size%3D%223%22%3EFor%20TCP%3AGet-Process%20-Id%20(Get-NetTCPConnection%20-LocalPort%20YourPortNumberHere).OwningProcess%3C%2FFONT%3E%3C%2FEM%3E%3CBR%20%2F%3E%3CEM%3E%3CFONT%20size%3D%223%22%3EFor%20UDP%3AGet-Process%20-Id%20(Get-NetUDPEndpoint%20-LocalPort%20YourPortNumberHere).OwningProcess%3C%2FFONT%3E%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B4.%20For%20non-prod%20environment%2C%20remote%20debug%20would%20be%20a%20good%20way%20to%20get%20more%20insight%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%3CEM%3EPlease%20ref%20more%20details%20via%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fservice-fabric%2Fservice-fabric-debugging-your-application%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EDebug%20your%20application%20in%20Visual%20Studio%3C%2FA%3E%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B5.%20List%20underlying%20exceptions%20and%20capture%20dump%20via%20procmon%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%3CEM%3EStart-bitstransfer%20%22%3CA%20href%3D%22https%3A%2F%2Fdownload.sysinternals.com%2Ffiles%2FProcdump.zip%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Ehttps%3A%2F%2Fdownload.sysinternals.com%2Ffiles%2FProcdump.zip%3C%2FA%3E%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20procdump.exe%20-accepteula%20-e%201%20-f%20%22%22%26nbsp%3B%20-w%20%22processname%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Ping_Wu_0-1612232836718.png%22%20style%3D%22width%3A%20699px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F251204i6ED7AB021C2F3590%2Fimage-dimensions%2F699x346%3Fv%3D1.0%22%20width%3D%22699%22%20height%3D%22346%22%20role%3D%22button%22%20title%3D%22Ping_Wu_0-1612232836718.png%22%20alt%3D%22Ping_Wu_0-1612232836718.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B6.%20Then%20capture%20the%20dump%20for%20specific%20exception.%3C%2FP%3E%0A%3CP%3E%3CEM%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3Bprocdump.exe%20-ma%20-e%201%20-f%20%22NullReferenceException%22%20-w%20%22processname%22%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%207.%20Use%20Windbg%2C%20Debugdiag%20to%20get%20details%20about%20exception%20like%20method%20call%20stack.%3C%2FP%3E%0A%3CP%20class%3D%22lia-indent-padding-left-30px%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Ping_Wu_1-1612232836777.png%22%20style%3D%22width%3A%20742px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F251205i6B9DFD37D7986444%2Fimage-dimensions%2F742x490%3Fv%3D1.0%22%20width%3D%22742%22%20height%3D%22490%22%20role%3D%22button%22%20title%3D%22Ping_Wu_1-1612232836777.png%22%20alt%3D%22Ping_Wu_1-1612232836777.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CDIV%20class%3D%22ms-editor-squiggler%22%20style%3D%22color%3A%20initial%3B%20font%3A%20initial%3B%20font-feature-settings%3A%20initial%3B%20font-kerning%3A%20initial%3B%20font-optical-sizing%3A%20initial%3B%20font-variation-settings%3A%20initial%3B%20forced-color-adjust%3A%20initial%3B%20text-orientation%3A%20initial%3B%20text-rendering%3A%20initial%3B%20-webkit-font-smoothing%3A%20initial%3B%20-webkit-locale%3A%20initial%3B%20-webkit-text-orientation%3A%20initial%3B%20-webkit-writing-mode%3A%20initial%3B%20writing-mode%3A%20initial%3B%20zoom%3A%20initial%3B%20place-content%3A%20initial%3B%20place-items%3A%20initial%3B%20place-self%3A%20initial%3B%20alignment-baseline%3A%20initial%3B%20animation%3A%20initial%3B%20appearance%3A%20initial%3B%20aspect-ratio%3A%20initial%3B%20backdrop-filter%3A%20initial%3B%20backface-visibility%3A%20initial%3B%20background%3A%20initial%3B%20background-blend-mode%3A%20initial%3B%20baseline-shift%3A%20initial%3B%20block-size%3A%20initial%3B%20border-block%3A%20initial%3B%20border%3A%20initial%3B%20border-radius%3A%20initial%3B%20border-collapse%3A%20initial%3B%20border-inline%3A%20initial%3B%20inset%3A%20initial%3B%20box-shadow%3A%20initial%3B%20box-sizing%3A%20initial%3B%20break-after%3A%20initial%3B%20break-before%3A%20initial%3B%20break-inside%3A%20initial%3B%20buffered-rendering%3A%20initial%3B%20caption-side%3A%20initial%3B%20caret-color%3A%20initial%3B%20clear%3A%20initial%3B%20clip%3A%20initial%3B%20clip-path%3A%20initial%3B%20clip-rule%3A%20initial%3B%20color-interpolation%3A%20initial%3B%20color-interpolation-filters%3A%20initial%3B%20color-rendering%3A%20initial%3B%20color-scheme%3A%20initial%3B%20columns%3A%20initial%3B%20column-fill%3A%20initial%3B%20gap%3A%20initial%3B%20column-rule%3A%20initial%3B%20column-span%3A%20initial%3B%20contain%3A%20initial%3B%20contain-intrinsic-size%3A%20initial%3B%20content%3A%20initial%3B%20content-visibility%3A%20initial%3B%20counter-increment%3A%20initial%3B%20counter-reset%3A%20initial%3B%20counter-set%3A%20initial%3B%20cursor%3A%20initial%3B%20cx%3A%20initial%3B%20cy%3A%20initial%3B%20d%3A%20initial%3B%20display%3A%20block%3B%20dominant-baseline%3A%20initial%3B%20empty-cells%3A%20initial%3B%20fill%3A%20initial%3B%20fill-opacity%3A%20initial%3B%20fill-rule%3A%20initial%3B%20filter%3A%20initial%3B%20flex%3A%20initial%3B%20flex-flow%3A%20initial%3B%20float%3A%20initial%3B%20flood-color%3A%20initial%3B%20flood-opacity%3A%20initial%3B%20grid%3A%20initial%3B%20grid-area%3A%20initial%3B%20height%3A%20initial%3B%20hyphens%3A%20initial%3B%20image-orientation%3A%20initial%3B%20image-rendering%3A%20initial%3B%20inline-size%3A%20initial%3B%20inset-block%3A%20initial%3B%20inset-inline%3A%20initial%3B%20isolation%3A%20initial%3B%20letter-spacing%3A%20initial%3B%20lighting-color%3A%20initial%3B%20line-break%3A%20initial%3B%20list-style%3A%20initial%3B%20margin-block%3A%20initial%3B%20margin%3A%20initial%3B%20margin-inline%3A%20initial%3B%20marker%3A%20initial%3B%20mask%3A%20initial%3B%20mask-type%3A%20initial%3B%20max-block-size%3A%20initial%3B%20max-height%3A%20initial%3B%20max-inline-size%3A%20initial%3B%20max-width%3A%20initial%3B%20min-block-size%3A%20initial%3B%20min-height%3A%20initial%3B%20min-inline-size%3A%20initial%3B%20min-width%3A%20initial%3B%20mix-blend-mode%3A%20initial%3B%20object-fit%3A%20initial%3B%20object-position%3A%20initial%3B%20offset%3A%20initial%3B%20opacity%3A%20initial%3B%20order%3A%20initial%3B%20origin-trial-test-property%3A%20initial%3B%20orphans%3A%20initial%3B%20outline%3A%20initial%3B%20outline-offset%3A%20initial%3B%20overflow-anchor%3A%20initial%3B%20overflow-wrap%3A%20initial%3B%20overflow%3A%20initial%3B%20overscroll-behavior-block%3A%20initial%3B%20overscroll-behavior-inline%3A%20initial%3B%20overscroll-behavior%3A%20initial%3B%20padding-block%3A%20initial%3B%20padding%3A%20initial%3B%20padding-inline%3A%20initial%3B%20page%3A%20initial%3B%20page-orientation%3A%20initial%3B%20paint-order%3A%20initial%3B%20perspective%3A%20initial%3B%20perspective-origin%3A%20initial%3B%20pointer-events%3A%20initial%3B%20position%3A%20initial%3B%20quotes%3A%20initial%3B%20r%3A%20initial%3B%20resize%3A%20initial%3B%20ruby-position%3A%20initial%3B%20rx%3A%20initial%3B%20ry%3A%20initial%3B%20scroll-behavior%3A%20initial%3B%20scroll-margin-block%3A%20initial%3B%20scroll-margin%3A%20initial%3B%20scroll-margin-inline%3A%20initial%3B%20scroll-padding-block%3A%20initial%3B%20scroll-padding%3A%20initial%3B%20scroll-padding-inline%3A%20initial%3B%20scroll-snap-align%3A%20initial%3B%20scroll-snap-stop%3A%20initial%3B%20scroll-snap-type%3A%20initial%3B%20shape-image-threshold%3A%20initial%3B%20shape-margin%3A%20initial%3B%20shape-outside%3A%20initial%3B%20shape-rendering%3A%20initial%3B%20size%3A%20initial%3B%20speak%3A%20initial%3B%20stop-color%3A%20initial%3B%20stop-opacity%3A%20initial%3B%20stroke%3A%20initial%3B%20stroke-dasharray%3A%20initial%3B%20stroke-dashoffset%3A%20initial%3B%20stroke-linecap%3A%20initial%3B%20stroke-linejoin%3A%20initial%3B%20stroke-miterlimit%3A%20initial%3B%20stroke-opacity%3A%20initial%3B%20stroke-width%3A%20initial%3B%20tab-size%3A%20initial%3B%20table-layout%3A%20initial%3B%20text-align%3A%20initial%3B%20text-align-last%3A%20initial%3B%20text-anchor%3A%20initial%3B%20text-combine-upright%3A%20initial%3B%20text-decoration%3A%20initial%3B%20text-decoration-skip-ink%3A%20initial%3B%20text-indent%3A%20initial%3B%20text-overflow%3A%20initial%3B%20text-shadow%3A%20initial%3B%20text-size-adjust%3A%20initial%3B%20text-transform%3A%20initial%3B%20text-underline-offset%3A%20initial%3B%20text-underline-position%3A%20initial%3B%20touch-action%3A%20initial%3B%20transform%3A%20initial%3B%20transform-box%3A%20initial%3B%20transform-origin%3A%20initial%3B%20transform-style%3A%20initial%3B%20transition%3A%20initial%3B%20user-select%3A%20initial%3B%20vector-effect%3A%20initial%3B%20vertical-align%3A%20initial%3B%20visibility%3A%20initial%3B%20-webkit-app-region%3A%20initial%3B%20border-spacing%3A%20initial%3B%20-webkit-border-image%3A%20initial%3B%20-webkit-box-align%3A%20initial%3B%20-webkit-box-decoration-break%3A%20initial%3B%20-webkit-box-direction%3A%20initial%3B%20-webkit-box-flex%3A%20initial%3B%20-webkit-box-ordinal-group%3A%20initial%3B%20-webkit-box-orient%3A%20initial%3B%20-webkit-box-pack%3A%20initial%3B%20-webkit-box-reflect%3A%20initial%3B%20-webkit-highlight%3A%20initial%3B%20-webkit-hyphenate-character%3A%20initial%3B%20-webkit-line-break%3A%20initial%3B%20-webkit-line-clamp%3A%20initial%3B%20-webkit-mask-box-image%3A%20initial%3B%20-webkit-mask%3A%20initial%3B%20-webkit-mask-composite%3A%20initial%3B%20-webkit-perspective-origin-x%3A%20initial%3B%20-webkit-perspective-origin-y%3A%20initial%3B%20-webkit-print-color-adjust%3A%20initial%3B%20-webkit-rtl-ordering%3A%20initial%3B%20-webkit-ruby-position%3A%20initial%3B%20-webkit-tap-highlight-color%3A%20initial%3B%20-webkit-text-combine%3A%20initial%3B%20-webkit-text-decorations-in-effect%3A%20initial%3B%20-webkit-text-emphasis%3A%20initial%3B%20-webkit-text-emphasis-position%3A%20initial%3B%20-webkit-text-fill-color%3A%20initial%3B%20-webkit-text-security%3A%20initial%3B%20-webkit-text-stroke%3A%20initial%3B%20-webkit-transform-origin-x%3A%20initial%3B%20-webkit-transform-origin-y%3A%20initial%3B%20-webkit-transform-origin-z%3A%20initial%3B%20-webkit-user-drag%3A%20initial%3B%20-webkit-user-modify%3A%20initial%3B%20white-space%3A%20initial%3B%20widows%3A%20initial%3B%20width%3A%20initial%3B%20will-change%3A%20initial%3B%20word-break%3A%20initial%3B%20word-spacing%3A%20initial%3B%20x%3A%20initial%3B%20y%3A%20initial%3B%20z-index%3A%20initial%3B%22%3E%26nbsp%3B%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2110648%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20blog%20is%20about%20the%20issue%20that%20stateless%2Fstateful%20service%20cannot%20be%20started%20in%20service%20fabric%20cluster.%3C%2FP%3E%0A%3CP%3EIt%20includes%20below%20items%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EStateless%20and%20stateful%20service%20life%20cycle%3C%2FLI%3E%0A%3CLI%3EEvents%20and%26nbsp%3Bcancellation%20token%26nbsp%3B%3C%2FLI%3E%0A%3CLI%3ETroubleshooting%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22sf.jpeg%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F251215i157132A252760E09%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22sf.jpeg%22%20alt%3D%22sf.jpeg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-TEASER%3E
Microsoft

This blog is about the issue that stateless/stateful service cannot be started in service fabric cluster.

It provides a overview about reliable service life cycle, details of some important events like CreateServiceInstanceListener, RunAsync and troubleshooting steps to identify the issue event and exceptions details.


Stateless And Stateful Service Life Cycle

 

The lifecycle of a stateless service is straightforward. Here's the order of events:

   1. The service is constructed.

   2. Then, in parallel, two things happen:

        * StatelessService.CreateServiceInstanceListeners() is invoked and any returned listeners are opened. 

           ICommunicationListener.OpenAsync() is called on each listener.

        * The service's StatelessService.RunAsync() method is called.

   3. If present, the service's StatelessService.OnOpenAsync() method is called.

       This call is an uncommon override, but it is available. Extended service initialization tasks can be started at this time.

 

Stateful services have a similar pattern to stateless services, with a few changes. For starting up a stateful service, the order of events is as follows:

   1. The service is constructed.

   2. StatefulServiceBase.OnOpenAsync() is called. This call is not commonly overridden in the service.

       The following things happen in parallel:

        *  StatefulServiceBase.CreateServiceReplicaListeners() is invoked.

        *  If the service is a Primary service, all returned listeners are opened. 

           ICommunicationListener.OpenAsync() is called on each listener.

        *  If the service is a Secondary service, only those listeners marked as ListenOnSecondary = true are opened.

           Having listeners that are open on secondaries is less common.

        *  If the service is currently a Primary, the service's StatefulServiceBase.RunAsync() method is called.

   3. After all the replica listener's OpenAsync() calls finish and RunAsync() is called, StatefulServiceBase.OnChangeRoleAsync() is called.

      This call is not commonly overridden in the service.


Events and Cancellation Token

 

CreateServiceInstanceListener is to supply the communication listeners for the service instance, it is normally override in stateless service like using Kestrel , https and so on.

 

RunAsync() is executed in its own task. Note that in the code snippet above, we jumped right into a while loop. There is no need to schedule a separate task for your workload. Cancellation of your workload is a cooperative effort orchestrated by the provided cancellation token. The system will wait for your task to end (by successful completion, cancellation, or fault) before it moves on. It is important to honor the cancellation token, finish any work, and exit RunAsync() as quickly as possible when the system requests cancellation. It will be triggered for stateful primary replica or all stateless instances and normally override in stateful service.

 

Cancellation token is provided to coordinate when your service instance needs to be closed. In Service Fabric, this open/close cycle of a service instance can occur many times over the lifetime of the service as a whole. This can happen for various reasons, including:

        *  The system moves your service instances for resource balancing.

        *  Faults occur in your code

        *  The application or system is upgraded.

        *  The underlying hardware experiences an outage.


Troubleshooting

 

Below are some ways to get more insight:

   1. RDP to service fabric node. (primary replica node if it is stateful service)

   2. Check Application event logs for any exception if no exceptions go to step 3.

   3. Check if the port is occupied by the other services.

For TCP:Get-Process -Id (Get-NetTCPConnection -LocalPort YourPortNumberHere).OwningProcess
For UDP:Get-Process -Id (Get-NetUDPEndpoint -LocalPort YourPortNumberHere).OwningProcess

   4. For non-prod environment, remote debug would be a good way to get more insight 

      Please ref more details via Debug your application in Visual Studio

   5. List underlying exceptions and capture dump via procmon

      Start-bitstransfer "https://download.sysinternals.com/files/Procdump.zip"

      procdump.exe -accepteula -e 1 -f ""  -w "processname"

Ping_Wu_0-1612232836718.png

   6. Then capture the dump for specific exception.

       procdump.exe -ma -e 1 -f "NullReferenceException" -w "processname"

    7. Use Windbg, Debugdiag to get details about exception like method call stack.

Ping_Wu_1-1612232836777.png