invalid deletion count: 41 vs docCount=29 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_d5")))

[PeterAtValantic]

Hi,

As a followup from umbraco/Umbraco-CMS#16163
(I will answer below your suggestions)

• Ensure that the Examine assembly deployed is in fact the latest version

  • I deploy from Visual Studio and yesterday I have manually removed all dll files and redeployed it all.

• Provide as much information as possible and potentially steps to replicate

  • Stack traces

at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)
Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)
Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)
Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryCreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock, Directory& directory)
Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0()
System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) System.Lazy1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
System.Lazy1.CreateValue() System.Lazy1.get_Value()
Examine.Lucene.Providers.LuceneIndex.GetLuceneDirectory()
Examine.Lucene.Providers.LuceneIndex.IndexExistsImpl()
Examine.Lucene.Providers.LuceneIndex.IndexExists()
Examine.Lucene.Providers.LuceneIndex.EnsureIndex(Boolean forceOverwrite)
Examine.Lucene.Providers.LuceneIndex.get_IndexWriter()
Examine.Lucene.Providers.LuceneIndex.CreateSearcher()
System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) System.Lazy1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
System.Lazy1.CreateValue() System.Lazy1.get_Value()
Examine.Lucene.Providers.LuceneIndex.get_Searcher()
SuperBold.Framework.UmbracoCustom.Services.SiteService.GetInitialQuery(String indexName, Boolean allowLeadingWildcard, Boolean onlyVisible)
AspNetCore.Views_Partials_BlockList_Components_itemMainNavigation.ExecuteAsync() in C:\home\site\wwwroot\Views\Partials\BlockList\Components\itemMainNavigation.cshtml:line 56
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
AspNetCore.Views_Partials_Blocklist_custom.ExecuteAsync() in C:\home\site\wwwroot\Views\Partials\Blocklist\custom.cshtml:line 18
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
AspNetCore.Views_Partials_Menu.ExecuteAsync() in C:\home\site\wwwroot\Views\Partials\Menu.cshtml:line 93
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.RenderPartialCoreAsync(String partialViewName, Object model, ViewDataDictionary viewData, TextWriter writer)
Microsoft.AspNetCore.Mvc.ViewFeatures.HtmlHelper.PartialAsync(String partialViewName, Object model, ViewDataDictionary viewData)
AspNetCore.Views_Master.<>c__DisplayClass34_0.<<ExecuteAsync>b__1>d.MoveNext() in C:\home\site\wwwroot\Views\Master.cshtml:line 103--- End of stack trace from previous location ---
Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.SetOutputContentAsync()
AspNetCore.Views_Master.ExecuteAsync() in C:\home\site\wwwroot\Views\Master.cshtml:line 99
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderLayoutAsync(ViewContext context, ViewBufferTextWriter bodyWriter)
Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable1 statusCode) Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable1 statusCode)
Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ActionContext actionContext, IView view, ViewDataDictionary viewData, ITempDataDictionary tempData, String contentType, Nullable`1 statusCode)
Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()--- End of stack trace from previous location ---
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()--- End of stack trace from previous location ---
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
Umbraco.Cms.Web.Common.Middleware.BasicAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()--- End of stack trace from previous location ---
Umbraco.Cms.Web.BackOffice.Middleware.BackOfficeExternalLoginProviderErrorMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()--- End of stack trace from previous location ---
Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 112
Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()--- End of stack trace from previous location ---
Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()--- End of stack trace from previous location ---
Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()--- End of stack trace from previous location ---
SeoToolkit.Umbraco.Redirects.Core.Middleware.RedirectMiddleware.Invoke(HttpContext context)
SeoToolkit.Umbraco.RobotsTxt.Core.Middleware.RobotsTxtMiddleware.Invoke(HttpContext context)
SeoToolkit.Umbraco.Sitemap.Core.Middleware.SitemapMiddleware.Invoke(HttpContext context, ISitemapGenerator sitemapGenerator, ISitemapIndexGenerator sitemapIndexGenerator)
SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
Umbraco.Forms.Web.HttpModules.ProtectFormUploadRequestsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()--- End of stack trace from previous location ---
Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task).

• What indexes get corrupt, 1, 2, all, etc...
  • most of the time: ExternalIndex, but I have also seen that the other indexes are getting corrupt as well: CustomExternalIndex, InternalIndex
• Any relevant config
  • depends what part of the config you find relevant, but I have this:

"Umbraco": {
"CMS": {
"RuntimeMinification": {
"UseInMemoryCache": true,
},
"Global": {
"MainDomLock": "FileSystemMainDomLock"
},
"Hosting": {
"LocalTempStorageLocation": "EnvironmentTemp"
},
"Examine": {
"LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory"
}
}

I am on Umbraco 13.5.3 (latest version of Umbraco 13) an the latest version of Examine.

It occurs at random, but also often I have deployed from Visual Studio. First I stop the application and then I press publish and then I start the application and then when I visit the web application I get a 502 error and I see this error in the log.

To replicate:
I am on an Azure App Service. Not Loadbalanced.

These are my Publsih settings:

• Configuration: Debug
• Target Framework: net8.0
• Deployment Mode: Framework-dependent
• Target Runtime: Portable
• File Publish Options: Remove additional files at destination -> unchecked
• Other checkboxes of other settings are also unchecked

Server configuration:

• Web app
  • Publishing model: Code
  • Runtime Stack: Dotnetcore
• Hosting
  • Plan Type: App Service plan
  • Operating System: Windows
• Platform settings
  • Platform: 32 bit
  • Managed pipeline version: Integrated

Please let me know what other settings you need to know.

[Shazwazza]

@PeterAtValantic any chance you have the log outputs for these statements?

• Examine/src/Examine.Lucene/Directories/SyncedFileSystemDirectoryFactory.cs

  Line 229 in 6ec92e4

  _logger.LogError(ex, "{IndexName} index is corrupt, a new one will be created", indexName);

• Examine/src/Examine.Lucene/Directories/SyncedFileSystemDirectoryFactory.cs

  Lines 284 to 325 in 6ec92e4

  	_logger.LogDebug("{IndexName} health check report {IndexReport}", indexName, writer.ToString());
  	if (status.MissingSegments)
  	{
  	_logger.LogWarning("{IndexName} index is missing segments, it will be deleted.", indexName);
  	result = CreateResult.MissingSegments;
  	}
  	else if (!status.Clean)
  	{
  	_logger.LogWarning("Checked main index {IndexName} and it is not clean.", indexName);
  	result = CreateResult.NotClean;
  	if (doFix)
  	{
  	_logger.LogWarning("Attempting to fix {IndexName}. {DocumentsLost} documents will be lost.", indexName, status.TotLoseDocCount);
  	try
  	{
  	checker.FixIndex(status);
  	status = checker.DoCheckIndex();
  	if (!status.Clean)
  	{
  	_logger.LogError("{IndexName} index could not be fixed, it will be deleted.", indexName);
  	result |= CreateResult.NotFixed;
  	}
  	else
  	{
  	_logger.LogInformation("Index {IndexName} fixed. {DocumentsLost} documents were lost.", indexName, status.TotLoseDocCount);
  	result |= CreateResult.Fixed;
  	}
  	}
  	catch (Exception ex)
  	{
  	_logger.LogError(ex, "{IndexName} index could not be fixed, it will be deleted.", indexName);
  	result |= CreateResult.ExceptionNotFixed;
  	}
  	}
  	}
  	else
  	{
  	_logger.LogInformation("Checked main index {IndexName} and it is clean.", indexName);

[readingdancer]

@Shazwazza - I hope you are well?

We are still getting this error on Umbraco Cloud, we are running your latest versions of Examine, but still no joy. I have just dropped an email to your hotmail email account, I hope that still works :)

Chris

[PeterAtValantic]

@Shazwazza Yes, I have those:

Timestamp 2025-02-07T14:22:08.7675053+00:00
@MessageTemplate Checked main index {IndexName} and it is clean.
IndexName InternalIndex
SourceContext
ActionId 70f783a3-fe27-4c79-a0d0-18e69cf2551e
ActionName Umbraco.Cms.Web.BackOffice.Controllers.EntityController.SearchAll (Umbraco.Web.BackOffice)
RequestId 40000022-0000-f500-b63f-84710c7967bb
RequestPath /umbraco/backoffice/umbracoapi/entity/SearchAll
ProcessId 6260
ProcessName w3wp
ThreadId 25
ApplicationId ea6c9907b5dfb977abd279e5dd977d85f29f78cb
MachineName
Log4NetLevel INFO
HttpRequestId 8efd5241-14fc-48ee-b33c-8bce8fef87e5
HttpRequestNumber
HttpSessionId e28e401d-9ed3-e91a-1aa4-3a4f1e7c4403

Feb 7, 2025 3:22:08 PM Error WN1MDWK0003OG "InternalIndex" index is corrupt, a new one will be created
Lucene.Net.Index.CorruptIndexException: invalid deletion count: 79 vs docCount=18 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\InternalIndex\segments_ff")))
at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)
at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)
Timestamp 2025-02-07T14:22:08.835312+00:00
@MessageTemplate {IndexName} index is corrupt, a new one will be created
IndexName InternalIndex
SourceContext
ActionId 70f783a3-fe27-4c79-a0d0-18e69cf2551e
ActionName Umbraco.Cms.Web.BackOffice.Controllers.EntityController.SearchAll (Umbraco.Web.BackOffice)
RequestId 40000022-0000-f500-b63f-84710c7967bb
RequestPath /umbraco/backoffice/umbracoapi/entity/SearchAll
ProcessId 6260
ProcessName w3wp
ThreadId 25
ApplicationId ea6c9907b5dfb977abd279e5dd977d85f29f78cb
MachineName
Log4NetLevel ERROR
HttpRequestId 8efd5241-14fc-48ee-b33c-8bce8fef87e5
HttpRequestNumber
HttpSessionId e28e401d-9ed3-e91a-1aa4-3a4f1e7c4403

Feb 4, 2025 11:07:37 AM Warning WN1MDWK0003IF "ExternalIndex" index is missing segments, it will be deleted.
Timestamp 2025-02-04T10:07:37.8782717+00:00
@MessageTemplate {IndexName} index is missing segments, it will be deleted.
IndexName ExternalIndex
SourceContext
ActionId e446cc63-263f-4761-b480-fe0ca2bb8187
ActionName Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)
RequestId 40000038-0000-f100-b63f-84710c7967bb
RequestPath /
ProcessId 4876
ProcessName w3wp
ThreadId 14
ApplicationId ea6c9907b5dfb977abd279e5dd977d85f29f78cb
MachineName
Log4NetLevel WARN
HttpRequestId d9154b2d-3cc5-43f2-a23f-097f23078b05
HttpRequestNumber
HttpSessionId a2db047a-2211-69fc-3d47-57071bf952f8

I also found this error, but I am not sure if it caused by a deployment or not.

System.ObjectDisposedException: this Directory is disposed.
Object name: 'Lucene.Net.Store.NRTCachingDirectory'.
at Lucene.Net.Store.NRTCachingDirectory.OpenInput(String name, IOContext context)
at Lucene.Net.Replicator.LocalReplicator.ObtainFile(String sessionId, String source, String fileName)
at Lucene.Net.Replicator.ReplicationClient.DoUpdate()
at Lucene.Net.Replicator.ReplicationClient.ReplicationThread.Run()
Timestamp 2025-02-04T09:41:11.5131309+00:00
@MessageTemplate Index replication error occurred
SourceContext
ActionId 09891039-f317-4981-b350-b9cba37f0d2f
ActionName Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)
RequestId 40000077-0003-f900-b63f-84710c7967bb
RequestPath /
ProcessId 7752
ProcessName w3wp
ThreadId 34
ApplicationId ea6c9907b5dfb977abd279e5dd977d85f29f78cb
MachineName
Log4NetLevel ERROR
HttpRequestId be5b3aa4-da88-4855-b0d0-5cb36b5649f2
HttpRequestNumber
HttpSessionId 0

and also this one:

Feb 4, 2025 11:08:30 AM Error wn1mdwk00045G Index replication error occurred
System.IO.FileNotFoundException: File not found: C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex_es.fnm
at Lucene.Net.Support.IO.WindowsFsyncSupport.HandleWrapper..ctor(String path, Boolean isDir)
at Lucene.Net.Support.IO.WindowsFsyncSupport.Fsync(String path, Boolean isDir)
at Lucene.Net.Util.IOUtils.Fsync(String fileToSync, Boolean isDir)
at Lucene.Net.Store.FSDirectory.Fsync(String name)
at Lucene.Net.Store.FSDirectory.Sync(ICollection1 names) at Lucene.Net.Store.NRTCachingDirectory.Sync(ICollection1 fileNames)
at Lucene.Net.Replicator.IndexReplicationHandler.RevisionReady(String version, IDictionary2 revisionFiles, IDictionary2 copiedFiles, IDictionary`2 sourceDirectory)
at Lucene.Net.Replicator.ReplicationClient.DoUpdate()
at Lucene.Net.Replicator.ReplicationClient.ReplicationThread.Run()
Timestamp 2025-02-04T10:08:30.8081664+00:00
@MessageTemplate Index replication error occurred
SourceContext
ActionId 544ce548-c25e-43ea-8a57-de24df3e77ec
ActionName Umbraco.Cms.Web.Common.Controllers.RenderController.Index (Umbraco.Web.Common)
RequestId 40000003-0001-f500-b63f-84710c7967bb
RequestPath /
ProcessId 4968
ProcessName w3wp
ThreadId 33
ApplicationId ea6c9907b5dfb977abd279e5dd977d85f29f78cb
MachineName
Log4NetLevel ERROR
HttpRequestId 7714cad2-b83d-4ab6-968f-7b3785a7ce77
HttpRequestNumber
HttpSessionId 0

If you need more logs (I can imagine the first logs before and right after the index occurred would be interesting), please let me know

[Shazwazza]

@PeterAtValantic according to the logs, Examine should take steps to basically remove the index so the site can actually start/operate and then index rebuilding will occur. I know you are seeing a lot of these logs, but does your site remain operable and you can access Examine management dashboard?

The last 2 logs are more interesting, looks like timing issues with app shutdown, disposing directories while lucene's background thread is trying to access these instances.

[Shazwazza]

I've found potentially a source for those logs and will investigate further - this is a good thing and hopefully get further to the bottom of this issue. Just need to find a way to replicate in tests.

[PeterAtValantic]

@PeterAtValantic according to the logs, Examine should take steps to basically remove the index so the site can actually start/operate and then index rebuilding will occur. I know you are seeing a lot of these logs, but does your site remain operable and you can access Examine management dashboard?

No, when we have the index issue the whole site is down due to that error. (I think that is because we use Examine on lots of pages)
And we have upgraded umbraco to version 13.6.0 where they have made a fix that you can access the Examine management dashboard, but before it was impossible to access that too. We have not had the issue since the upgrade, but I think that is more luck then wisdom. We are still monitoring it closely.

[Shazwazza]

Hi @PeterAtValantic + @readingdancer

I've published some changes to hopefully resolve the issue. The recent Lucene.NET upgrade seemingly worked fine but there were some additional nuances that weren't taken care of. I also discovered some other issues relating to disposal.

If you are able to test this pre-release, that would be very helpful, you can find the Nuget artifacts here https://github.com/Shazwazza/Examine/actions/runs/13549030183#artifacts

examine-nuget-3.6.1-1

[readingdancer]

Hi @Shazwazza - Thanks, I have just installed it locally, after doing a few tests I will push it up to the cloud and we will see how things go. Thank you for your perseverance to try and get to the bottom of this issue.

[Shazwazza]

@readingdancer thanks for the assistance and patience. Looking forward to hearing the results.

[readingdancer]

@PeterAtValantic

I know you are running on your own Azure hosting and Umbraco 13+ whereas my client site is on Umbraco 10, so this might not be relevant for you, however just in case, this is how I have deployed this change to Umbraco Cloud:

1. I copied the contents of the Examine ZIP file linked above into a folder within my solution
2. I added a source to my project NuGet.config file to allow it to find this folder:

    <add key="Local" value="src/Examine.PreRelease" />

3. After I used Visual Studio NuGet Package Manager to update Examine (pulling the files from the above configured folder)

4. I received a couple of errors, so need to add the following to my web project:

    <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" Version="8.0.0" />
    <PackageReference Include="System.Formats.Asn1" Version="8.0.1" />

After this, it has deployed cleanly to Umbraco Cloud and I've pushed it though to my Client's live server, so now, fingers crossed :)

Thanks again @Shazwazza

[Kull1z]

We have the same issue on two of our biggest sites with a lot of content. I'm following this thread and hope the fix works. Will you update us soon, @readingdancer?

Thanks in advance!

[alexsharp1]

We are currently experiencing the same issue on a site running Umbraco Version 13.6.0. Until now, we have not encountered any significant Examine/Lucene issues. Our Azure configuration appears to be correct, and this is occurring on our CMS single instance with the "SyncedTempFileSystemDirectoryFactory" setting enabled. The issue also keeps repeating, causing excessive log spam.

Customer feedback: After creating a news item in the CMS, the system duplicated it. When the customer attempted to delete the duplicate, this issue appeared.

{"@t":"2025-02-28T10:51:44.6254280Z","@mt":"Failed (will repeat).","@l":"Error","@x":"System.AggregateException: One or more errors occurred. (Exceptions were thrown by listed actions. (invalid deletion count: 5 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=\"C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_m2\"))))) ---> System.AggregateException: Exceptions were thrown by listed actions. (invalid deletion count: 5 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=\"C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_m2\")))) ---> Lucene.Net.Index.CorruptIndexException: invalid deletion count: 5 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=\"C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_m2\"))) at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName) at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists) at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf) at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode) at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter) at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryCreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock, Directory& directory) at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock) at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)\r\n--- End of stack trace from previous location ---
at System.Lazy1.CreateValue() at Examine.Lucene.Providers.LuceneIndex.GetLuceneDirectory() at Examine.Lucene.Providers.LuceneIndex.IndexExistsImpl() at Examine.Lucene.Providers.LuceneIndex.IndexExists() at Examine.Lucene.Providers.LuceneIndex.EnsureIndex(Boolean forceOverwrite) at Examine.Lucene.Providers.LuceneIndex.get_IndexWriter() at Examine.Lucene.Providers.LuceneIndex.CreateSearcher() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)\r\n--- End of stack trace from previous location ---
at System.Lazy1.CreateValue() at Umbraco.Cms.Infrastructure.Examine.UmbracoContentIndex.PerformDeleteFromIndex(IEnumerable1 itemIds, Action1 onComplete) at Umbraco.Cms.Infrastructure.Examine.ExamineUmbracoIndexingHandler.DeferredDeleteIndex.Execute(ExamineUmbracoIndexingHandler examineUmbracoIndexingHandler, Int32 id, Boolean keepIfUnpublished) at Umbraco.Cms.Infrastructure.Examine.DeferredActions.Execute() at Umbraco.Cms.Core.Scoping.ScopeContext.EnlistedObject1.Execute(Boolean completed)
at Umbraco.Cms.Core.Scoping.ScopeContext.ScopeExit(Boolean completed)
--- End of inner exception stack trace ---
at Umbraco.Cms.Core.Scoping.ScopeContext.ScopeExit(Boolean completed)
at Umbraco.Cms.Infrastructure.Scoping.Scope.<>c__DisplayClass56_0.g__HandleScopeContext|0()
at Umbraco.Cms.Infrastructure.Scoping.Scope.TryFinally(Action[] actions)
--- End of inner exception stack trace ---
at Umbraco.Cms.Infrastructure.Scoping.Scope.TryFinally(Action[] actions)
at Umbraco.Cms.Infrastructure.Scoping.Scope.Dispose()
at Umbraco.Cms.Infrastructure.Services.CacheInstructionService.ProcessInstructions(CacheRefresherCollection cacheRefreshers, ServerRole serverRole, CancellationToken cancellationToken, String localIdentity, DateTime lastPruned, Int32 lastId)
at Umbraco.Cms.Infrastructure.Sync.DatabaseServerMessenger.Sync()
at Umbraco.Cms.Infrastructure.BackgroundJobs.Jobs.ServerRegistration.InstructionProcessJob.RunJobAsync()","SourceContext":"Umbraco.Cms.Infrastructure.BackgroundJobs.Jobs.ServerRegistration.InstructionProcessJob","Log4NetLevel":"ERROR"}
`