Terminating Running Sitecore Publish Jobs: A Comprehensive Solution

Reference

https://github.com/mikeedwards83/Glass.PublishViewer

Sitecore Version

Sitecore 10.4

Introduction

In Sitecore environments with large content databases and multiple publishing targets, publishing operations can sometimes take a very long time to complete. When a publishing job gets stuck or needs to be stopped for any reason, the standard approach has often been to restart the Content Management (CM) server - a disruptive and inefficient solution.

While Sitecore PowerShell Extensions (SPE) provides functionality to view and cancel queued publishing jobs, it doesn't offer a built-in way to terminate jobs that are already running. This post presents a comprehensive solution for safely terminating running publish jobs in Sitecore without restarting the server.

Understanding the Sitecore Publishing Architecture

Before diving into the solution, it's important to understand the key components of Sitecore's publishing architecture:

Key Components

1. PublishContext: The central context for the entire publishing operation

   • Contains the publishing queue of items to be processed
   • Maintains statistics about the publishing operation
   • Tracks processed items to prevent duplication

2. PublishStatus: Tracks progress and state of the publishing job

   • Contains methods like SetState() to update the job state
   • Provides a way to add status messages
   • Serves as a bridge between the Job system and the publishing system

3. ProcessQueue Processor: Core component that processes publishing items

   • Iterates through queued items for publishing
   • Contains error handling mechanisms that continue processing despite individual item failures
   • Uses a retry mechanism for failed items

4. Job Management System:

   • DefaultJobManager: Manages all tasks' lifecycle (queued, running, finished)
   • JobCollection: Stores jobs in different states
   • Controls state transitions through methods like FinishJob and RunJob

Publishing Pipeline Flow

<publish help="Processors should derive from Sitecore.Publishing.Pipelines.Publish.PublishProcessor">
    <!--  This processor overrides the DisableDatabaseCaches and the MaxDegreeOfParallelism properties of the PublishContext class.

        The DisableDatabaseCaches property of the PublishContext is overridden and set to true if child items are being published,
        including when you perform a full site publish.
        If only a single item is being published, the property is set to false.
        If you disable this processor, the publish context uses the value of the Publishing.DisableDatabaseCaches setting for all the
        publishing operations.

        If only a single item is being published, the MaxDegreeOfParallelism property of the PublishContext is overridden to 1.
        This disables parallel publishing for single item publishing operations.  -->
    <processor type="Sitecore.Publishing.Pipelines.Publish.OverridePublishContext, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.Publish.AddLanguagesToQueue, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.Publish.AddItemsToQueue, Sitecore.Kernel"/>
    <!-- 
        This handler manually injects arbitrary items into the publish queue so that the next publish to occur will cause the injected items to get published. 
        See http://www.velir.com/blog/index.php/2013/11/22/how-to-create-a-custom-publish-queue-in-sitecore/ et. al.
        -->
    <processor type="Unicorn.Publishing.ManualPublishQueueHandler, Unicorn" patch:source="Unicorn.AutoPublish.config"/>
    <!--  Extending publish pipeline to always add bucket folders to the queue when a bucketed item is being published   -->
    <processor type="Sitecore.Buckets.Pipelines.Publish.AddBucketFoldersToQueue, Sitecore.Buckets" patch:source="Sitecore.Buckets.config"/>
    <processor type="Sitecore.Publishing.Pipelines.Publish.RaiseQueuedEvents, Sitecore.Kernel" resolve="true">
        <param type="Sitecore.Abstractions.BaseEventQueueProvider, Sitecore.Kernel" resolve="true"/>
    </processor>
    <processor type="Sitecore.Publishing.Pipelines.Publish.ProcessQueue, Sitecore.Kernel"/>
    <!-- 
        Move the Unicorn publish queue after AddBucketFoldersToQueue to avoid issues with bucket publishing.
        -->
    <processor type="Sidekick.AuditLog.Pipelines.Publish.AuditPublish, Sidekick.AuditLog" patch:source="Sidekick.AuditLog.config"/>
</publish>
<publishItem help="Processors should derive from Sitecore.Publishing.Pipelines.PublishItem.PublishItemProcessor">
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.RaiseProcessingEvent, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.CheckVirtualItem, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.CheckSecurity, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.DetermineAction, Sitecore.Kernel"/>
    <processor type="Sitecore.Buckets.Pipelines.PublishItem.ProcessActionForBucketStructure, Sitecore.Buckets" patch:source="Sitecore.Buckets.config"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.MoveItems, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.PerformAction, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.AddItemReferences, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.RemoveUnknownChildren, Sitecore.Kernel"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.RaiseProcessedEvent, Sitecore.Kernel" runIfAborted="true"/>
    <processor type="Sitecore.Publishing.Pipelines.PublishItem.UpdateStatistics, Sitecore.Kernel" runIfAborted="true">
        <traceToLog>false</traceToLog>
    </processor>
</publishItem>

The publishing process in Sitecore involves two main pipelines:

1. publish pipeline: The outer pipeline that coordinates the entire publishing operation

   • ProcessQueue is a key processor in this pipeline that manages the queue of items to be published

2. publishItem pipeline: Executed for each individual item being published

   • Handles individual item processing logic
   • Determines whether an item should be created, updated, or deleted

The Challenge of Terminating Running Publish Jobs

The primary challenge in terminating running publish jobs lies in the design of Sitecore's ProcessQueue class:

protected virtual void ProcessEntries(PublishContext context)
{
    List<IEnumerable<PublishingCandidate>> publishingCandidatesList = new List<IEnumerable<PublishingCandidate>>();
    foreach (IEnumerable<PublishingCandidate> source in context.Queue)
    {
        PublishingCandidate[] array = source.ToArray<PublishingCandidate>();
        try
        {
            this.ProcessEntries((IEnumerable<PublishingCandidate>) array, context);
        }
        catch (Exception ex)
        {
            PublishingLog.Warn("An error during Publish Pipeline Process Queue execution. Will be tried to republish after the main part of publish will be done.", ex);
            foreach (PublishingCandidate publishingCandidate in array)
                context.ProcessedPublishingCandidates.TryRemove(new ProcessedPublishingCandidate(publishingCandidate.ItemId, publishingCandidate.Language), out byte _);
            publishingCandidatesList.Add((IEnumerable<PublishingCandidate>) array);
        }
    }
    foreach (IEnumerable<PublishingCandidate> entries in publishingCandidatesList)
        this.ProcessEntries(entries, context);
}

This code reveals that:

1. Exceptions during processing don't stop the entire process
2. Failed items are collected and retried after the main processing is complete
3. There's no built-in mechanism to abort the entire process

Previous Attempted Solutions

A common approach has been to use a "Terminator" processor in the publishItem pipeline that throws an exception when a termination flag is set:

public class Terminator : PublishItemProcessor
{
    public override void Process(PublishItemContext context)
    {
        PublishJobManager.Instance.CheckTermination();
    }
}

// In PublishJobManager:
public void CheckTermination()
{
    if (_terminateFlag)
    {
        throw new PublishTerminatedException("Publishing process has been terminated by user.");
    }
}

However, this approach has a key issue: the exception only terminates the current item's processing, but due to the error handling in ProcessQueue, the publishing process continues with the next batch of items. This is why many implementations required clicking "terminate" twice - the first click would set the flag but the process would continue, and only with repeated exceptions would the process eventually fail.

Our Comprehensive Solution

Our solution takes a more direct approach by replacing the core ProcessQueue processor with a custom implementation that explicitly checks for termination flags at critical points in the process.

The Solution Components

1. CustomProcessQueue: Replaces the standard ProcessQueue processor
2. PublishJobManager: Manages termination flags and job tracking
3. Terminator: An additional safety mechanism in the publishItem pipeline
4. PowerShell Script: For triggering termination with confirmation

Implementation Details

1. Terminable Process Queue

public class TerminableProcessQueue : Sitecore.Publishing.Pipelines.Publish.ProcessQueue
{
    public override void Process(Sitecore.Publishing.Pipelines.Publish.PublishContext context)
    {
        Assert.ArgumentNotNull(context, nameof(context));
        PublishingLog.Debug("[Publishing] MaxDegreeOfParallelism: " + context.MaxDegreeOfParallelism);
        PublishingLog.Debug("[Publishing] DisableDatabaseCaches: " + context.DisableDatabaseCaches.ToString());

        using (new Timer("[Publishing] - ProcessQueue"))
        {
            try
            {
                Switcher<bool, DisableCachePopulationSwitcher>.Enter(context.DisableDatabaseCaches);

                // Check before processing if termination was requested
                if (ShouldTerminatePublish())
                {
                    PublishingLog.Info("[Publish Termination] Termination flag detected before queue processing started.");
                    TerminatePublish(context);
                    return;
                }

                this.ProcessEntries(context);

                // Check again if termination was requested during processing
                if (ShouldTerminatePublish())
                {
                    PublishingLog.Info("[Publish Termination] Termination flag detected during queue processing.");
                    TerminatePublish(context);
                    return;
                }

                this.ProcessEntries(context.DeleteCandidates, context);
            }
            catch (PublishTerminatedException ex)
            {
                PublishingLog.Info("[Publish Termination] Publish process was terminated: " + ex.Message);
                // Don't rethrow - we want to gracefully stop
            }
            catch (Exception ex)
            {
                PublishingLog.Error("An error during Publish Pipeline Process Queue execution.", ex);
                throw;
            }
            finally
            {
                Switcher<bool, DisableCachePopulationSwitcher>.Exit();
            }
        }

        this.UpdateJobStatus(context);
    }

    protected override void ProcessEntries(Sitecore.Publishing.Pipelines.Publish.PublishContext context)
    {
        // Check for termination before processing
        if (ShouldTerminatePublish())
        {
            TerminatePublish(context);
            return;
        }

        List<IEnumerable<Sitecore.Publishing.PublishingCandidate>> publishingCandidatesList = new List<IEnumerable<Sitecore.Publishing.PublishingCandidate>>();

        foreach (IEnumerable<Sitecore.Publishing.PublishingCandidate> source in context.Queue)
        {
            // Check for termination inside the loop
            if (ShouldTerminatePublish())
            {
                TerminatePublish(context);
                return;
            }

            Sitecore.Publishing.PublishingCandidate[] array = source.ToArray<Sitecore.Publishing.PublishingCandidate>();
            try
            {
                this.ProcessEntries(array, context);
            }
            catch (PublishTerminatedException)
            {
                // If termination exception was thrown, gracefully stop
                TerminatePublish(context);
                return;
            }
            catch (Exception ex)
            {
                PublishingLog.Warn("An error during Publish Pipeline Process Queue execution. Will be tried to republish after the main part of publish will be done.", ex);
                foreach (Sitecore.Publishing.PublishingCandidate publishingCandidate in array)
                    context.ProcessedPublishingCandidates.TryRemove(new Sitecore.Publishing.ProcessedPublishingCandidate(publishingCandidate.ItemId, publishingCandidate.Language), out byte _);
                publishingCandidatesList.Add(array);
            }
        }

        // Only proceed with retry if no termination was requested
        if (!ShouldTerminatePublish())
        {
            foreach (IEnumerable<Sitecore.Publishing.PublishingCandidate> entries in publishingCandidatesList)
                this.ProcessEntries(entries, context);
        }
    }

    protected virtual bool ShouldTerminatePublish()
    {
        bool isTerminated = PublishJobManager.Instance.IsTerminationRequested();
        if (isTerminated)
        {
            PublishingLog.Info("[Publish Termination] Termination flag detected in ProcessQueue");
        }
        return isTerminated;
    }

    protected virtual void TerminatePublish(Sitecore.Publishing.Pipelines.Publish.PublishContext context)
    {
        PublishingLog.Info($"[Publish Termination] Beginning termination process. Items processed so far: {context.Statistics.Created + context.Statistics.Updated + context.Statistics.Deleted}");

        int queueCount = 0;
        // Count and clear remaining items
        foreach (var queue in context.Queue)
        {
            queueCount += queue.Count();
        }
        PublishingLog.Info($"[Publish Termination] Clearing {queueCount} remaining items from publish queue");

        // Clear any remaining queue to prevent further processing
        context.Queue.Clear();

        // Add a message to the job status
        BaseJob job = context.Job;
        if (job != null)
        {
            job.Status.Messages.Add("Publish terminated by user.");
            PublishingLog.Info($"[Publish Termination] Added termination message to job: {job.Name}");
        }

        // Log stats before termination
        PublishingLog.Info($"[Publish Termination] Termination stats - Created: {context.Statistics.Created}, Updated: {context.Statistics.Updated}, Deleted: {context.Statistics.Deleted}, Skipped: {context.Statistics.Skipped}");

        // Reset the termination flag
        PublishJobManager.Instance.ResetTermination();
        PublishingLog.Info("[Publish Termination] Termination flags reset, new publish jobs can now be started");
    }
}

2. Publish Job Manager

public class PublishJobManager
{
    private static PublishJobManager _instance;
    private bool _terminateFlag = false;
    private HashSet<string> _terminatedJobs = new HashSet<string>();

    public static PublishJobManager Instance
    {
        get
        {
            if (_instance == null)
            {
                _instance = new PublishJobManager();
            }
            return _instance;
        }
    }

    public void FlagTermination()
    {
        Sitecore.Diagnostics.Log.Info("[Publish Termination] Termination requested by user", this);
        _terminateFlag = true;

        // Get all running publish jobs
        var runningJobs = JobManager.GetJobs()
            .Where(j => j.Category == "PublishManager" && j.Status.State == JobState.Running);

        Sitecore.Diagnostics.Log.Info($"[Publish Termination] Found {runningJobs.Count()} running publish jobs", this);

        foreach (var job in runningJobs)
        {
            // Skip already terminated jobs
            if (_terminatedJobs.Contains(job.Handle.ToString()))
            {
                Sitecore.Diagnostics.Log.Info($"[Publish Termination] Job {job.Name} already marked for termination, skipping", this);
                continue;
            }

            try
            {
                // 1. Mark the job in our tracking collection
                _terminatedJobs.Add(job.Handle.ToString());
                Sitecore.Diagnostics.Log.Info($"[Publish Termination] Added job {job.Name} to terminated jobs collection", this);

                // 2. Access the PublishStatus to update its state
                var publishStatus = job.Options.Parameters[1] as PublishStatus;
                if (publishStatus != null)
                {
                    // Log current job progress
                    Sitecore.Diagnostics.Log.Info($"[Publish Termination] Job {job.Name} current progress: {publishStatus.Processed} items processed", this);

                    // Add a termination message
                    publishStatus.Messages.Add("Publish terminated by user.");
                    Sitecore.Diagnostics.Log.Info($"[Publish Termination] Added termination message to PublishStatus", this);
                }

                Sitecore.Diagnostics.Log.Info($"[Publish Termination] Successfully flagged publish job for termination: {job.Name}", this);
            }
            catch (Exception ex)
            {
                Sitecore.Diagnostics.Log.Error($"[Publish Termination] Error flagging publish job for termination: {job.Name}", ex, this);
            }
        }
    }

    public bool IsTerminationRequested()
    {
        if (_terminateFlag)
        {
            Sitecore.Diagnostics.Log.Info("[Publish Termination] Termination flag is active", this);
        }
        return _terminateFlag;
    }

    public void CheckTermination()
    {
        // We don't reset the flag here - let the TerminableProcessQueue do it
        if (_terminateFlag)
        {
            Sitecore.Diagnostics.Log.Info("[Publish Termination] Termination detected in Terminator processor, throwing exception", this);
            throw new PublishTerminatedException("Publishing process has been terminated by user.");
        }
    }

    public void ResetTermination()
    {
        Sitecore.Diagnostics.Log.Info($"[Publish Termination] Resetting termination flags. Cleared {_terminatedJobs.Count} terminated jobs", this);
        _terminateFlag = false;
        _terminatedJobs.Clear();
    }
}

public class Terminator : Sitecore.Publishing.Pipelines.PublishItem.PublishItemProcessor
{
    public override void Process(Sitecore.Publishing.Pipelines.PublishItem.PublishItemContext context)
    {
        PublishJobManager.Instance.CheckTermination();
    }
}

public class PublishTerminatedException : Exception
{
    public PublishTerminatedException(string message) : base(message) { }
}

3. Configuration Patch

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <!-- Replace the standard ProcessQueue with our custom implementation -->
      <publish>
        <processor type="Sitecore.Foundation.SitecoreExtensions.Pipelines.TerminableProcessQueue, Sitecore.Foundation.SitecoreExtensions"
                   patch:instead="processor[@type='Sitecore.Publishing.Pipelines.Publish.ProcessQueue, Sitecore.Kernel']" />
      </publish>

      <!-- Keep the Terminator processor as an additional safety mechanism -->
      <publishItem>
        <processor type="Sitecore.Foundation.SitecoreExtensions.Pipelines.Terminator, Sitecore.Foundation.SitecoreExtensions" 
                   patch:after="processor[@type='Sitecore.Publishing.Pipelines.PublishItem.UpdateStatistics, Sitecore.Kernel']" />
      </publishItem>
    </pipelines>
  </sitecore>
</configuration>

4. PowerShell Script with Confirmation

foreach($job in $selectedData)
{
    if ($job.Status.State -eq 'Running')
    {
        try {
            # Show confirmation dialog using the proper SPE cmdlet
            $result = Show-Confirm -Title "Are you sure you want to terminate this running publish job?`n`nJob: $($job.Name)`nPath: $($job.Options.Parameters[0].RootItem.Paths.FullPath)"

            # If user confirmed
            if ($result) {
                # Call termination method
                [Sitecore.Foundation.SitecoreExtensions.Pipelines.PublishJobManager]::Instance.FlagTermination()
                Show-Alert "Termination signal sent to publishing job. The job will be terminated at the next safe point."

                # Wait a second to allow the system to process the termination request
                Start-Sleep -Seconds 1
            }
            else {
                Show-Alert "Termination canceled."
            }
        }
        catch {
            Show-Alert "Failed to terminate publish job: $_"
        }
    }
    else 
    {
        Show-Alert "Can only terminate running jobs. Use Cancel for queued jobs."
    }
}

# Refresh the view to show updated job status
Import-Function -Name Refresh-PublishStatus
Refresh-PublishStatus

How It Works

The solution works through a multi-layered approach:

1. Termination Request

   • User clicks "Terminate" button in the PowerShell interface
   • After confirmation, PublishJobManager.FlagTermination() is called
   • This sets the _terminateFlag to true and marks the job in the _terminatedJobs collection

2. Detection at Multiple Levels

   • Process Level: TerminableProcessQueue checks the termination flag at the beginning and end of processing
   • Batch Level: Checks before processing each batch of publishing candidates
   • Item Level: The Terminator processor checks for each item being processed

3. Termination Execution

   • When termination is detected, TerminatePublish() is called
   • This method clears the publishing queue, preventing further processing
   • Adds a termination message to the job status
   • Resets the termination flag for future publishing jobs

4. State Reset

   • ResetTermination() is called to clear the termination flag and terminated jobs collection
   • This ensures new publishing jobs can be started without issues

Advantages of This Approach

1. Single-Click Termination

   • Works reliably with just one click
   • No need for multiple attempts to terminate a job

2. Clean State Management

   • Properly resets flags and states after termination
   • Ensures new publishing jobs can be started immediately after termination

3. Comprehensive Logging

   • Detailed logs of the termination process
   • Helps with troubleshooting and understanding what happened

4. Works with Sitecore's Architecture

   • Integrates cleanly with Sitecore's pipeline architecture
   • Doesn't disrupt other system functionality

5. User-Friendly Interface

   • Confirmation dialog prevents accidental termination
   • Clear feedback when termination is initiated

Implementation Considerations

1. Version Compatibility

   • This solution has been tested with Sitecore 10.4
   • Earlier versions may require adjustments to the code

2. Performance Impact

   • The additional termination checks have minimal impact on publishing performance
   • The benefit of being able to terminate stuck jobs far outweighs any small overhead

3. Error Handling

   • The solution includes robust error handling to prevent issues even if termination fails
   • Detailed logging helps diagnose any problems

4. Integration with SPE

   • Works seamlessly with Sitecore PowerShell Extensions
   • Extends SPE's publishing management capabilities

Conclusion

The ability to terminate running publishing jobs is a critical feature for Sitecore administrators, especially in large-scale environments with complex content hierarchies. This solution provides a robust, reliable way to terminate running publish jobs without restarting the server.

By replacing the core ProcessQueue processor and implementing a multi-layered termination detection system, we've created a solution that works with a single click and maintains system stability. The detailed logging and clean state management ensure that the system remains in a consistent state even after termination.

This solution addresses a common pain point in Sitecore content management and can save significant time and frustration for content authors and administrators alike.

Do you have experiences with managing complex publishing operations in Sitecore? Have you encountered other challenges that required custom solutions? Share your thoughts in the comments below!

评论

Reply