Skip to content

Intermittent SockerException from TestCluster #6845

Closed

Description

@RehanSaeed

I ocassionally get a SocketException from tests when using the TestCluster.

  Message: 
    System.Net.Sockets.SocketException : An attempt was made to access a socket in a way forbidden by its access permissions.
  Stack Trace: 
    Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
    Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
    Socket.Bind(EndPoint localEP)
    SocketConnectionListener.Bind()
    SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
    ConnectionListener.BindAsync(CancellationToken cancellationToken)
    SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)
    MonitoredObserver.OnStart(CancellationToken ct)
    LifecycleSubject.OnStart(CancellationToken ct)
    AsyncClosureWorkItem.Execute()
    Silo.StartAsync(CancellationToken cancellationToken)
    SiloHost.StartAsync(CancellationToken cancellationToken)
    <<CreateAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location ---
    InProcessSiloHandle.CreateAsync(String siloName, IList`1 configurationSources)
    TestCluster.StartSiloAsync(Int32 instanceNumber, TestClusterOptions clusterOptions, IReadOnlyList`1 configurationOverrides, Boolean startSiloOnNewPort)
    TestCluster.InitializeAsync()
    TestCluster.DeployAsync()
    TestCluster.DeployAsync()

Here is my test code:

public class HelloGrainTest : ClusterFixture
{
    public HelloGrainTest(ITestOutputHelper testOutputHelper)
        : base(testOutputHelper)
    {
    }

    [Fact]
    public async Task SayHello_PassName_ReturnsGreetingAsync()
    {
        var grain = this.Cluster.GrainFactory.GetGrain<IHelloGrain>(Guid.NewGuid());

        var greeting = await grain.SayHelloAsync("Rehan").ConfigureAwait(false);

        Assert.Equal("Hello Rehan!", greeting);
    }
}

public class ClusterFixture : IAsyncLifetime
{
    public ClusterFixture(ITestOutputHelper testOutputHelper)
    {
        this.TestOutputHelper = testOutputHelper;

        Log.Logger = new LoggerConfiguration()
            .WriteTo.Debug()
            .WriteTo.TestOutput(testOutputHelper, LogEventLevel.Verbose)
            .CreateLogger();

        this.Cluster = new TestClusterBuilder()
            .AddClientBuilderConfigurator<TestClientBuilderConfigurator>()
            .AddSiloBuilderConfigurator<TestSiloConfigurator>()
            .Build();
    }

    public TestCluster Cluster { get; }

    public ITestOutputHelper TestOutputHelper { get; }

    public Task DisposeAsync() => this.Cluster.DisposeAsync().AsTask();

    public Task InitializeAsync() => this.Cluster.DeployAsync();
}

public class TestClientBuilderConfigurator : IClientBuilderConfigurator
{
    public void Configure(IConfiguration configuration, IClientBuilder clientBuilder) =>
        clientBuilder.AddSimpleMessageStreamProvider(StreamProviderName.Default);
}

public class TestSiloConfigurator : ISiloConfigurator
{
    public void Configure(ISiloBuilder siloBuilder) =>
        siloBuilder
            .ConfigureServices(services => services.AddSingleton<ILoggerFactory>(x => new SerilogLoggerFactory()))
            .AddMemoryGrainStorageAsDefault()
            .AddMemoryGrainStorage("PubSubStore")
            .AddSimpleMessageStreamProvider(StreamProviderName.Default);
}

Activity

  1. jbezak commented on Dec 9, 2020

    @jbezak

    I see a similar issue in our build from time to time:

    2020-12-09T12:32:49.0223204Z info
    2020-12-09T12:32:49.0223889Z : Orleans.Runtime.SiloLifecycleSubject[100452]
    2020-12-09T12:32:49.0224843Z        ConnectionManager started in stage 1999 in 0.0367 Milliseconds
    2020-12-09T12:32:49.0225412Z fail
    2020-12-09T12:32:49.0226100Z : Orleans.Runtime.SiloLifecycleSubject[100450]
    2020-12-09T12:32:49.0227876Z        GatewayConnectionListener failed to start due to errors at stage 1999: System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions
    2020-12-09T12:32:49.0229781Z           at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
    2020-12-09T12:32:49.0231079Z           at System.Net.Sockets.Socket.Bind(EndPoint localEP)
    2020-12-09T12:32:49.0233137Z           at Orleans.Networking.Shared.SocketConnectionListener.Bind() in D:\build\agent\_work\25\s\src\Orleans.Core\Networking\Shared\SocketConnectionListener.cs:line 58
    2020-12-09T12:32:49.0236930Z           at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken) in D:\build\agent\_work\25\s\src\Orleans.Core\Networking\Shared\SocketConnectionListenerFactory.cs:line 41
    2020-12-09T12:32:49.0240419Z           at Orleans.Runtime.Messaging.ConnectionListener.<BindAsync>d__23.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Networking\ConnectionListener.cs:line 70
    2020-12-09T12:32:49.0242091Z        --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0243812Z           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0246970Z           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0250563Z           at Orleans.Runtime.Messaging.GatewayConnectionListener.<OnRuntimeInitializeStart>d__16.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Networking\GatewayConnectionListener.cs:line 85
    2020-12-09T12:32:49.0252677Z        --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0254319Z           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0257458Z           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0260885Z           at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.<OnStart>d__12.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Lifecycle\SiloLifecycleSubject.cs:line 116
    2020-12-09T12:32:49.0262583Z info
    2020-12-09T12:32:49.0263278Z : Orleans.Runtime.SiloLifecycleSubject[100452]
    2020-12-09T12:32:49.0264282Z        SiloConnectionListener started in stage 1999 in 143.4755 Milliseconds
    2020-12-09T12:32:49.0264917Z fail
    2020-12-09T12:32:49.0265606Z : Orleans.Runtime.SiloLifecycleSubject[100450]
    2020-12-09T12:32:49.0267210Z        Lifecycle start canceled due to errors at stage 1999: System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions
    2020-12-09T12:32:49.0268964Z           at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
    2020-12-09T12:32:49.0270257Z           at System.Net.Sockets.Socket.Bind(EndPoint localEP)
    2020-12-09T12:32:49.0272344Z           at Orleans.Networking.Shared.SocketConnectionListener.Bind() in D:\build\agent\_work\25\s\src\Orleans.Core\Networking\Shared\SocketConnectionListener.cs:line 58
    2020-12-09T12:32:49.0276111Z           at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken) in D:\build\agent\_work\25\s\src\Orleans.Core\Networking\Shared\SocketConnectionListenerFactory.cs:line 41
    2020-12-09T12:32:49.0279600Z           at Orleans.Runtime.Messaging.ConnectionListener.<BindAsync>d__23.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Networking\ConnectionListener.cs:line 70
    2020-12-09T12:32:49.0281256Z        --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0282964Z           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0286129Z           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0289786Z           at Orleans.Runtime.Messaging.GatewayConnectionListener.<OnRuntimeInitializeStart>d__16.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Networking\GatewayConnectionListener.cs:line 85
    2020-12-09T12:32:49.0292029Z        --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0293759Z           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0296884Z           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0300334Z           at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.<OnStart>d__12.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Lifecycle\SiloLifecycleSubject.cs:line 133
    2020-12-09T12:32:49.0302334Z        --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0303967Z           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0307087Z           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0309851Z           at Orleans.LifecycleSubject.<OnStart>d__7.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Core\Lifecycle\LifecycleSubject.cs:line 97
    2020-12-09T12:32:49.0310811Z fail
    2020-12-09T12:32:49.0311285Z : Orleans.Runtime.Silo[100439]
    2020-12-09T12:32:49.0311839Z        Exception during Silo.Start
    2020-12-09T12:32:49.0313008Z  System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions
    2020-12-09T12:32:49.0314576Z     at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
    2020-12-09T12:32:49.0315838Z     at System.Net.Sockets.Socket.Bind(EndPoint localEP)
    2020-12-09T12:32:49.0317813Z     at Orleans.Networking.Shared.SocketConnectionListener.Bind() in D:\build\agent\_work\25\s\src\Orleans.Core\Networking\Shared\SocketConnectionListener.cs:line 58
    2020-12-09T12:32:49.0321581Z     at Orleans.Networking.Shared.SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken) in D:\build\agent\_work\25\s\src\Orleans.Core\Networking\Shared\SocketConnectionListenerFactory.cs:line 41
    2020-12-09T12:32:49.0325129Z     at Orleans.Runtime.Messaging.ConnectionListener.<BindAsync>d__23.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Networking\ConnectionListener.cs:line 70
    2020-12-09T12:32:49.0326905Z  --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0328536Z     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0331677Z     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0335392Z     at Orleans.Runtime.Messaging.GatewayConnectionListener.<OnRuntimeInitializeStart>d__16.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Networking\GatewayConnectionListener.cs:line 85
    2020-12-09T12:32:49.0337461Z  --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0339113Z     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0342331Z     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0345752Z     at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.<OnStart>d__12.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Lifecycle\SiloLifecycleSubject.cs:line 133
    2020-12-09T12:32:49.0347685Z  --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0349333Z     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0352576Z     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0355394Z     at Orleans.LifecycleSubject.<OnStart>d__7.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Core\Lifecycle\LifecycleSubject.cs:line 111
    2020-12-09T12:32:49.0356619Z  --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0358250Z     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0361362Z     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0364600Z     at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.<Execute>d__9.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Scheduler\ClosureWorkItem.cs:line 35
    2020-12-09T12:32:49.0366251Z  --- End of stack trace from previous location where exception was thrown ---
    2020-12-09T12:32:49.0368030Z     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    2020-12-09T12:32:49.0371179Z     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    2020-12-09T12:32:49.0373866Z     at Orleans.Runtime.Silo.<StartAsync>d__54.MoveNext() in D:\build\agent\_work\25\s\src\Orleans.Runtime\Silo\Silo.cs:line 248
    2020-12-09T12:32:49.0375276Z ----------------------------- STARTING NEW UNIT TEST SILO HOST: Orleans.TestingHost.TestCluster -------------------------------------
    
  2. ReubenBond commented on Dec 10, 2020

    @ReubenBond
    Member

    Which version of Orleans are you each using?

  3. RehanSaeed commented on Dec 10, 2020

    @RehanSaeed
    Author

    3.3.0

  4. jbezak commented on Dec 10, 2020

    @jbezak

    same

  5. imakhlin commented on Dec 13, 2020

    @imakhlin

    We are facing the same issue with the same version of Orleans (3.3.0) as well as with earlier versions.

    I think I can explain the root cause (at least in Windows environments)
    There are few ranges of ports the have been excluded by the OS and should not be considered as free or available ports

    • You can list those ports using the following command: netsh interface ipv4 show excludedportrange protocol=tcp
    • In my case the output clearly shows that port ranges used by the TestClusterPortAllocator (hardcoded!) are interfere with the excluded port ranges of OS.

    This leads to transient and inconsistent failures of the tests as allocator use to pick required port ranges from the the larger range randomly.

    I would suggest make necessary changes to the port allocator to avoid reserved ports or at least give us an option to specify the lookup ranges based on our environment settings.

    Example:

    >netsh interface ipv4 show excludedportrange protocol=tcp
    
    Protocol tcp Port Exclusion Ranges
    
    Start Port    End Port
    ----------    --------
          5357        5357
          8081        8081
         10252       10252
         10254       10254
         49670       49769
         49770       49869
         49870       49969
         50000       50059     *
         54000       54099
         55101       55200
         55279       55378
         63441       63540
    
    * - Administered port exclusions.
    

    Actual ports provided by allocator are in the range of "excluded" ports

    [Information] Orleans.Configuration.EndpointOptions[0]
      Orleans Silo Port: 27015
    
    [Information] Orleans.Configuration.EndpointOptions[0]
      Orleans Gateway Port: 49840
    ….
    
    [Error] Orleans.Runtime.SiloLifecycleSubject[100450]
      GatewayConnectionListener failed to start due to errors at stage 1999: System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
    
  6. StephaneMartignier commented on Jan 4, 2021

    @StephaneMartignier

    Hi,
    any news from this? We are facing the same issue and It's pretty annoying because it's occurs on an DevOps pipeline which is supposed to generate nuggets

  7. self-assigned this
    on Jan 11, 2021
  8. ReubenBond commented on Jan 12, 2021

    @ReubenBond
    Member

    Which platforms are you all running on? It should be marginally less common on Windows, especially if you're running all your tests in a single process.

    It's an unfortunate issue. Essentially, what happens is this: whenever a TCP socket closes, the kernel (win/lin) will keep that socket open for up to 2 minutes to be extra sure that no more data will be received on the socket. So we have some code which tries to reduce the effects of clashes by means of a centralized port allocator: #6399. It is not 100% effective, though, and suggestions/contributions are welcome.

  9. jbezak commented on Jan 12, 2021

    @jbezak

    We are running it on windows-latest with github actions.

  10. StephaneMartignier commented on Jan 13, 2021

    @StephaneMartignier

    We are using Windows (Server) too

  11. StephaneMartignier commented on Jan 14, 2021

    @StephaneMartignier

    Hey,

    First thanks for the reactivity :)

    I can see that the pull request was merged =D

    Any idea on a release date (stable/preview version)?

    Thanks a lot the amazing work

  12. ReubenBond commented on Jan 21, 2021

    @ReubenBond
    Member

    That PR was a part of 3.2.0. It helps, but doesn't eliminate all issues. One option is for us to implement an in-memory transport for testing, rather than using a socket-based transport.

  13. ohadschn commented on May 21, 2021

    @ohadschn

    Which platforms are you all running on? It should be marginally less common on Windows, especially if you're running all your tests in a single process.

    It's an unfortunate issue. Essentially, what happens is this: whenever a TCP socket closes, the kernel (win/lin) will keep that socket open for up to 2 minutes to be extra sure that no more data will be received on the socket. So we have some code which tries to reduce the effects of clashes by means of a centralized port allocator: #6399. It is not 100% effective, though, and suggestions/contributions are welcome.

    We are indeed hitting this on linux (Ubuntu 18.04). What about a few retries then (using Polly or some such for exponential backoff)? I would rather have my test take 2 minutes longer once every 20 executions than fail my build...

  14. aelij commented on May 30, 2021

    @aelij
    Contributor

    A better solution would be to stop using TCP at all in the test silo.
    Like ASP.NET's TestServer, which allows creating a fake HTTP transport. It provides a serialization boundary, but all bytes are copied in-memory.

  15. ghost added
    staleIssues with no activity for the past 6 months
    on Dec 7, 2021
  16. ghost closed this as completedon Mar 4, 2022
  17. ghost locked as resolved and limited conversation to collaborators on Apr 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

staleIssues with no activity for the past 6 months

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions