Repository navigation
Closed
Labels
staleIssues with no activity for the past 6 monthsIssues with no activity for the past 6 months
Description
I ocassionally get a SocketException from tests when using the TestCluster.
Message:
System.Net.Sockets.SocketException : An attempt was made to access a socket in a way forbidden by its access permissions.
Stack Trace:
Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
Socket.Bind(EndPoint localEP)
SocketConnectionListener.Bind()
SocketConnectionListenerFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
ConnectionListener.BindAsync(CancellationToken cancellationToken)
SiloConnectionListener.OnRuntimeInitializeStart(CancellationToken cancellationToken)
MonitoredObserver.OnStart(CancellationToken ct)
LifecycleSubject.OnStart(CancellationToken ct)
AsyncClosureWorkItem.Execute()
Silo.StartAsync(CancellationToken cancellationToken)
SiloHost.StartAsync(CancellationToken cancellationToken)
<<CreateAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
InProcessSiloHandle.CreateAsync(String siloName, IList`1 configurationSources)
TestCluster.StartSiloAsync(Int32 instanceNumber, TestClusterOptions clusterOptions, IReadOnlyList`1 configurationOverrides, Boolean startSiloOnNewPort)
TestCluster.InitializeAsync()
TestCluster.DeployAsync()
TestCluster.DeployAsync()
Here is my test code:
public class HelloGrainTest : ClusterFixture
{
public HelloGrainTest(ITestOutputHelper testOutputHelper)
: base(testOutputHelper)
{
}
[Fact]
public async Task SayHello_PassName_ReturnsGreetingAsync()
{
var grain = this.Cluster.GrainFactory.GetGrain<IHelloGrain>(Guid.NewGuid());
var greeting = await grain.SayHelloAsync("Rehan").ConfigureAwait(false);
Assert.Equal("Hello Rehan!", greeting);
}
}
public class ClusterFixture : IAsyncLifetime
{
public ClusterFixture(ITestOutputHelper testOutputHelper)
{
this.TestOutputHelper = testOutputHelper;
Log.Logger = new LoggerConfiguration()
.WriteTo.Debug()
.WriteTo.TestOutput(testOutputHelper, LogEventLevel.Verbose)
.CreateLogger();
this.Cluster = new TestClusterBuilder()
.AddClientBuilderConfigurator<TestClientBuilderConfigurator>()
.AddSiloBuilderConfigurator<TestSiloConfigurator>()
.Build();
}
public TestCluster Cluster { get; }
public ITestOutputHelper TestOutputHelper { get; }
public Task DisposeAsync() => this.Cluster.DisposeAsync().AsTask();
public Task InitializeAsync() => this.Cluster.DeployAsync();
}
public class TestClientBuilderConfigurator : IClientBuilderConfigurator
{
public void Configure(IConfiguration configuration, IClientBuilder clientBuilder) =>
clientBuilder.AddSimpleMessageStreamProvider(StreamProviderName.Default);
}
public class TestSiloConfigurator : ISiloConfigurator
{
public void Configure(ISiloBuilder siloBuilder) =>
siloBuilder
.ConfigureServices(services => services.AddSingleton<ILoggerFactory>(x => new SerilogLoggerFactory()))
.AddMemoryGrainStorageAsDefault()
.AddMemoryGrainStorage("PubSubStore")
.AddSimpleMessageStreamProvider(StreamProviderName.Default);
}Reacted by jbezak
Metadata
Metadata
Assignees
Labels
staleIssues with no activity for the past 6 monthsIssues with no activity for the past 6 months
Activity
jbezak commentedon Dec 9, 2020
I see a similar issue in our build from time to time:
ReubenBond commentedon Dec 10, 2020
Which version of Orleans are you each using?
RehanSaeed commentedon Dec 10, 2020
3.3.0
jbezak commentedon Dec 10, 2020
same
imakhlin commentedon Dec 13, 2020
We are facing the same issue with the same version of Orleans (3.3.0) as well as with earlier versions.
I think I can explain the root cause (at least in Windows environments)
There are few ranges of ports the have been
excludedby the OS and should not be considered as free or available portsnetsh interface ipv4 show excludedportrange protocol=tcpTestClusterPortAllocator(hardcoded!) are interfere with theexcludedport ranges of OS.This leads to transient and inconsistent failures of the tests as allocator use to pick required port ranges from the the larger range randomly.
I would suggest make necessary changes to the port allocator to avoid reserved ports or at least give us an option to specify the lookup ranges based on our environment settings.
Example:
Actual ports provided by allocator are in the range of "excluded" ports
StephaneMartignier commentedon Jan 4, 2021
Hi,
any news from this? We are facing the same issue and It's pretty annoying because it's occurs on an DevOps pipeline which is supposed to generate nuggets
ReubenBond commentedon Jan 12, 2021
Which platforms are you all running on? It should be marginally less common on Windows, especially if you're running all your tests in a single process.
It's an unfortunate issue. Essentially, what happens is this: whenever a TCP socket closes, the kernel (win/lin) will keep that socket open for up to 2 minutes to be extra sure that no more data will be received on the socket. So we have some code which tries to reduce the effects of clashes by means of a centralized port allocator: #6399. It is not 100% effective, though, and suggestions/contributions are welcome.
jbezak commentedon Jan 12, 2021
We are running it on windows-latest with github actions.
StephaneMartignier commentedon Jan 13, 2021
We are using Windows (Server) too
StephaneMartignier commentedon Jan 14, 2021
Hey,
First thanks for the reactivity :)
I can see that the pull request was merged =D
Any idea on a release date (stable/preview version)?
Thanks a lot the amazing work
ReubenBond commentedon Jan 21, 2021
That PR was a part of 3.2.0. It helps, but doesn't eliminate all issues. One option is for us to implement an in-memory transport for testing, rather than using a socket-based transport.
ohadschn commentedon May 21, 2021
We are indeed hitting this on linux (Ubuntu 18.04). What about a few retries then (using Polly or some such for exponential backoff)? I would rather have my test take 2 minutes longer once every 20 executions than fail my build...
aelij commentedon May 30, 2021
A better solution would be to stop using TCP at all in the test silo.
Like ASP.NET's
TestServer, which allows creating a fake HTTP transport. It provides a serialization boundary, but all bytes are copied in-memory.