I am building a distributed application that needs TCP for communication. The final experiments will involve thousands of servers with about 10-100 fold as many clients communicating with them. The current design is to make connection, communicate and then close the connection, for every transaction.
Since I had only done some simple multi threaded servers with a few clients, I chose blocking sockets for the communication(they sounded easy). I am not sure if this approach will scale high enough. Can some one share their experience?
Currently, the code is running on a single machine(48 cores/local loopback) with three servers processes and a few clients. Clients have a tight for loop where they make a connection, communicate with any one server and then break the connection. These initial tests require the clients to make somewhere like a million iterations of the for loop. Basically, it is rigorous testing as we scale the experiments up.Randomly, some of the clients get stuck while they try to connect to the server.At the sane time, this server may be communicating fine with other clients. All calls are blocking.Can some one suggest what's happening? The listen queue of the server(multi threaded) is 50 and one client does about 1500 connections a second. ( Its a distributed system for high end work loads :))Can a prior unbroken connection between this client-server be a reason?
Any sharing of experience would be helpful...