A few days ago, we encountered an issue with a client where numerous I/O dispatcher threads appeared to be locked. Although the thread state indicated they were running, they seemed to be stuck, as evident from the details below. Our concern is: is there a method to trace the origin of this issue? Could the bottleneck be attributed to a slow hard drive?
Edit: looking into the threaddump it's shown 1436 threads like this and there are another 700 threads with the name "pool-169-thread-1"(only changing the number in the middle), all this with only a few users(10-15 online).
The system operates on Azure with Linux, equipped with a 4-core CPU, 16GB RAM, Tomcat 9.0.70, and AdoptOpenJDK 13.0.2. Disk performance can peak at 70 IOPS
"I/O dispatcher 5097" #182118 prio=5 os_prio=0 cpu=466.57ms elapsed=31223.88s tid=0x0000000002442000 nid=0x5ef7 runnable [0x00007f85ca22d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPoll.wait(java.base@13.0.2/Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@13.0.2/EPollSelectorImpl.java:120) at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124) - locked <0x00000006cae638a8> (a sun.nio.ch.Util$2) - locked <0x00000006cae638b8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)I tried to simulate in a local machine with lots of different scenarios, muktiple users, scripting to open dozens pages, opening multiple trheads. We use the very same system inside the company and we do not have the same behaviour with 130 active users every day for a week.
Sorry for my bad english, tried to use gpt to make it better