When instant messaging latency is high, do not rush to restart services or switch networks. Latency occurs at different stages, and the corresponding troubleshooting targets are completely different. Break the user-reported "slowness" into four symptoms: slow connection establishment, slow first-byte return, slow send-to-delivery, and slow message rendering; then compare timestamps in client and server logs to basically locate the problem to the network layer, server, or client. This article provides executable steps and judgment criteria in a layered order. If the user reports that messages cannot be sent, the troubleshooting direction is different; refer toSystematic Troubleshooting for Message Sending Failures.
First locate the layer: use timestamps and symptoms to triage
There are four key timestamps: client send time t1, server receive time t2, server delivery time t3, and client receive time t4. If t2-t1 is significantly large, check the network layer first; if t3-t2 is large, the problem is in server-side message processing; if t4-t3 is large, there is a problem on the return network or client receive buffer; if the gap between the client's actual on-screen time t5 and t4 is large, it indicates UI thread rendering lag. Do not look at only one side's logs; compare at least the sender-side and receiver-side logs, otherwise it is easy to mistake slow server processing for network packet loss.
It is best to also classify user-reported "slowness" by symptom: slow connection establishment usually occurs during the DNS, TCP/TLS stage; slow first-byte return means slow server processing; slow send-to-delivery may be a server push pipeline or persistent connection problem; slow message rendering is mostly client rendering. Spend a minute first tying symptoms to timestamps, and later troubleshooting will not go off track.
Network layer troubleshooting: connection establishment and persistent connection status
- DNS resolution:Use
mtrordigto check resolution time. If domain resolution exceeds 200ms, switch to public DNS first or check the local cache and hosts configuration. - TCP/TLS handshake:Use
ss -tito check connection status and RTT, or usetcpdumpto capture packets and check the SYN/ACK time difference. If the handshake takes more than 300ms, it usually indicates link quality issues or a problem with the server accept queue. - Packet loss and retransmission:In the test environment, use
tcto simulate weak network conditions, and observeretransthe field. If the retransmission rate consistently exceeds 1%, it directly amplifies message latency; troubleshoot the link or adjust the retransmission policy. - Persistent connection heartbeat:For WebSocket or private persistent connections, the heartbeat interval should not exceed 30 seconds. If the heartbeat interval is too long, NAT devices may reclaim the connection early, causing intermittent latency or seemingly dead connections.
Server-side troubleshooting: from the access layer to the push pipeline
- Access layer connection count:Check whether the connection count on the persistent connection gateway or access layer has reached its ceiling and whether the accept queue is backlogged.
netstat -sThe listen overflow in the accept queue is a common signal. - Message routing latency:Check the latency from message receipt at the server to delivery. If P95 exceeds 200ms, investigate the message queue, routing logic, and thread pool saturation.
- Storage and caching:Slow queries, message persistence, and group member fan-out can all become bottlenecks. For large groups, a single write fan-out can easily cause latency spikes and should be replaced with batch or asynchronous processing.
- Offline push:Offline messages are delivered through the push channel, and the latency of the push service itself is often counted toward IM latency. The push path needs to be timed separately. Failure to receive offline messages is usually a separate type of issue; seeCommon Causes and Fixes for Persistent Offline Message Loss.
Key server-side metrics to monitor include: P95 message processing latency, queue backlog size, GC pause duration, and CPU spikes. Occasional 100% CPU is not necessarily a problem, but if it appears together with latency spikes, continue drilling down.
Client-side troubleshooting: distinguish between network waiting and rendering jank.
- Compare the time t4 when the message is received in the client logs with the actual on-screen time t5. If t5-t4 exceeds 100ms, suspect UI thread jank first rather than the network or server side.
- Temporarily disable modules such as animations, list reuse, and large image loading, and observe whether the situation improves. If it improves, it indicates that rendering or main-thread tasks are overloaded, and layout and image decoding need to be optimized.
- Compare the behavior of the same message across multiple clients under weak network simulation. If only a single device has latency, server-side issues can basically be ruled out.
When the client shows latency but server-side logs are normal, continue tracing forward through client receive buffering and UI scheduling; do not treat network probing results as the conclusion.
Validation and Rollback: Single-Variable Changes and Baseline Records
Change only one variable at a time. Record the P50/P95 latency baseline before the change, and observe at least one full business cycle after the change. If P95 does not decrease by more than 10%, or if P50 worsens instead, roll back the change immediately. Do not adjust network, server-side, and client-side parameters at the same time; otherwise, even if recovery occurs, the effective action cannot be identified. Rollback conditions should be written down in advance to avoid continuing to stack changes under pressure.
Common Issues
Can the same set of troubleshooting steps be used for high latency and message sending failures?
Not entirely. For sending failures, focus on whether delivery succeeds; for latency, focus on the time distribution. The layered troubleshooting approach is the same for both, but the judgment metrics differ. For sending failures, refer to the message sending failure troubleshooting article; for latency, first separate traffic by timestamp.
What should be checked first when the WebSocket connection is normal but message latency is high?
First check server-side message processing time and push queue backlog, then check whether the client is delaying retrieval or the UI thread is blocking. A normal connection does not mean there is no queueing in the path.
How should you proceed if the client shows latency but the server logs are normal?
Capture client logs and compare the receive time and display time, focusing on the receive buffer and rendering thread. If t5-t4 is large, the problem is on the client.
What are the key differences in latency behavior between XMPP and WebSocket?
XMPP is based on XML stanzas and server routing. Compared with WebSocket, it adds one more hop and negotiation overhead, so latency is amplified on weak networks. WebSocket only handles transport, so latency is more directly affected by the network and server-side processing. Clarify protocol boundaries before making changes.
What are the differences in the data available for latency troubleshooting between self-hosted IM and SaaS?
Self-hosted IM gives you full-chain logs and kernel metrics, enabling more precise diagnosis. SaaS usually provides only client logs and limited server-side metrics, requiring the vendor to provide sampled data. When selecting a solution, if the budget is limited, refer toSmall and Medium Enterprise Instant Messaging Selection Criteria.
