Troubleshooting Scenarios

• 503 errors returned to client
• All requests to the endpoint fail
• Health check endpoint not responding
• Error rate metric spikes to 100%

• Call the /health endpoint first
• Check application logs for startup errors
• Check if the service process is running
• Check disk and memory — may have caused crash

Restart the service. If it crashes again immediately — check logs for the crash reason before restarting again. Escalate to L3 if restart does not hold.

• Writes failing across all services
• DB transactions returning write errors
• Log files stopped growing
• Applications crashing on file writes

• Run df -h to confirm disk at 100%
• Run du -sh /var/log/* | sort -rh | head
• Identify the biggest folder consuming space
• Check if old logs are the culprit (most common)

Clean logs. Archive or delete old log files to free space. Monitor disk after cleanup. If it fills quickly again — a rogue log is being written continuously. Find and fix the source.

• Slow application response times
• Requests timing out across all endpoints
• CPU metric above 90% on dashboard
• Load average growing over time

• Run top — press P to sort by CPU
• Run ps aux --sort=-%cpu | head -6
• Identify the process using the most CPU
• Check how long it has been running

Kill the process — kill -15 PID for graceful stop. Get bridge approval before killing on PROD. Confirm CPU recovers after kill. Never use kill -9 on a DB process.

• Transactions stuck in PENDING state
• Delay reported by clients
• MQ queue depth growing, not decreasing
• Consumer not picking up messages

• Check MQ queue depth — is it growing?
• Check consumer service — is it running?
• Check consumer logs for crash errors
• Note the queue depth before touching anything

Restart consumer in safe order: stop consumer → get bridge approval → restart MQ if needed → restart consumer → monitor queue depth decreasing. Messages are not lost — they wait safely in the queue.

• Customer complaint — payment not received
• Transaction ID shows no result in client app
• Money debited but destination not credited
• Status unknown — client has no update

• Search DB for the TXN reference ID
• Check TRANSACTIONS_LOG for current status
• Check MX_MESSAGE for SBP/external response
• Grep logs for TXN ID — full history

Trace status. If DB shows SUCCESS but client not notified — trigger notification. If PENDING and SBP confirmed — manually update (L3 approval). If FAILED — check reconciliation and refund eligibility.

• Slow API responses — taking seconds not ms
• TIMEOUT errors in logs
• Transactions failing after 30 seconds
• Client reports "request timed out"

• Identify timeout type from the log message
• QUERY_TIMEOUT → DB query too slow
• SOCKET_TIMEOUT → external API not responding
• Check network if SOCKET_TIMEOUT

Restart gateway if external timeout. If DB query timeout — check for slow queries and add index (do not restart DB). If network — check connectivity to the external endpoint.

• TLS error in logs or client response
• SSL handshake failure messages
• All HTTPS connections to endpoint failing
• Browser shows "certificate not trusted"

• Check the certificate expiry date
• Run: openssl s_client -connect host:443
• Confirm it is the cert and not the config
• Identify who issued and manages the cert

Renew the certificate. Contact the certificate authority or use the internal cert management tool. After renewing — restart the web server or load balancer to apply the new cert. Verify with openssl after.

• Transaction response times very high
• QUERY_TIMEOUT errors in logs
• All services depending on DB are slow
• DB CPU spiking on the server

• Check pg_stat_activity for long-running queries
• Identify which query is taking the most time
• Check if a lock is being held on a table
• Do NOT restart DB — diagnose first

Optimise query — add index. Never restart the DB for a slow query — that wastes time and risks data integrity. Ask DBA to add the missing index. Response time drops immediately after the index is created.