Advanced server optimization: pinpointing elusive database contention issues?

We're running a high-traffic SaaS application, and while our basic server maintenance routines and initial query optimizations for our PostgreSQL database are solid, we've been hitting a wall with intermittent, elusive performance bottlenecks. Specifically, we're experiencing periodic database contention that manifests as spikes in CPU/IO and noticeable slowdowns in response times. The frustrating part is that our standard monitoring tools aren't consistently pinpointing the root cause, making it incredibly difficult to diagnose and resolve.

We've been diligently using tools like pg_stat_activity, htop, and iostat during these events. We frequently observe high wait_event_type values, such as 'ClientRead' or 'Lock', and an increase in wa (wait I/O) states. However, specific blocking queries or deadlocks that would clearly identify the culprit are not consistently visible in pg_stat_activity, or they resolve before we can properly capture them. It feels like we're dealing with very transient, granular contention points that aggregate into a larger problem. Here’s a simplified example of what we might see in pg_stat_activity during one of these periods, illustrating multiple backend processes in various wait states:

 pid | user | database | client_addr | application_name | backend_type | state | wait_event_type | wait_event | query
-----+------+----------+-------------+------------------+--------------+-------+-----------------+------------+-----------------------------------
 123 | app  | mydb     | 10.0.0.1    | web_app          | client_query | active| ClientRead      | ClientRead | SELECT * FROM users WHERE id = $1;
 124 | app  | mydb     | 10.0.0.2    | web_app          | client_query | active| Lock            | tuple      | UPDATE products SET stock = $1 WHERE id = $2;
 125 | app  | mydb     | 10.0.0.3    | web_app          | client_query | active| ClientRead      | ClientRead | INSERT INTO orders (...) VALUES (...);
 126 | app  | mydb     | 10.0.0.4    | background       | client_query | active| IO              | BufFileRead| COPY large_table TO STDOUT;
 127 | app  | mydb     | 10.0.0.5    | web_app          | client_query | idle  |                 |            | <IDLE>

My core question is this: beyond these routine checks, what advanced strategies and tools are experts using to diagnose deep database contention issues in high-concurrency environments? We're particularly interested in techniques for profiling transient lock contention, identifying hidden bottlenecks, or perhaps even specialized PostgreSQL extensions or monitoring solutions designed for such scenarios. Eager for expert insights on profiling and resolving these complex server optimization challenges.