Custom SERP analysis parsing issues

hey everyone, been working on a custom python script to pull more granular data for our long-tail keyword research. we're trying to automate some of the serp analysis beyond what commercial tools offer, especially for very niche local terms.

the main issue is with parsing the html responses from google serp pages after using a proxy rotator. i'm hitting a wall with inconsistent xpath results and sometimes getting captcha/block pages even with good proxies. its real frustrating.

i've tried different parsing libraries (beautifulsoup, lxml), various xpath selectors, and even some headless browser solutions (selenium with undetected_chromedriver). for proxies, i'm rotating through a pool of residential ips, and i've implemented delays and user-agent rotation. i also experimented with different header sets, trying to mimic real browser requests as closely as possible.

despite all this, i'm still getting a significant number of failed requests or pages that are clearly captcha walls, which then throws off my parsing logic. the lxml parser often returns empty lists for selectors that should work based on manual inspection. here's a snippet of the kind of lxml output i'm seeing for what should be a valid result:

# Attempting to extract organic result titles
titles = tree.xpath('//div[@class="g"]//h3/text()')
print(f"Titles found: {titles}")
# Output often looks like this for a failed parse:
Titles found: []

# And sometimes I get this in my logs, even with good proxies:
ERROR:root:Proxy IP 123.45.67.89 failed with status 429 - Too Many Requests.
WARNING:root:Captcha detected for URL: https://www.google.com/search?q=niche+local+service

i'm wondering if anyone has found a more robust way to handle google's anti-bot measures when scraping serp data directly, especially with python. are there specific headers or request patterns that are more effective? or am i missing some fundamental aspect of advanced serp analysis beyond basic proxy/ua rotation? help a brother out please...