How to accurately parse SERP analysis data for content optimization?

hey folks, our keyword density tool is still kinda struggling, but the new headache is trying to get good competitor analysis data. we're trying to feed it content from top-ranking pages for better SERP analysis, but reliably extracting clean, main content is proving super inconsistent. sometimes we get navigation, sometimes footers, and often just broken text. it's really messing with our content optimization insights.

i've included a fake console output below to show some of the common parsing errors we're seeing:

[ERROR] 2023-10-27 14:35:01 - Failed to extract main content from URL: https://example.com/page1
  Reason: Content block too small, likely navigation.
[WARNING] 2023-10-27 14:35:15 - Partial content extraction from URL: https://example.com/page2
  Details: Detected footer text & sidebar ads in main content area.
[ERROR] 2023-10-27 14:35:30 - Parsing error on URL: https://example.com/page3
  Message: 'div.main-content' not found, falling back to 'body'.
[INFO] 2023-10-27 14:35:45 - Successfully extracted from URL: https://example.com/page4
  Size: 2450 words

what are you guys using for robust content extraction from diverse webpage structures? we need something that can consistently pull just the main body content for accurate content optimization insights and more reliable competitor analysis. any specific libraries or approaches you recommend for dealing with dynamic content or tricky html?

Thanks in advance!