On-page SEO tool data problems?

Hey everyone,

We just launched our new 'Keyword Density & Frequency Checker' tool over on AdsVolt.com, and it's designed to help users really nail their on-page SEO by deeply analyzing content for keyword usage. The idea is to give a clear picture of how well their target keywords are integrated.

However, we're running into some frustrating inconsistencies with the data, especially when it comes to keyword density and frequency calculations. It seems to be particularly problematic with longer articles or pages that rely heavily on dynamic elements or JavaScript to render content. Sometimes it outright miscounts keywords, either over-reporting or under-reporting, and other times it just misses keywords entirely, even when they're clearly visible on the page. It's making our results less reliable than we'd like.

We've really thrown a lot at this problem trying to get it right. Here’s a rundown of what we’ve tried so far:

• We've implemented various parsing libraries, including server-side logic similar to BeautifulSoup and client-side focused ones like JSDOM, trying to find the most robust way to process page content.
• We've experimented endlessly with different text extraction methods – carefully stripping HTML, making sure to ignore script and style tags, and even trying to handle comments gracefully to prevent them from skewing results.
• We’ve also tested with different user-agents and tried using headless browsers to mimic real user visits as closely as possible, thinking maybe it was a rendering issue.
• Of course, we've double-checked for encoding issues (like making sure everything is consistently UTF-8) to rule out basic character interpretation problems.
• Perhaps most tellingly, we’ve compared our tool's output with several other established keyword density tools out there, and ours often deviates significantly, which is a clear indicator that something isn't quite right.

So, we're really looking for some community wisdom here. Specifically, we're wondering:

• What's considered the most robust and accurate way to extract *only* the visible, relevant text from a webpage for keyword analysis? This is especially tricky when dealing with modern web structures like SPAs (Single Page Applications), pop-ups, or heavily templated sites where content might be injected dynamically.
• Are there any common pitfalls we might be overlooking when calculating keyword density? We're mostly focused on exact matches first, but should we be thinking more about handling stop words, stemming, or lemmatization even at this initial stage to improve accuracy?
• Does anyone have recommendations for specific libraries, APIs, or even external services that specialize in highly accurate content extraction specifically for on-page SEO analysis? We're open to exploring new avenues if it means getting reliable data.

Has anyone faced similar data accuracy challenges with their web tools or on-page SEO analysis features? Any insights or suggestions would be massively appreciated!