Keyword density content optimization bug

hey everyone, we're running into a wall with our keyword density & frequency checker tool, specifically how it processes content for accurate content optimization metrics. it's been a real headache for the past few weeks and we're kinda stuck.

the main issue is robustly extracting only visible, semantic text from complex HTML docs. our current parser, while decent, often includes text from script tags, style blocks, or even hidden divs that aren't actually part of the user-visible content. this naturally skews our density calculations pretty badly, which is critical for proper content optimization analysis. we need that pure, readable content.

we’ve tried a few things to tackle this. initially, we went with a simple regex approach to strip common tags, but that was just too crude and brittle. then, we moved to a more sophisticated DOM parser, specifically cheerio in node.js, to traverse the tree and pick out text nodes. but even with cheerio, distinguishing between actual body content and things like alt attributes or title tags on non-textual elements, or even text within noscript blocks that isn't really 'content' for content optimization purposes, is proving to be a nightmare. it's like chasing ghosts sometimes.

large pages, especially those built with modern JS frameworks like React or Vue, are particularly problematic. the DOM structure can be incredibly deeply nested, and trying to filter out boilerplate stuff like navigation menus, footers, or sidebars while retaining *only* the actual article body content is where it consistently fails us. we need to precisely identify what constitutes 'body content' that an actual user would read for true content optimization.

so, we're looking for battle-tested strategies or perhaps specific parsing libraries/algorithms that can intelligently extract only user-visible, semantic text for keyword density analysis. how do you guys handle this without introducing massive overhead or sacrificing accuracy, especially for content optimization tools operating at scale? we're talking about about potentially millions of pages.

any insights from folks who've built similar content optimization tools would be super helpful. waiting for an expert reply.