Inconsistent Keyword Density Calculations for Dynamic Content in Our Web Tool During On-Page Optimization

Our 'Keyword Density & Frequency Checker' web tool is a critical component for many users focused on effective on-page optimization. It's designed to provide precise insights into keyword distribution, which is fundamental for content strategy.

However, we're encountering significant challenges in obtaining consistently accurate keyword density and frequency data, particularly for modern web pages that heavily rely on client-side JavaScript rendering and AJAX-loaded content. Our current backend, which primarily leverages BeautifulSoup for static HTML parsing, is proving woefully insufficient for dynamic content. Initial attempts to address this with headless browser solutions, specifically Playwright, have introduced significant performance bottlenecks and unacceptably high resource consumption, making them impractical for the scale we need. This fundamental technical block in reliable web scraping of fully rendered content is severely impacting the utility of our tool for a growing segment of websites.

We are actively seeking robust, scalable, and highly efficient strategies or alternative libraries for extracting the fully rendered text content from complex JavaScript-driven websites. The goal is to feed this clean, complete text into our keyword analysis algorithms without the overhead currently associated with full headless browser instances. Has anyone in the community successfully navigated this specific web scraping challenge at scale, especially for an SEO-focused tool? Any insights on optimized headless solutions, alternative rendering engines, or clever parsing techniques would be invaluable.