Optimizing Keyword Density: Deep Dive into Tokenization Challenges for Accurate Text Analysis

I'm currently engaged in a significant enhancement project for our 'Keyword Density & Frequency Checker' tool, and I've hit a rather substantial technical hurdle concerning accurate text analysis. While our existing tokenization approach performs adequately for standard English text segments, its limitations become starkly apparent when dealing with more complex linguistic structures.

• The Core Problem: Our current methodology, which largely relies on simple whitespace or punctuation-based delimiters, struggles profoundly with multi-word keywords, hyphenated terms (e.g., "state-of-the-art"), and other nuanced linguistic patterns. This often leads to highly imprecise density and frequency counts, which directly impacts the utility of our tool for SEO professionals.
• Specific Technical Challenge: The issue manifests when our tokenizer inadvertently breaks down relevant keyword phrases or, conversely, incorrectly groups unrelated words. For instance, a phrase like "advanced SEO strategies" might be tokenized into individual words, losing its contextual value as a single keyword. This skewing significantly compromises the integrity of the overall "text analysis" we provide for content optimization purposes.
• Seeking Expert Algorithm/Library Recommendations: I am actively searching for robust algorithms or established NLP libraries that offer more intelligent and context-aware tokenization and phrase identification capabilities. Specifically, I'm interested in solutions that can effectively identify and count keyword phrases, intelligently considering n-grams, various stemming algorithms, and the dynamic handling of stop words within a given block of text.

Any insights into advanced NLP techniques, particularly those proven effective for this specific application in a production environment, would be incredibly helpful. I'm keen to hear from anyone who has tackled similar challenges. Waiting for an expert reply!