Word Tokenizer and Vocabulary

Language models work with tokens, not raw text. A simple tokenizer lowercases the text and treats every run of letters a-z and digits 0-9 as a word; everything else (spaces, punctuation, other characters) separates words.

Build the vocabulary: the distinct words, sorted by how often they appear (most frequent first), with ties broken alphabetically. A word's position in this list is its token ID.

Implement build_vocabulary(text).

Input

text: Any text.

Output

Return the vocabulary as a list of words. Return an empty list if the text has no words.

เงื่อนไข

0 ≤ length of text ≤ 100,000

ตัวอย่าง

อินพุต:
text = "The cat sat on the mat. The end!"

ผลลัพธ์ที่คาดหวัง:
expected = ["the", "cat", "end", "mat", "on", "sat"]

อินพุต:
std::string text = "The cat sat on the mat. The end!";

ผลลัพธ์ที่คาดหวัง:
std::vector<std::string> expected = {"the", "cat", "end", "mat", "on", "sat"};

อินพุต:
string text = "The cat sat on the mat. The end!";

ผลลัพธ์ที่คาดหวัง:
string[] expected = new string[] {"the", "cat", "end", "mat", "on", "sat"};

อินพุต:
text = "AI, ai and A.I. (2024)"

ผลลัพธ์ที่คาดหวัง:
expected = ["ai", "2024", "a", "and", "i"]

อินพุต:
std::string text = "AI, ai and A.I. (2024)";

ผลลัพธ์ที่คาดหวัง:
std::vector<std::string> expected = {"ai", "2024", "a", "and", "i"};

อินพุต:
string text = "AI, ai and A.I. (2024)";

ผลลัพธ์ที่คาดหวัง:
string[] expected = new string[] {"ai", "2024", "a", "and", "i"};

แหล่งอ้างอิง

โจทย์นี้ปรับให้เหมาะกับ CodeUtility กรุณาทำตามคำชี้แจงและข้อจำกัดด้านบน

• scikit-learn, Text feature extraction
• Jay Alammar, The Illustrated Transformer