Условие
Language models work with tokens, not raw text. A simple tokenizer lowercases the text and treats every run of letters a-z and digits 0-9 as a word; everything else (spaces, punctuation, other characters) separates words.
Build the vocabulary: the distinct words, sorted by how often they appear (most frequent first), with ties broken alphabetically. A word's position in this list is its token ID.
Implement build_vocabulary(text).
Input
text: Any text.
Output
Return the vocabulary as a list of words. Return an empty list if the text has no words.
Ограничения
0 ≤ length of text ≤ 100,000
Примеры
Ввод:
text = "The cat sat on the mat. The end!"
Ожидаемый вывод:
expected = ["the", "cat", "end", "mat", "on", "sat"]Ввод:
std::string text = "The cat sat on the mat. The end!";
Ожидаемый вывод:
std::vector<std::string> expected = {"the", "cat", "end", "mat", "on", "sat"};Ввод:
string text = "The cat sat on the mat. The end!";
Ожидаемый вывод:
string[] expected = new string[] {"the", "cat", "end", "mat", "on", "sat"};Пояснение
Lowercased, the words are the, cat, sat, on, the, mat, the, end. "the" appears 3 times; the rest appear once and are sorted alphabetically: ["the", "cat", "end", "mat", "on", "sat"].
Ввод:
text = "AI, ai and A.I. (2024)"
Ожидаемый вывод:
expected = ["ai", "2024", "a", "and", "i"]Ввод:
std::string text = "AI, ai and A.I. (2024)";
Ожидаемый вывод:
std::vector<std::string> expected = {"ai", "2024", "a", "and", "i"};Ввод:
string text = "AI, ai and A.I. (2024)";
Ожидаемый вывод:
string[] expected = new string[] {"ai", "2024", "a", "and", "i"};Пояснение
Words: ai, ai, and, a, i, 2024. "ai" appears twice; the others once, sorted alphabetically with digits before letters: ["ai", "2024", "a", "and", "i"].
Ссылки
Это задание адаптировано для CodeUtility. Следуйте условию и ограничениям выше.