Byte Pair Encoding (BPE) Merges

GPT-style tokenizers use Byte Pair Encoding: start from single characters and repeatedly merge the most common pair of neighbors into a new symbol.

1. Split every word in words into characters. Repeated words count multiple times.
2. Count every pair of adjacent symbols across all words.
3. Pick the most frequent pair. If pairs tie, pick the one that is smallest when you compare (first symbol, second symbol) as strings.
4. Replace every occurrence of that pair, left to right, with the merged symbol, and record the merged symbol.
5. Repeat num_merges times, or stop early when no word has two symbols left.

Implement bpe_merges(words, num_merges).

Input

words: The training words (lowercase letters).

num_merges: How many merges to learn.

Output

Return the merged symbols in the order they were learned.

เงื่อนไข

1 ≤ number of words ≤ 1,000, 1 ≤ length of each word ≤ 30

0 ≤ num_merges ≤ 200

ตัวอย่าง

อินพุต:
words = ["low", "lower", "lowest"]
num_merges = 2

ผลลัพธ์ที่คาดหวัง:
expected = ["lo", "low"]

อินพุต:
std::vector<std::string> words = {"low", "lower", "lowest"};
long long num_merges = 2;

ผลลัพธ์ที่คาดหวัง:
std::vector<std::string> expected = {"lo", "low"};

อินพุต:
string[] words = new string[] {"low", "lower", "lowest"};
long num_merges = 2;

ผลลัพธ์ที่คาดหวัง:
string[] expected = new string[] {"lo", "low"};

อินพุต:
words = ["aaa"]
num_merges = 3

ผลลัพธ์ที่คาดหวัง:
expected = ["aa", "aaa"]

อินพุต:
std::vector<std::string> words = {"aaa"};
long long num_merges = 3;

ผลลัพธ์ที่คาดหวัง:
std::vector<std::string> expected = {"aa", "aaa"};

อินพุต:
string[] words = new string[] {"aaa"};
long num_merges = 3;

ผลลัพธ์ที่คาดหวัง:
string[] expected = new string[] {"aa", "aaa"};

แหล่งอ้างอิง

โจทย์นี้ปรับให้เหมาะกับ CodeUtility กรุณาทำตามคำชี้แจงและข้อจำกัดด้านบน

• Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units
• Hugging Face course, Byte-Pair Encoding tokenization