Multi-Head Self-Attention प्रोग्रामिंग चैलेंज

समस्या

Multi-head attention lets a Transformer attend to different things at once. With input X (n tokens × d columns) and weight matrices of size d × d:

Q = X × w_q, K = X × w_k, V = X × w_v
Split the d columns into num_heads heads of d_k = d / num_heads consecutive columns (head h uses columns h×d_k to (h+1)×d_k - 1).
For each head, run scaled dot-product attention on its columns: for token i, score_j = (Q_i · K_j) / √d_k over that head's columns, softmax over all n tokens j, and output = Σ weight_j × V_j (that head's columns).
Put the heads' outputs back in their columns (concatenate) and multiply by w_o.

Implement multi_head_attention(x, w_q, w_k, w_v, w_o, num_heads).

Input

x: n × d input matrix.

w_q, w_k, w_v, w_o: d × d weight matrices.

num_heads: Number of heads (d is divisible by it).

Output

Return the n × d output matrix.

सीमाएँ

1 ≤ n ≤ 30, 1 ≤ d ≤ 32, 1 ≤ num_heads ≤ d and d is divisible by num_heads

-5 ≤ each value ≤ 5

उदाहरण

उदाहरण 1

इनपुट:
x = [
    [1.0, 0.0],
    [0.0, 1.0]
]
w_k = [
    [1.0, 0.0],
    [0.0, 1.0]
]
w_o = [
    [1.0, 0.0],
    [0.0, 1.0]
]
w_q = [
    [0.0, 0.0],
    [0.0, 0.0]
]
w_v = [
    [1.0, 0.0],
    [0.0, 1.0]
]
num_heads = 1

अपेक्षित आउटपुट:
expected = [
    [0.5, 0.5],
    [0.5, 0.5]
]

इनपुट:
std::vector<std::vector<double>> x = {
    {1.0, 0.0},
    {0.0, 1.0}
};
std::vector<std::vector<double>> w_k = {
    {1.0, 0.0},
    {0.0, 1.0}
};
std::vector<std::vector<double>> w_o = {
    {1.0, 0.0},
    {0.0, 1.0}
};
std::vector<std::vector<double>> w_q = {
    {0.0, 0.0},
    {0.0, 0.0}
};
std::vector<std::vector<double>> w_v = {
    {1.0, 0.0},
    {0.0, 1.0}
};
long long num_heads = 1;

अपेक्षित आउटपुट:
std::vector<std::vector<double>> expected = {
    {0.5, 0.5},
    {0.5, 0.5}
};

इनपुट:
double[][] x = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
double[][] w_k = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
double[][] w_o = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
double[][] w_q = new double[][] {
    new double[] {0.0, 0.0},
    new double[] {0.0, 0.0}
};
double[][] w_v = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
long num_heads = 1;

अपेक्षित आउटपुट:
double[][] expected = new double[][] {
    new double[] {0.5, 0.5},
    new double[] {0.5, 0.5}
};

व्याख्या

w_q is all zeros, so every score is 0 and each token averages the value rows: V = X, so both rows become [0.5, 0.5].

उदाहरण 2

इनपुट:
x = [
    [2.0, 4.0]
]
w_k = [
    [1.0, 0.0],
    [0.0, 1.0]
]
w_o = [
    [0.0, 1.0],
    [1.0, 0.0]
]
w_q = [
    [1.0, 0.0],
    [0.0, 1.0]
]
w_v = [
    [1.0, 0.0],
    [0.0, 1.0]
]
num_heads = 2

अपेक्षित आउटपुट:
expected = [
    [4.0, 2.0]
]

इनपुट:
std::vector<std::vector<double>> x = {
    {2.0, 4.0}
};
std::vector<std::vector<double>> w_k = {
    {1.0, 0.0},
    {0.0, 1.0}
};
std::vector<std::vector<double>> w_o = {
    {0.0, 1.0},
    {1.0, 0.0}
};
std::vector<std::vector<double>> w_q = {
    {1.0, 0.0},
    {0.0, 1.0}
};
std::vector<std::vector<double>> w_v = {
    {1.0, 0.0},
    {0.0, 1.0}
};
long long num_heads = 2;

अपेक्षित आउटपुट:
std::vector<std::vector<double>> expected = {
    {4.0, 2.0}
};

इनपुट:
double[][] x = new double[][] {
    new double[] {2.0, 4.0}
};
double[][] w_k = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
double[][] w_o = new double[][] {
    new double[] {0.0, 1.0},
    new double[] {1.0, 0.0}
};
double[][] w_q = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
double[][] w_v = new double[][] {
    new double[] {1.0, 0.0},
    new double[] {0.0, 1.0}
};
long num_heads = 2;

अपेक्षित आउटपुट:
double[][] expected = new double[][] {
    new double[] {4.0, 2.0}
};

व्याख्या

With a single token each head attends only to itself, so the concatenated output is V = [2, 4]. w_o swaps the two columns: [4, 2].

संदर्भ

यह चुनौती CodeUtility के लिए अनुकूलित की गई है। ऊपर दिए गए स्टेटमेंट और सीमाओं का पालन करें।

Multi-Head Self-Attention

आप क्या सीखेंगे

समस्या

Input

Output

सीमाएँ

उदाहरण

व्याख्या

व्याख्या

संदर्भ

समुदाय समाधान