Does BERTA support matryoshka dimensions?
#2
by dantetemplar - opened
Hello, can't find information on that - FRIDA, BERTA, others models do not declare matryoshka representation, but it looks I have no loss when truncate dim up to 384.
This is indeed an interesting observation. Although BERTA were not trained using Matryoshka Representation Learning (MRL), using a vector truncated by 2–3 times shows almost no drop in accuracy for most tasks. In my own testing, I tried various truncation methods (such as [:384], [384:], [0::2], and [1::2]) and did not observe any significant degradation either.
sergeyzh changed discussion status to closed