Curious about the methodology
Hey, big fan. I'm just curious about the end goal behind the methodology you're using here.
Task Arithmetic already subtracts the base from each component before doing the weighted sum operation.
Going off of Llama-3-Minus-Base-8B, this is how my intuition is parsing that config:
models:
# Finetune model
- model: meta-llama/Meta-Llama-3-8B-Instruct
parameters:
weight: 1.0
# Base model
- model: meta-llama/Meta-Llama-3-8B
parameters:
weight: -1.0
merge_method: task_arithmetic
base_model: meta-llama/Meta-Llama-3-8B-Instruct
parameters:
normalize: false
dtype: float32
In my head, the task vector produced by meta-llama/Meta-Llama-3-8B-Instruct - meta-llama/Meta-Llama-3-8B-Instruct is going to be a tensor of zeroes, proof by abstract nonsense etc.
The task vector produced by meta-llama/Meta-Llama-3-8B-Instruct - meta-llama/Meta-Llama-3-8B would be like an... "inverse task vector".
Flipping the sign on this as you're doing with weight: -1.0, and adding it to a tensor of zeroes should, then, just produce a task vector equivalent to meta-llama/Meta-Llama-3-8B-Instruct - meta-llama/Meta-Llama-3-8B, which you're then adding meta-llama/Meta-Llama-3-8B-Instruct to.
Am I misunderstanding something? Or if I'm not, I'm curious as to what this gets us that simply doubling the weight on a standard Task Arithmetic merge wouldn't?
I'm not looking to criticize, I'm just a little lost trying to get my head around all this haha
Don't worry about wrapping your head around it. The reason I did the whole negative weight TA steps was 'cus my scatter brained ass forgot that LoRA extraction was a thing.
I'm eventually going to create a merge that uses LoRA extraction, but I want to learn unsloth so I can add training steps to fix damage done from merging.
Hahaha, I've been there!
LoRA extraction is an interesting one; recently my friend Fizz did a whole blog post on how LoRA extraction can be very disruptive, in the context of an inference backend using this as a hot-swapping method.
Basically with models trained using LoRA it's nearly perfect, but when they're not (or when they're merges, a thing I'd like her to get into some more) a lot gets lost; occasionally results align more with the inverse of the model's values.
Not to say that LoRA extraction is useless per se, we're absolutely not above ripping and tearing chunks out of models and gluing them someplace else, but I'd expect it to be incredibly stochastic. (But like, what isn't, right? Merging is alchemy of the most brutal form. And LoRA extraction created Sunrose, which was a critical component of my own Mag Mell, so I know for sure that there's value in it.)
At any rate, my main confusion was "isn't this just reinventing Task Arithmetic" but now that you've brought up LoRA extraction, and I've had a chance to talk it out with some folks, I do actually see what you were getting at, I've had ideas in a similar direction, and I'm curious to see where it ends up.