so in my slides here I'm going to talk
about debugging and understanding NLP
models and this is how to tell uh when
for example both your implementations
are wrong and uh for example your
underlying assumptions are wrong or your
model is failing on particular segments
of data or stuff like that so going to
go through uh a variety of things that
can go wrong with your experiments
basically so a typical situation is
you've implemented some NLP system you
know based on neural networks of course
because that's what we use nowadays um
and you've looked at the code it
basically looks okay um but it has low
accuracy or it makes incomprehensible
errors and you would like to uh fix
these or you'd like to improve the
accuracy or something like that and so
what do I
do and I think there's three dimensions
of how you can understand your model and
your Model Behavior um the first one is
debugging the implementation so it's
identifying problems that you have when
you uh implemented something uh second
thing is actionable evaluation so
identifying typical error cases and how
you what you can do to fix them and
finally uh interpreting predictions or
interpreting what's happening inside the
model and uh this can maybe give you a
deeper idea about what's happening in
happening in individual cases and
there's a lot of reasons why you might
want to do that uh both like to make
your models better and also for example
if you want to be sure that your ition
isn't doing something illegal like
discriminating against people uh due to
protected attributes or other things
like that so um there's a number of
reasons why you'd want to do that so I'm
going to talk about the first two and
Nishant is mainly going to talk about
the second one so uh going right into
it so in neural network models uh
debugging is really important because
they're opaque they're unpredictable and
uh if you make little mistakes they can
cause big problems with your
output and another thing is that
everything is a hyperparameter including
your network size your model variations
your batch size your strategy your
Optimizer and your learning rate
and finally unlike kind of more
traditional machine learning methods
like logistic progression or support
Vector machines or something like that
you might that you might have studied in
your machine learning class um
stochastic optimization has no guarantee
about convergence um your loss might go
down then it might go up and there might
be absolutely nothing wrong with your
training or it might be you know a
serious problem so that's another issue
you need to deal
with so first I'd like to go into
possible causes of problems with your
implementation and I'm going to break
them down into a typology and based on
what part of the typology you're running
into problems with you will need to fix
them in different ways so your first
goal when you're experiencing the
problem is identifying why you're
experiencing the problem uh because that
will lead you to a
solution so for training time problems
there's a bunch of uh things that could
be wrong uh the first is a lack of model
capacity so your model is not able to
model the phenomena that you want to
model in the first place um you could
have a poor training
algorithm uh you could just have a bug
in your code at training time another
thing is uh test time problems and these
can include a disconnect between what
you're doing at training time and what
you're testing at testing time uh
failure of search
algorithms and another thing you want to
deal with is overfitting so you're
actually doing well on the training set
but you're doing poorly on the test
Set uh finally you could have um optimiz
a mismatch between the function you're
optimizing at evaluation time and uh
what you're actually evaluating sorry
the fun the function that you're
optimizing at training time and what
you're actually evaluating at test time
and my my best piece of advice for
figuring out why things are going wrong
is don't uh try to do all of them at
once and rather uh start from the top
and work it down because the ones at the
top are often easier to uh diagnose and
the ones at the
bottom so looking at how you can debug
systems at training time uh there's a
number of ways you can do this uh but
the most important thing for training
time uh issues is looking at the loss
function calculated on the training set
and what I mean by this is don't look uh
we talked about how we can't optimize
error or accuracy easily so instead we
optimize likelihood um and so you might
want to look at accuracy to see whether
your model is working well but I would
urge you first to look at your
likelihood or your loss function on the
training set instead of your accuracy on
the test set for example to diagnose
these variety of
problems and the sorts of things you
want to look at are um is the loss
function going down so is it you know
converging into a good place
um in general if this is your your
loss um the first thing you should know
is like what is a good loss uh in most
cases a good loss is zero like log
likelihood the best loss you can achieve
is zero so you have zero down here um
something
like
this is uh essentially a good loss
function something like that uh
especially if this is a relatively High
number is usually a bad loss
function
um something like that on your training
set is a very bad loss function uh
something something is going seriously
wrong if you see this on your Dev set
that could be or your test set that
could be uh overfitting but but if
you're seeing that on your training set
that's usually symptomatic of a problem
so uh these are uh things that you
should be uh knowing um is it going down
basically to zero if you run training
long enough um for many epochs over your
training data so if it's not going down
to zero and it's sticking up here then
that's also an
issue and um if it's not going down to
close to zero on whatever training set
you're training on um let's say you make
your training set extremely small
uh at least in that case it should go
down to zero otherwise you might have a
serious problem in your
implementation so these are good things
to check first when you're training a
model um and there's a number of reasons
why this might not be helping or why
this might not be happening so um your
Mo model might be too weak and so in
general larger models tend to perform
better uh especially if you're using a
pre-trained model and um this is just an
example from the T5 paper where they
scale up the T5 model um from a
relatively small model to what at the
time was a very large model of 11
billion parameters now this is you know
a moderately sized model or maybe even
small model by some standards but anyway
you can see that it uh in continues to
increase one really interesting
phenomenon is uh that actually larger
models can learn faster or at least with
fewer steps than uh smaller
models and so this
is an interesting example this paper uh
on neural scaling was it's a very
influential paper but basically what
they show is the darker purple ones are
smaller models the yellow ones are
bigger models and what you can see here
is the purple model and on the left side
they have the number of tokens processed
the right side they have the number of
uh compute or the amount of compute um
and so what you can see is if you just
look at the number of tokens processed
the larger the model the faster it
converges which
is maybe a little bit surprising maybe a
little bit you or maybe uh like some
people have the intuition that this
should be the case but when I first saw
this I found it a little bit surprising
because I thought it would be so large
and noisy that the model would have
trouble fit
you know fitting the data as quickly but
there's actually a good reason for this
does anyone have a guess about why this
is
thee we've talked a little bit about the
underlying phenomena for this in
previous classes so you might be able to
think back to some of the things you
yeah yeah so um just to repeat there's a
lot of different parameters so it can
try to converge along a lot of different
dimensions so if we think back to the
like model pruning class and other stuff
like that um part of the reason why we
can prune large models so efficiently is
because only like a small number of the
parameters are actually useful and so if
you start out with a much larger model
it's more likely to have useful subsets
of the parameters basically um which is
called the lottery ticket hypothesis uh
there there's a famous paper called the
lottery ticket hypothesis examines this
phenomenon so um one one interesting
thing is you can see that even if you
scale up the compute even if you measure
based on compute the uh larger models
eventually surpass the smaller models in
terms of how efficient they are at
modeling the data and that's just
because models tend to learn well for a
while and then they basically reach
their capacity and stop learning well or
they start learning very slowly and once
you get to that point the larger models
work better so there's a kind of
counterintuitive thing that if you want
to train faster you actually can train a
larger model and uh that will that will
uh get you to a good solution at some
point that will get you to a good
solution faster than a smaller model
would you know of course you need memory
and stuff but why are looking
at so this is test loss training loss
also looks like this um I think on
this particular
on this particular paper they never
repeated data and if you never repeat
data actually your training loss looks
very similar to your test loss because
it if you like actually if you can
assume your training data set and your
test data set are um uh identically
distributed your training loss of new
training data should be exactly the same
as your test loss so I think that's
basically why they were justified in
doing that good but they probably did
test loss to like I swashed the concern
that this was overfitting comp or
something but good
question um cool so the these are are
good things to know um so basically if
you see your model doing something like
this um plateauing out maybe your
model's too small and you need to tr a
big
basically another uh piece of trouble
that you can have is trouble with
optimization and basically um you should
check your Optimizer um usually people
are using atom variants nowadays like
atom or atom W so just use that um
learning rate uh so make sure that the
learning rate you're using is standard
for kind of the model size that you're
using and the best way to do this is uh
look at previous papers and see what
they're
using um initialization most people
nowadays will not be training from
scratch but if you are training from
scratch how you initialize your model is
really important and normally the way
you do this is you do this with some
sort of uniform random noise and uh
specifically you can pick the uniform
random noise in intelligent ways based
on the the data size which I'll talk
about in a second um also mini batching
um are you using sufficiently large
batches of data if you're using small
batches of data you might have too much
noise in your training and it might
diverge so uh these are things you need
think about as
well
cool um so these are training time
things um the next thing is debugging at
test
time and this is particularly important
if you're doing any sort
of like I guess a lot of this has kind
of been commoditized and it's
implemented in hugging face and stuff
like that and as long as you're using
the standard implementations you're less
likely to run into these bugs but if you
are implementing anything on your own
this is actually really tricky and you
can easily make mistakes so uh it's
important to to know about it so um what
one of the reasons why you can have
training and test disconnects especially
if you're doing something like text
generation is that usually your loss
calculation and prodiction functions
will be implemented in different
functions and like anything in software
engineering Um this can be a source of
bugs duplicated sour code can be a
source of bugs because you might
Implement one thing in one place in one
way another thing in another place in
another way so this is no exception to
that um it's especially true for
structured prediction models so anything
where you're not just making a single
prediction but you're making multiple
predictions in a row so you need to be a
little bit careful about
that um another thing that you need to
be pay attention about is often uh
especially if you're doing your own
implementation loss calculation it's
mini batched and generation is not or in
highly optimized versions of um of
inference you might be doing inference
with Dynamic batching and stuff like
that and it might become complicated you
might make
mistakes um so how do
we make sure that we're not making any
mistakes here um there's a really simple
way to debug any sort of mini batched
loss calculation because normally when
we mini batch loss calculations we're
simultaneously calculating uh the loss
for like uh four four or eight or
whatever sequences at a time and so you
can calculate the loss with a large
batch size like 32 and then calculate
the loss for each uh sentence
individually and sum them together and
these uh value should be the same and
this can help make sure that you don't
have any you know issues with your
padding or your masking or other things
like this um so this is particularly
important if you're not just using out
of the box things so you have a slightly
unusually structured model with like
hierarchical encoding or anything like
that you need to be really careful about
that um you can even create unit tests
that test this so like um in machine
learning code we don't write unit test
or especially neural network based
machine learning code we don't write
unit tests that often because it's kind
of hard to do there's lots of Randomness
and other stuff like that um but this is
one thing that you can easily test and
and make sure that you don't hear the
mistakes um any sort of uh generation
algorithm uh so when you're generating
or decoding um you can make sure that
your decoding code is getting the same
score is when you calculate the loss and
an easy way to do this is you call the
decoding function to generate an output
and normally when you're doing any sort
of search or sampling or something like
that during the search or sampling
you're calculating the logits or the log
probabilities of each step that you
sample so you keep track of that during
your sampling
algorithm and then after that you call
the loss function on the generated
output and you calculate the loss
according to the loss function and the
score of these two things should be the
same uh
so um you know you do your
generate and that gives you an
output in
score and then you do um
loss on the
output and that gives you the score
two and then you just compare these two
things together and this can uh in in my
experience this has allowed me to find
the majority of the bugs in um these two
things um have allowed me to find the
majority of the bugs whenever I was
doing any sort of like complex thing
with respect to generation or models and
stuff like that so um it's a very common
place for bugs even if you're pretty
familiar with models so I I would highly
recommend
that um this is particularly bad when
you're doing something like a search
algorithm like beam search um and
so beam search uh as you know from the
generation class instead of picking one
high probability uh you know word in
your next step you maintain several
paths and one way that you can fix this
is as you make search better the model
score should get better so the log
likelihood of the output should get
better almost all of the time so you can
search with varying beam sizes and make
sure that you get a better overall model
score at the end so um and you can even
create a unit test testing this as well
I don't think that that many people will
be reimplementing beam search so you
might not need to worry about that too
much but in case you are doing anything
with respect to search algorithms it's a
good thing to
know
cool um any questions about these two so
far no okay um so the second the next
thing I want to talk about this is
something that people think about a
little bit less uh but it's actually
something really important to know
because it will affect you it will
affect everybody uh to some extent it
will affect you to a greater or lesser
extent depending on um what uh type of
you
know system you're building but it will
definitely affect everybody and that's
the mismatch between the the function
that you're optimizing at training time
and the evaluation metric that you're
evaluating and
so uh like as I said in the
reinforcement learning class it's very
common to optimize for maximum
likelihood for training uh but there's
all kinds of problems with this you know
um with respect to the mistake it not
being sensitive to mistakes it not being
sensitive to your generation
algorithm um but even though your
likelihood is getting better accuracy
can get worse and this is a super simple
example with uh image classification on
mest and I I ran this experiment with
like 10 lines of pytorch code or
something like this uh maybe more like
40 lines of P um and so here um on the
left side we have the loss on the
training set and the test set or the dev
set and here we have accuracy on the
training set in the test
set
and so oops I showed you the answer so I
was going to do a quiz but I
accidentally showed you the answer um
but the problem here is basically
because um the the loss you're
calculating the likelihood of the
correct answer and the likelihood of the
correct answer is the probability of
getting the correct
answer the accuracy is the number of
times you're getting the correct answer
so as you train a model to get more and
more confident it gets better it gets
better and better at getting more
answers correct but it also gets more
and more confident in its answers and so
if the you know there's any example that
it's really bad at um it might get very
confident in
that answer that bad answer and the log
likelihood of that answer will go up or
sorry the log likelihood will go down so
the negative log likelihood will go up
is the loss so basically
um the
uh the loss that you're calculating and
the thing that you care about in the end
accuracy can be decorrelated
um so there's also an interesting
example um in text generation and this
is part of the reason why uh we have all
these other text generation algorithms
like nucleus samp playing or topk samp
playing or other things like this is um
actually in a maximum likelihood trained
model better
search uh in in other words finding a
better model scope
doesn't necessarily give you a better
generation result and this is an example
uh from machine translation from a
really long time
ago uh but you know it still persists
today which is they did beam search with
a larger and larger beam
and the be the best Beam for finding um
the best scoring output basically was
four and then the accuracy goes down and
down and down as they find a better
output and does anyone remember when we
talked about the generation class where
this comes
from I don't know how explicitly we said
we mentioned it in the generation class
but basically the problem is um maximum
likelihood train models like shorter
outputs generally because if as we make
the output longer uh the probability of
the longer outputs goes down so as you
improve the beam it will start
generating shorter and shorter outputs
and because of that the score goes down
because blue score doesn't like outputs
that are too short essentially so there
are um there are hex around this for
beam search where essentially what you
do is you uh take the average log
likelihood of each token instead of the
overall log likelihood of each token um
and that improves a little bit but still
you can see as you search more the the
accuracy goes down so um so that's the
the general idea
here there's a bunch of ways you can fix
this um the most principled way is to
use a method like reinforcement learning
or something uh some sort of you know
structured training algorithm that
allows you to train your models so that
you don't get these bad
outputs um another way that's much
easier is to do early stopping with the
evaluation metric as opposed to um early
stopping with the loss and by doing this
you would stop here so you would stop
where you get the highest evaluation
metric uh that you care about instead of
stopping here uh so that's um that's one
way you can fix this
problem does anyone have an idea about
why this might be a bad
idea why might it be a bad idea to stop
here instead of stopping here for
example
yeah it's kind of overfitting it's
overfitting in a particular way um but
remember here this is still the accuracy
on the dev set so we're not overfitting
so much that the dev accuracy is going
down that would be a different variety
of overfitting but any any
ideas go for it we don't want to be too
confident yeah exactly we don't want it
to be too confident in its wrong answers
and we talked about
calibration um where calibration is
basically like how accurate are the
probability estimates so this model over
here is going to be really poorly
calibrated it's going to be very
confident regardless of whether it's
correct or not and that could be a
problem in dopram uh dopram tasks
there's also another thing that I I
forgot to put on
[Music]
um
that I forgot to put on the slides but
it's a um an interesting phenomenon that
actually um kind of a lot of people in
interpretability are interested in it's
this uh generalization gring
generalization Beyond overfitting on
small algorithmic data sets and
basically what they
show is um you can be training for a
very very long time
um
and uh like reducing the loss reducing
the loss reducing the loss and reducing
the loss and it's only after a very long
time does your Model start generalizing
well and getting good
accuracy um the this paper the types of
data sets it's talking about are data
sets where you need to get many things
in a row correct before you get the
final answer correct so basically you
need to get like 20 steps in a row or 50
steps in a row correct before you get
the final answer correct and um
basically the reason why this happens is
because this accuracy will keep going up
but you only get the accuracy of each
individual decision will keep going up
but you only get marked like
correct uh
after you get like all 50 in a row
correct so um it this difference can be
even more Stark when you're talking
about things that require like 50 steps
of reasoning or like multiple steps of
reasoning but like 50 token Generations
correct before you get them right so um
that's another thing to be aware
of cool um so now I want to switch gears
a little bit to actionable evaluation
and how you can um evaluate your models
in a way that makes it easy to find uh
next steps to be
improving uh are there any questions
about the debugging part before we get
into this
part okay I'll
go so um my first suggestion with
respect to how you can actually you know
improve systems is make sure that you're
looking at the data that you're
using and um both bugs and new research
directions can be found by looking at
your model
outputs um
so to give one example um of a very
common mistake that you can make when
you're creating a a generation algorithm
it's these sort of off by one erors um
so like let's say you implemented a
translation system and it's generating
outputs like went to the store yesterday
bought a dog um you can immediately look
at this and say hey this doesn't look
like natural English what's going uh
what's going on and the the problem here
is
you're um you're doing something
like output uh
one uh and you have a slice of like one
instead of zero here or something like
this and so this is a really silly error
that you might just make a mistake on
python on your you know pre-processing
or postprocessing or something like this
um but the problem is like if you look
at your blue score based evaluation or
something like that you'll have like
you'll be one point worse or two points
worse or something like that and you'll
be like Oh I'm I'm two points worse why
am I two point wor two points worse in
the state of the art and it turns out it
was a really like silly thing like this
and immediately you'll see this if you
look at your data but if you're doing
all your experiments and just looking at
the numbers it's really hard to tell you
know why this is
happening
um another thing is uh if you
have a good eye and can like just look
through the data points
um we as humans are pretty good uh
pattern recognizers and especially you
know CMU students uh you're uh very good
and quick at picking up on things so if
you look at the data and pour through
things you can uh probably pick up
patterns about why things are failing
and so um you know you might look and
see that uh compared to some other model
your model is really bad at answering
questions about people or something like
that and then you figure out you'll need
a better model of uh people or your rag
systems uh that you're building for
assignment two is maybe failing on all
the research related questions so you
need to come up with the research uh
like scrape more research data or
something like
that
um so there are methods to do this more
systematically and this is something I
picked up when I was doing an internship
at Google and it really stuck with me
for you know 14 uh 14 years now I guess
13 years um so uh a very simple way to
do this more systematically than just
browsing through things is to randomly
sample a 100 outputs and look at a 100
erors and try to group them into some
sort of typology and say oh uh this kind
of air is particularly
frequent and this is just one example of
a typology that was defined by V at all
um where they tried to take machine
translation errors and group them into
uh various varieties like correct words
filler words local uh local range long
range um uh sorry word word level word
ordering erors local range long range
phrase level local range long range and
stuff like this um you can definitely
look at previous work and see the
typologies of errors that they used but
the problem is like systems get better
and actually I don't think this is a
super relevant typology for machine
translation anymore uh because machine
translation systems like they don't make
a whole lot of local range Word level
errors anymore and rather we might want
to know more fine grain like are they
making mistakes on named entities or
other things like that so actually
we
um
did a re a more recent thing it's I
guess four years ago now um but it was
when uh people first started saying that
machine translation systems are about as
good as humans at doing a
translation and when we did this we
compared we compared machine translation
systems to humans and we tried to find
you know different types of things and
we were inspired by V but we recreated
our typology based on the things that we
thought were you know the most important
types of errors in like 2020 instead of
2006 so this is really helpful the
reason why it's really helpful is if you
can do this even for a small sample of
the outputs that you're looking at and
identify the most like prominent types
of eras that you're facing it often
leads you to the most successful ways of
improving the accuracy of your systems
because you might if you don't do this
you might be focusing on an air type
that's not actually an error it's kind
of like if you learned in uh programming
you know software engineering or
something like that you should never
optimize your code until you run a
profiler um because actually your code
might be slow in a place that you never
expected and so it's kind of the same
principle here right so don't optimize
your systems errors in a place uh where
like actually it's not having in years
so um that's a general principle
here uh cool another thing you can do is
quantitative analysis so um if you can
think of the phenomenon that you choose
to focus on um is that phenomenon
getting better so if you focused on uh
something that should improve the
quality of low frequency words uh you
can check if the accuracy on low
frequency words is increasing if you
focused on something that should be
improving the syntax in a low resource
language you can measure um whether it's
doing better on word ordering or uh long
distance
dependencies um if you focused on
improving a search algorithm for you
know generation or something like that
are the number of search errors that
you're encountering being reduced so
depending on what you planned on uh you
know improving it's often a good idea to
measure more directly whether it's
improving the the thing that you think
it should
improve um one example of um so I I
basically
created since my experience doing this
manually uh when I I was on an
internship at Google um I've
gradually improved my methodology for
doing
this and um and worked on automating
things and
so the first thing I had was a super
hacky uh hacky script that basically
writes out HTML
files um and then I I had something
called explainer board where we had a
leader board and uh recently one of the
things I've worked on is uh this uh
together with um Alex Alex Cabrera who's
a student here um is this toolkit called
Zeno and um this is just an example from
machine translation
it's being a little bit
slow um but basically what it does is it
allows you to look at the data on the
right side um and so these are just
examples um but you can go in and do
things like say Okay I want to look at
all machine translation examples
from
uh housea and so it shows you the ones
from housea I want to look
at all
examples let me clear that off I want to
look at all examples where the accuracy
is
low um and so now I can look at all the
examples where the accuracy is low and I
I can go in and uh uh examine them so uh
you can also go in and build charts like
this so like what is the overall
performance um what is is the
performance what is the performance
um on different scripts so you can see
which model which model is doing better
at scripts and stuff like that so um or
you can put things side by side and say
okay I want to find all the examples
where uh chat GPT is doing much worse
than GPT
4 uh or like GPT 3.5 is doing much worse
than gp4 and here we can see that oh in
this case it's generating something in
the wrong script or something like that
so um there's also tooling that you can
use to make this easier as
well and the way uh the way you use this
is you basically
um uh create a pandas data frame with
all of your data in it and you upload
the pandas data frame with any metadata
you want to use and you can uh use and I
think VJ will be having a recitation on
this if you're interested in taking a
look cool um so that is the my part and
then we'll be doing Nishant next while
Nishant comes up to set up are there any
questions about the thing that I talked
about
here yeah
so that when I
regular um
is that does that make a difference in
terms of like what we're expecting when
we're evaluating the model
model yeah so just to repeat the
question it's a a great question so if
you apply
regularization um will that change the
overall expectation for the model loss
so I was saying loss should converge to
zero once you start applying
regularization or weight Decay or
something like that it definitely might
not converge to Z um and the reason why
is because once you start applying
regularization there is no zero loss
solion um because in order to reduce the
loss you need to make move things away
move weights away from zero um but when
you move weights away from zero the
regularization L becomes n negative so
one thing you can do however is measure
the losses separately so measure the
regularization component of the loss and
the um
the log like we had component with the
loss and with any reasonable
regularization and a reasonably
parameterized model I do think the loss
should be getting closer to Zer like the
actual likely should be getting closer
to
zero uh you were using an extremely
small model in the mini L signed though
so that might make it more
difficult yeah and any other
things okay if not
I'll
all right can everyone hear
me sweet okay move this it looks like
I'm talking to someone instead of
between both of
you
all right so hi everyone um I'm going to
talk about model interpretability for
for those who don't know me I'm one of
your Tas I'm a first year PhD student
working with Mona diab on model
interpretability
um where what do I
click your your mouse should be there
yeah just
cool okay um
so what I want you to take away if you
if you fall asleep this is too boring
here are sort of the two main takeaways
one I want to convince you that model
interpretability is important to study
and two I want I want you to find this
interesting um and something you want to
explore more there's a bunch of details
here this is going to be kind of a
whirlwind tour you're not going to get
super deep into anything um so hopefully
this acts as a starting point um then
than anything
else so interpretability in AI um the
the definition is it's the study of
understanding the decisions that AI
systems make and putting them into
easily human understandable terms this
can mean a lot of different things and
this is often really hard um and the why
is to use that understanding to
iteratively better Design Systems that
are better they're more more performant
but also those that are more human
understandable
um so interpretability is this big blah
but there's a bunch of other uh spheres
that intersect with it this is a super
incomplete list uh so bear with me the
causality and data integrate with this
there's aspects that are interpretable
there's aspects that matter here um
explainable AI is another thing that
you've probably heard this sits firmly
in the interpretability blob and
connects with ideas and causality and uh
in data too um model interpretability
sits on this kind of other side of
things it intersects a little bit with
causality and explainable AI but uh is a
little bit separate for it um and from
it and mechanistic interpretability
which which you've probably heard of
it's gotten a lot of Buzz recently kind
of sits inside of model interpretability
it's a special case of model
interpretability I hope the mech people
agree with me
but um so yeah so historically we've
been dealing with really really really
small models you had Bas Nets this is a
this is very small model um if all these
are binary variables this is uh eight
total parameters and only four of which
are independent uh we also used to work
with linear regression a lot and in the
first case that's a nice line can be two
parameters the multivariate case again
that's a a small number of parameters
we've moved to of more things we've
moved to
MLPs that have larger weight matrices
but all these are kind of digestible and
interpretable so the interpretability
world was sort of uh not super concerned
with large ginormous things but we're
not there
anymore uh this is a language model this
is part of still part of a language
model now it's getting more and more and
more
hairing and this is just not
interpretable um I mentioned
on on the first day of class that I hate
when we update parameters of models also
hate when models are this big and this
is a six layer Transformer this is way
smaller than basically anything that we
have um and this makes things very very
uninterpretable um so we'll talk about
one one way that people sort of uh five
years ago started addressing this
problem and this is and this is the idea
of
probing so how do we make sense of a
giant model this is one way so we take
our giant model we cut the top off
basically um and now we have this thing
we stick a probe which actually in a lot
of cases looks very similar to a
language modeling head uh usually it's a
small two layer or one layer
MLP um and we basically treat the model
as something that uh that exists and we
only really care about the output of of
the model so more specifically what is a
probe it's a classifier this this green
thing here uh that is specifically
trained to predict some specific
property from the pre-trained models
representations
alone so um in 2019 Ian Tenny and folks
introduced Edge probing so this is a
general method um it works to probe
different types of information out of a
model so this bottom part here uh yeah
this bottom part here it you pass it in
a sequence you pass it into a model this
is Burt in their experiments often uh
and that outputs a set of contextual
vectors these contextual vectors can be
at any layer um often it's near the
often it's near the top but we'll talk
about uh the the fact that this can work
kind of across layers and different
layers and code different information
and on top of this you have this MLP
that you train to Output a prediction
your model is always always fixed um in
these cases so you can do things like
part of speech tagging where each
specific word you try to determine what
its part of speech is and in that case
this these S1 and S2 spans here uh only
one of them is active because you're
predicting for every single
contextualized Vector you're predicting
whether that thing is a noun or a verb
or something like this you can have
other sorts of tasks too like in ailment
where you have two sequences and two
spans um and you use the embeddings for
those spans um for like sentence one and
sentence two you pull them together in
some way and then you pass them to this
MLP and you see whether the MLP can uh
solve that test so they did this uh in
another paper uh Bert rediscovers the
NLP Pipeline and this there's a lot
going on in this figure the the only
major thing here to take away um is
these numbers that are in this like pink
purple color um so these are a bunch of
different uh properties supp part of
speech uh and and a bunch of other
things um and what they basically find
is that at earlier layers in the model
the things that are closer to the Token
level representation are more um
extractable using a probe and the things
that require more contextualized
information um is extractable later from
later layers in the model and so here's
sort of a brief uh description of what
these tasks are so the ones on the
bottom are more semantic more
contextualized like uh semantic prot
roles and relation relation
classification and then the first few
are more you know chunking in part of
speech tagging and um dependency
labeling in these in these sorts of
tests um so there's a bunch of issues
with probing um and there aren't many
probing papers now as there were many
years ago um and so if your probe let's
say your probe
works it's possible that the
representation actually encodes that
information it's also possible that it
doesn't and the probe solved the task by
itself uh keep in mind that you're
learning this probe you're training this
Probe on labeled data uh let's say your
probe doesn't work does that tell you
anything Maybe not maybe the
representation lacks the information or
maybe your probe doesn't doesn't
actually isn't actually able to
disentangle that information from your
representation maybe the probe is not
the right function class maybe you
poorly trained your probe there's
hyperparameters for your probe so often
times your probe doesn't give you that
much
information there's more problems too so
often we want to probe tasks themselves
and that requires a lot of supervised
data um but we can't collect a lot of
supervised data so we collect some of it
and then that instead produces this
convenient sample that we have that's a
data set that is a convenient sample of
your task so really what you're probing
is the data set and so with all these
limitations it's it's fallen out of
favor a little bit it's still very very
useful but it's fallen out of favor as
like a core model interpretability
idea um also probes designed in this way
are correlated they're correlative not
really causitive so your your underlying
model is trained in a specific way all
of that information is disentangled and
kind of thrown away and you're only
looking at the output representation and
you're saying is my output
representation correlated to the thing
that I'm training this probe for there's
no notion of intervening on this lat and
space there's no notion of of causation
really so you're just seeing whether
your representation is correlated with
your property that you're probing for um
and with these limitations the
community's moved a little bit away from
this area
uh there's a bunch of other probing
works so a bunch of people aim to solve
a bunch of these problems um and uh for
the sake of time I'm not going to go
into all of these but uh I'd encourage
you to look into these they for for some
of these problems they're able to
control for um control for like the
complexity of the of the probe and
things like this um but even despite
that probing is sort of slowly kind of
falling out of
favor uh so before I move into model
interpretability are there any questions
on
probing all right so what is model
interpretability so this is my
definition here uh this is the study of
understanding the internals of models
for example their weights and
activations putting those insights in
human intelligible terms and using that
insight to both patch current models and
develop better ones for not sort of able
to do both of these things patching
current models and develop better ones
we're kind of doing interpretability for
interpretability sake that's nice and
fun but it's not as applicable for the
for the community so you've probably
heard of the term mechanistic
interpretability it's in my opinion a
subfield of model interpretability and
this is sort of my definition I it
aligns reasonably well to the core
mechanistic interpretability people um
but it's the study of reverse
engineering parametric models often
neural networks because that's what we
use from their learned weights into more
human interpretable algorithmic units uh
and often they call these things
circuits um and and these are basically
functions that uh you can describe in a
human interpretable way that sit inside
models um there's a bunch of notable
work again for the sake of time I'm
going to just briefly talk about them
um so the first one is they they look
into analyzing small MLPs and
Transformers to build out the intuition
of what circuits exist um and this a lot
of this work came out of earlier work on
on lstms and doing similar sorts of
things with with
lstms um and they find a bunch of things
one thing that they find is this idea of
induction heads and these induction
heads they say is sort of sort of helps
prove why models can do in context
learning so so an induction head is
something that it's it's a specific
attention head that kind of allows you
to um when given a prefix allow you to
kind of copy the necessary resulting
token from the underlying training data
that the model seen before so in context
learning what you generally provide is
some sort of prefix and then you uh
provide some example and hopefully you
know you can classify the thing or
something like this um it's it's saying
that there's these attention heads
Loosely uh that exist that are able to
copy unearth that information um for for
a specific
context um other things that they' that
they've done is um on neurons so uh this
poly semanticity so what this this kind
of means is that your your neuron is a
uh you have a set of neurons in your
activation space so let's say at layer
10 in your model you have an output um
and so your activations is let's say a
thousand dimensional here those each of
those thousand individual neurons may
represent more than one specific
feature um and so they they talk about
this in that context and this is kind of
a theory but you can think about um
trying to process
input and when you're processing a a
vocab of size 50,000 or 250,000 at some
point in the model we're actually
compressing it down to the hidden
Dimension and so in some cases that
looks like you're going to compress a
much richer feature representation down
into a smaller set of neurons so it is
reasonable to believe that um a specific
neuron will represent multiple of those
features and given the structure of our
weight matrices um it it is the case
that if they are representing more
features than uh number of elements in
the actual or number of neurons in the
activation space then many of these
features linearly dependent and so we're
not really able to utilize them that
well um they they they talk about this
they don't talk about this in the the
most uh the the best way but uh it seems
kind of clear to me that um since you
have embedding matrices that are um not
square that you're that these neurons
have to exist um and they have to
incorporate multiple features at once m
multiple redundant features at
once um so before I move on to the rest
of model interpretability any questions
about mechanistic
interpretability
yeah so most of their studies are for uh
a very small set of of of models and
most of these are old GPT models there
have been a few works like in the last
couple of months on doing this for the
Llama based models um it seems like this
is a general more General phenomenal for
for language models it also is the case
that certain attention heads specialize
and talk about them a little bit um in
in the activations part um but yeah
there's they're not like all attention
heads aren't created equal they start
this way and it seems to be a general
princip of and one one other thing I you
might know about this better than I do
but I think there are some preliminary
words that say that Transformers seem to
be particularly good at doing things
like induction heads compared
to uh H for current models and there was
a paper really recently about comparing
like Mamba and um in Transformer based
models Mamba being a uh kind of more
like closer to a with network which we
also going to talk about us but um so I
I think there's some indication that
Transformers actually kind of are key or
are at least like better at kind of in
Contex learning than otheres are so
there is some
interesting implications of that which
is like well if Transformers are good
what's better than Transformer yeah like
naturally learning this s of thing so um
they're good at yeah they're like really
good at copying and like maintaining
information like more so um and yeah I
think it'd be cool to like be able to I
don't know how to do this but be able to
extract that kind of information like
what of the Transformers is actually
helping it do this copying mechanism or
like being a better in context learner
then we can develop a better structure a
slightly better structure than than a
Transformer um hopefully someone comes
up with that soon but cool any other
question
questions all right so let's move into
model interpretability so there are
weights and their activations I
mentioned these are these are the two
things these are the two things that
we're going to look at so what can you
do with the weights of an RD train model
really you can just edit them and then
kind of see what happens activations
similarly you can look at the
activations for different inputs you can
poke them with a stick and see what
happens a lot of my research is poking
models with a stick and looking at the
activations it's like predominantly what
I've done so we'll talk about that um
and the technical term for this is
intervening on them by adding some
vector or other sort of manipulation to
the lat space but really what you're
doing is like
Pok um so when you look at weights uh
one one class of methods or or area is
on model editing fine-tuning is like the
most extreme version of model editing
usually these things are much more
targeted um so in the model editing sort
of landscape your goal or your target is
you have a concept or a specific fact
that needs to be changed in the model um
and your approach here is you update or
edit the weights of the model to edit
the model's belief of that factor
concept and ideally you do this without
changing any of the other behavior of
the model so for example let's say
you're trying to say that Graham is no
longer a professor at CMU but is a
professor at Stanford you don't want
every single person at CMU to now be a
professor or uh now be affiliated with
Stanford right um gr pleas not to
Stanford um so here's one approach paper
that came out a couple years ago there's
a lot of work down here uh in in the
model editing World I'll give you sort
of a really brief overview of this but
basically they have facts that they want
to they want to manipulate um so for
example the the example that they give
in the figure is they want to associate
the Space Needle with Paris the Space
Needle is a a cool needle in in Seattle
has nothing to do with Paris but Paris
also has a tower so it's close um so
they use causal tracing to isolate the
causal effect uh of the individual
hidden States for this fact so they
basically continuously perturb the input
do a bunch of forward passes and
sequentially find the specific hidden
states that are associated kind of with
this fact um then they make an edit and
their edit
uh looks like this thing on the right um
so they treat this pair Space Needle and
Paris as this uh key value pair where
Space Needle is the key you pass this
into um into this weight Matrix this
original part of the model you want this
now instead of outputting Seattle to
Output Paris and they have some nice
math and a closed form solution to to
identify this this is super expensive
because they have to the causal tracing
part have to do a bunch of forward
passes um and they make this a little
bit better in future future work they
also do sort of a more
comprehensive um edit um so these are
kind like some of the things you can do
um I'm less excited about model editing
um there's there's some work on model
editing sort of it's it's hard to
control what other things break there's
a and there's some work with when you
edit a specific fact things start being
weird and being biased in other ways um
and so
yeah do all kind of seual information
like X is and Y would they Alles to the
same layer is it just with the
specific for this specific example it
looks at this specific point uh for
every example they'll probably find
different regions in a different degree
of
manipulation um and yeah that it gets a
little unprincipled kind of quickly it's
not like they're able to find you know a
specific attention head that or a
specific layer or specific weight Matrix
that corresponds to like
all yeah relations of a specific
type any for questions for yeah this is
actually just a question if you know um
it seems like more frequent facts might
appear in both places in the model is do
you know if that's actually the I have
no idea but uh I would imagine that um
it probably could occur in more places
but also um a lot of the information is
redundant anyway in the model especially
for larger models so you might have to
make targeted interventions in multiple
places but it's possible that one
intervention in one place sufficiently
destroys like contextualized information
in other places if it's close um it
depends on how big this intervention is
if it's like hitting it with a hammer
rather than some like nice fine grain
thing but that'd be a good be a good
experiment to see
um any other
questions all right so we'll move into
the stuff that I'm most familiar with
and some of my work so looking at
activations um so this is this is work
I've been doing for a while uh this idea
of steering vectors so I mentioned I
poke models so it's thick steering
Vector is that thick so it's basically a
fix length vector that steers a language
model to generate a specific sequence
exactly when added to the hidden sites
of a model at a specific
location um and I'll I'll read this
again it's there's a very like specific
form that I wrote this in so uh it's
it's a fix length Vector that steers a
language model to generate a specific
sequence exactly when added to the
hidden states of a model at a specific
point so this is different than um a
soft prompt or different than a model
editing sort of approach
um in this case there is a vector that
corresponds to a sequence and that
Vector doesn't correspond to any other
sequence there could be multiple vectors
and it turns out there are multiple
vectors that correspond to that sequence
it'll be a little bit clearer um based
on how we extract these
things um so this is the stick that
we're talking the language
model um so how do we extract them so
this is
gpt2 um basically this Z steer thing on
the left this is the steering Vector
this gets initialized randomly um with
small like in a in a reasonable way
uniformly and small um and for any
sequence a specific sequence that we
want the model to generate we
optimize this steering Vector Z steer uh
to generate that sequence keeping the
rest of the model entirely fixed so
think about it as we're nudging an a
frozen model to be able to generate a
specific sequence at a specific time um
and we have a lot of different options
on where to inject the steering intive
can put it basically anywhere in the
model we can put it at any time step any
number of these things in practice um
providing it just at the first time step
and somewhere in the middle of the model
basically not the first layer and not
the last layer works pretty well um and
so more formally um forget the kind of
notation um but right here we initialize
um this Z steer and for a few iterations
um we do forward passes first this
starts as random and then this gets
closer and closer and closer to being
able to generate this sequence and
eventually we get to a point uh and this
n is pretty small it's eight or 10 or
something like that um for most
sequences we get to a point where we
have found this stick that is allowed to
poke this model to generate that
sequence exactly now when we greedy
decode from the model we pass in just a
beginning of sequence token and this Z
steer the steering vector and it's able
to uncover a whole sequence that whole
sequence that we had at the beginning
entirely
um this is weird and interesting because
in a lot of cases um in like the
prompting world in the soft prompt World
usually you need a pretty large uh width
of a prompt to be able to do things with
um and this this generally in in that
structure you're doing a specific task
and you're providing kind of a large a
large prompt to do this with large soft
prompt to do this with this is often H
this often has a width of 50 and a and a
length of the the hidden size or the
embedding size of the model in our cases
all of our steering vectors are withth
one and they're of just the hidden size
of the
model um so what ends
up happening
um actually before I go to results any
questions this is a this is a weird
setup and weird relative to what other
people do so happy to take any
questions
[Music]
yeah similarly if your prompt was um of
a specific type so the prompt here is a
continuous Vector passed in it's a
single length width hidden size
continuous Vector so um it's kind of
like maybe collapsing your prompt into
this compressing it into this tiny
VOR you can think of that way
yeah any other questions
yeah this would be like
I'm
things potentially um this is something
that I want to work on it uh like a year
ago and didn't get didn't get this
sufficient Buy in and then had to apply
to grad school and all these things so
it went by the wayside but but
definitely something to something to
pursue um there's a lot of scope there
any other
questions all right so move over to
results so we can find steering vectors
and that's and that's interesting thing
um and we can find them pretty easily
and for most sequences even sequences
that the model hasn't seen before the
underlying language model hasn't seen
before
um it also works for and this is kind of
a negative but it also works for random
sequences of very small length but it's
harder to find so you can imagine if
your uh steering Vector is basically a
giant bulldozer it doesn't matter what
your model is learning learned similar
to the probe situation if you can
compress all that information of that
sequence into the vector you don't
really need the language model um so
there are cases when you're looking at
sequences of length like five seven
eight something like this you can
uniformly sample from the vocabulary at
random with replacement generate utter
garbage and find steering vectors for
them takes a little while but your model
is complex enough that you can basically
bulldo your model to be able to do this
even if that sequence is incredibly low
likelihood under the model but it works
better for things that are higher
likelihood under the model um
predictably the I think the thing that
surprised me the most was these steering
vectors themselves have interpretable
properties U so distances in steering
Vector space reflect semantic similarity
so if you have two sentences that are
close um they're also close in steering
Vector space that's kind of nice
um it does better than for example the
representations one would use for for
probing so mean pooling Bert hidden
States like we looked at before those do
actually worse than steering vectors um
just a bit
surprising um style transfer is possible
with simple Vector arithmetic so it' be
nice to say that I have a sequence I
want to subtract you know negativity and
add positivity for for sentiment or
other sorts of Styles um we can do this
and we can do this reasonably well in
steering VOR
space um we can also decode from
interpolations in the Laten space so you
take two steering vectors for two
sequences you look in the middle of them
you linearly interpolate between them
and you decode um if the space is kind
of weirdly peaky then you would have
issues and what you would generate is
garbage and there's no guarantee that
the space should be reasonable in
between but it turns out it
is um here's an example of one of these
style transfer cases so very very simple
easy easy sentence we found steering
vectors for The Taste is excellent and
and we took a sample of 100 positive
sentences and 100 negative sentences
found their steering vectors took the
mean and thought that you know that
looks like the positive concept steering
Vector negative concept steering Vector
we just did Vector arithmetic just did
uh current steering
Vector uh plus negative minus positive
and we got the taste is unpleasant um
and
similarly um in the reverse
directions it turns out that the
magnitude matters because um for every
single sequence there's kind of an end
dimensional ball around that steering
Vector that we found that also decodes
that specific sequence and so that shows
that the space is kind of reasonably
well formed there's there's of course uh
a lot of weird sort of areas um and so
if you go poke around in steering Vector
space and sort of try to sample from it
eventually you'll find some weird edge
cases and some garbage and repeated text
and little things like
this any questions here before I kind of
Rapid Fire through the the last few
things yeah like here
yeah so we went uh Beyond this um there
was
so in in these specific experiments we
looked at the middle of gpt2 um so this
was like layer six layer seven and at
the first time step we didn't do any um
like magnitude scaling and so you can
imagine if you put a giant Vector in
there the models never the rest of the
model has never seen something of that
magnitude so it's now in a weird State
and it's just going to break so if you
put this to like I don't know 500 or
something like this it break it just has
no idea it's like basically like telling
the rest your model hey it's like a
completely untrained model be it looks
similar to like random performance you
get repeats and things like this smaller
you end up staying in this ball for the
sequence two two seemed pretty
reasonable but we didn't spend a lot of
time just like the day before the paper
was do we were two seems reasonable we
went to three we went to five 10 broke
five somewhat broke two seems
reasonable
um decent signings
hopefully um cool so I'll talk about uh
a similar type of work uh that came out
more recently on inference time
intervention so basically they use some
of the ideas that we talked about
earlier they use linear probes um to
find a tension head that correspond to a
desired attribute they did this for
truthful QA so uh their Hope was to find
truthful directions in Len space
um and then they shifted the attention
head activations um during inference
along the directions determined by the
probes um so what this kind of looks
like is you take your attention heads
you probe them so you stick classify on
top um this classifier learns to
disentangle sort of truthful and
untruthful and now you have um now you
have a hyperplane and then you can move
orthogonally to this hyper plane in the
direction depending on which way you
want to shift so if you want to move
towards truthful you can move in that
direction or or away um and they do this
it works pretty well um I think they do
this for GPT model and maybe a llama
model um but can't can't remember the
exact details um and it's a similar
intervention um they basically add this
Vector um that they found and they they
have a little note on scaling they if
they scale if that if the magnitude of
the thing is too much things break so
they have a they like hyper parameter
search for the sort of magnitude of
activation um but it's sort of a very
similar approach to what we did but this
focuses on specific attention heads and
they don't do this for all the attention
heads so back to like your question
earlier do attention heads specialize it
seems like they do and so there are many
of them that uh have like no probing
accuracy or limited probing accuracy and
actually um are like distractors for the
CH FL
Direction any questions
here cool so more activation
manipulation so there's uh some work
recently on contrastive steering vectors
so the way we did this like sentiment
steering was we had some positive
sentences some negative sentences they
weren't tied together in any reasonable
way we found their steering vectors
separately you could imagine the case
and maybe a more useful case um with two
prompts that um you can design that go
two different ways you can sort of find
their representations and do the
manipulation the differences here um
like individually rather than um for a
whole concept or a whole attribute and
the value here is your context is um
preserved so if you're doing something
like you know you're doing retrieval
based things now you have some sort of
document and then you have a question if
your question sort of uh if you want to
ask it in two different ways for two
different things this would be a much
better approach if you want to use
steering vectors than the stuff I was
doing um and it seems to work a little
bit better they didn't compare against
our our things because it's not like an
Apples to Apples comparison but it seems
to work better and be more General um
and be more
useful
any
questions cool so what can model
interpretability give us these are these
are my concluding remarks so hopefully
we get a better understanding of how
language models work their their
internals their structure um we get to
understand uh kind of why they do really
well this is still like very very
unclear um and hopefully we find
lightweight methods to control and steer
models as models become more and more
useful um and and impact more more users
we need better ways to control and steer
them um and it's unclear how much
industry will devote to these things um
so it might be the role of Academia to
do more science in in order to figure
out how to control and steer these
better um and hopefully we can also find
potential Al Alternatives or
complimentary methods to to do alignment
um rhf is kind of expensive um and if if
we could do this with limited data and
um exploit structure um and information
that's already in the model more so than
than these methods um maybe maybe we can
align them better and these things don't
have to be uh Alternatives they can be
complimentary to to
[Music]
rhm um here's some resources this is an
extremely incomplete group but here are
some folks that work on model
interoperability there's many of these
um I cited some some work from some of
these teams but um there's a lot of
people working on it and in the last
like year there's been kind of an
explosion especially in the mechanistic
interpretability kind of World um Sasha
Rush had a recent tweet that uh asked
like prospective grad students what is
the topic that they're most excited
about and mechanistic interpretability
was a thing that seemed to have won out
um so I encourage you to to kind of dive
into this literature and read some of
the papers if you're if you're excited
about it and yeah thanks for your
attention and that's all I
have