so the class today is uh introduction to
natural language processing and I'll be
talking a little bit about you know what
is natural language processing why we're
motivated to do it and also some of the
difficulties that we encounter and I'll
at the end I'll also be talking about
class Logistics so you can ask any
Logistics questions at that
time so if we talk about what is NLP
anyway uh does anyone have any opinions
about the definition of what natural
language process would be oh one other
thing I should mention is I am recording
the class uh I put the class on YouTube
uh afterwards I will not take pictures
or video of any of you uh but if you
talk your voice might come in the
background so just uh be aware of that
um usually not it's a directional mic so
I try to repeat the questions after
everybody um but uh for the people who
are recordings uh listening to the
recordings um so anyway what is NLP
anyway does anybody have any ideas about
the definition of what NLP might
be
yes okay um it so the answer was it
helps machines understand language
better uh so to facilitate human human
and human machine interactions I think
that's very good um it's
uh similar to what I have written on my
slide here uh but natur in addition to
natural language understanding there's
one major other segment of NLP uh does
anyone uh have an idea what that might
be we often have a dichotomy between two
major segments natural language
understanding and natural language
generation yeah exactly so I I would say
that's almost perfect if you had said
understand and generate so very good um
so I I say natural technology to handle
human language usually text using
computers uh to Aid human machine
communication and this can include
things like question answering dialogue
or generation of code that can be
executed with uh
computers it can also Aid human human
communication and this can include
things like machine translation or spell
checking or assisted writing
and then a final uh segment that people
might think about a little bit less is
analyzing and understanding a language
and this includes things like syntactic
analysis text classification entity
recognition and linking and these can be
used for uh various reasons not
necessarily for direct human machine
communication but also for like
aggregating information across large
things for scientific studies and other
things like that I'll give a few
examples of
this um we now use an many times a day
sometimes without even knowing it so uh
whenever you're typing a doc in Google
Docs there's you know spell checking and
grammar checking going on behind it's
gotten frighten frighteningly good
recently that where it checks like most
of my mistakes and rarely Flags things
that are not mistakes so obviously they
have powerful models running behind that
uh
so and it can do things like answer
questions uh so I asked chat GPT who is
the current president of Carnegie melan
University and chat GPT said I did a
quick search for more information here
is what I found uh the current president
of car Mel University is faram Janan he
has been serving since July 1 etc etc so
as far as I can tell that's
correct um at the same time I asked how
many layers are included in the GP 3.5
turbo architecture and it said to me
GPT 3.5 turbo which is an optimized
version of GPT 3.5 for faster responses
doesn't have a specific layer art
structure like the traditional gpt3
models um and I don't know if this is
true or not but I'm pretty sure it's not
true I'm pretty sure that you know GPT
is a model that's much like other models
uh so it basically just made up the spec
because it didn't have any information
on the Internet or couldn't talk about
it so
um another thing is uh NLP can translate
text pretty well so I ran um Google
translate uh on Japanese uh this example
is a little bit old it's from uh you
know a few years ago about Co but I I
retranslated it a few days ago and it
comes up pretty good uh you can
basically understand what's going on
here it's not perfect but you can
understand the uh the general uh
gist at the same time uh if I put in a
relatively low resource language this is
Kurdish um it has a number of problems
when you try to understand it and just
to give an example this is talking about
uh some uh paleontology Discovery it
called this person a fossil scientist
instead of the kind of obvious English
term
paleontologist um and it's talking about
three different uh T-Rex species uh how
T-Rex should actually be split into
three species where T-Rex says king of
ferocious lizards emperator says emperor
of Savaged lizards and then T Regina
means clean of ferocious snail I'm
pretty sure that's not snail I'm pretty
sure that's lizard so uh you can see
that this is not uh this is not perfect
either some people might be thinking why
Google translate and why not GPD well it
turns out um according to one of the
recent studies we've done GPD is even
worse at these slow resource languages
so I use the best thing that's out
there um another thing is language
analysis can Aid scientific ific inquiry
so this is an example that I've been
using for a long time it's actually from
Martin sap another faculty member here
uh but I have been using it since uh
like before he joined and it uh this is
an example from computational social
science uh answering questions about
Society given observational data and
their question was do movie scripts
portray female or male characters with
more power or agency in movie script
films so it's asking kind of a so
societal question by using NLP
technology and the way they did it is
they basically analyzed text trying to
find
uh the uh agents and patients in a a
particular text which are the the things
that are doing things and the things
that things are being done to and you
can see that essentially male characters
in these movie scripts were given more
power in agency and female characters
were given less power in agency and they
were able to do this because they had
NLP technology that analyzed and
extracted useful data and made turned it
into a very easy form to do kind of
analysis of the variety that they want
so um I think that's a major use case of
NLP technology that does language
analysis nowadays turn it into a form
that allows you to very quickly do
aggregate queries and other things like
this um but at the same time uh language
analysis tools fail at very basic tasks
so these are
some things that I ran through a named
entity recognizer and these were kind of
very nice named entity recognizers uh
that a lot of people were using for
example Stanford core NLP and Spacey and
both of them I just threw in the first
thing that I found on the New York Times
at the time and it basically made at
least one mistake in the first sentence
and here it recognizes Baton Rouge as an
organization and here it recognized
hurricane EA as an organization so um
like even uh these things that we expect
should work pretty well make pretty
Solly
mistakes so in the class uh basically
what I want to cover is uh what goes
into building uh state-of-the-art NLP
systems that work really well on a wide
variety of tasks um where do current
systems
fail and how can we make appropriate
improvements and Achieve whatever we
want to do with nalp and this set of
questions that I'm asking here is
exactly the same as the set of questions
that I was asking two years ago before
chat GPT uh I still think they're
important questions but I think the
answers to these questions is very
different and because of that we're
updating the class materials to try to
cover you know the answers to these
questions and uh in kind of the era of
large language models and other things
like
that
so that's all I have for the intro maybe
maybe pretty straightforward are there
any questions or comments so far if not
I'll I'll just go
on okay great so I want to uh first go
into a very high Lev overview of NLP
system building and most of the stuff
that I want to do today is to set the
stage for what I'm going to be talking
about in more detail uh over the rest of
the
class and we could think of NLP syst
systems through this kind of General
framework where we want to create a
function to map an input X into an
output y uh where X and or Y involve
language and uh do some people have
favorite NLP tasks or NLP tasks that you
want to uh want to be handling in some
way or maybe what what do you think are
the most popular and important NLP tasks
nowadays
okay so translation is maybe easy what's
the input and output of
translation okay yeah so uh in
Translation inputs text in one language
output is text in another language and
then what what is a good
translation yeah corre or or the same is
the input basically yes um it also
should be fluent but I agree any other
things generation the reason why I said
it's tough is it's pretty broad um and
it's not like we could be doing
generation with lots of different inputs
but um yeah any any other things maybe a
little bit different yeah like
scenario a scenario and a multiple
choice question about the scenario and
so what would the scenario in the
multiple choice question are probably
the input and then the output
is an answer to the multiple choice
question um and then there it's kind of
obvious like what is good it's the
correct answer sure um interestingly I
think a lot of llm evaluation is done on
these multiple choice questions but I'm
yet to encounter an actual application
that cares about multiple choice
question answering so uh there's kind of
a funny disconnect there but uh yeah I
saw hand that think about V search comp
yeah Vector search uh that's very good
so the input
is can con it into or understanding and
it to
another okay yeah so I'd say the input
there is a query and a document base um
and then the output is maybe an index
into the document or or something else
like that sure um and then something
that's good here here's a good question
what what's a good result from
that what's a good
output be sort of simar the major
problem there I see is how you def SAR
and how you
a always like you understand
whether is actually
yeah exactly so that um just to repeat
it's like uh we need to have a
similarity a good similarity metric we
need to have a good threshold where we
get like the ones we want and we don't
get the ones we don't want we're going
to talk more about that in the retrieval
lecture exactly how we evaluate and
stuff but um yeah good so this is a good
uh here are some good examples I have
some examples of my own um the first one
is uh kind of the very generic one maybe
kind of like generation here but text in
continuing text uh so this is language
modeling so you have a text and then you
have the continuation you want to
predict the
continuation um text and text in another
language is translation uh text in a
label could be text classification uh
text in linguistic structure or uh some
s kind of entities or something like
that could be uh language analysis or um
information
extraction uh we could also have image
and text uh which is image captioning um
or speech and text which is speech
recognition and I take the very broad
view of natural language processing
which is if it's any variety of language
uh if you're handling language in some
way it's natural language processing it
doesn't necessarily have to be text
input text output um so that's relevant
for the projects that you're thinking
about too at the end of this course so
the the most common FAQ for this course
is does my project count and if you're
uncertain you should ask but usually
like if it has some sort of language
involved then I'll usually say yes it
does kind so um if it's like uh code to
code there that's not code is not
natural language it is language but it's
not natural language so that might be
borderline we might have to discuss
about
that cool um so next I'd like to talk
about methods for creating NLP systems
um and there's a lot of different ways
to create MLP systems all of these are
alive and well in
2024 uh the first one is Rule uh
rule-based system creation and so the
way this works is like let's say you
want to build a text classifier you just
write the simple python function that
classifies things into uh sports or
other and the way it classifies it into
sports or other is it checks whether
baseball soccer football and Tennis are
included in the document and classifies
it into uh Sports if so uh other if not
so has anyone written something like
this maybe not a text classifier but um
you know to identify entities or uh
split words
or something like
that has anybody not ever written
anything like
this yeah that's what I thought so um
rule-based systems are very convenient
when you don't really care about how
good your system is um or you're doing
that's really really simple and like
it'll be perfect even if you do the very
simple thing and so I I think it's worth
talking a little bit about them and I'll
talk a little bit about that uh this
time the second thing which like very
rapidly over the course of maybe three
years or so has become actually maybe
the dominant Paradigm in NLP is
prompting uh in prompting a language
model and the way this works is uh you
ask a language model if the following
sent is about sports reply Sports
otherwise reply other and you feed it to
your favorite LM uh usually that's GPT
something or other uh sometimes it's an
open source model of some variety and
then uh it will give you the
answer and then finally uh fine-tuning
uh so you take some paired data and you
do machine learning from paired data
where you have something like I love to
play baseball uh the stock price is
going up he got a hatrick yesterday he
is wearing tennis shoes and you assign
all these uh labels to them training a
model and you can even start out with a
prompting based model and fine-tune a a
language model
also so one major consideration when
you're Building Systems like this is the
data requirements for building such a
system and for rules or prompting where
it's just based on intuition really no
data is needed whatsoever it you don't
need a single example and you can start
writing rules or like just just to give
an example the rules and prompts I wrote
here I didn't look at any examples and I
just wrote them uh so this is something
that you could start out
with uh the problem is you also have no
idea how well it works if you don't have
any data whatsoever right so um you'll
you might be in trouble if you think
something should be working
so normally the next thing that people
move to nowadays when they're building
practical systems is rules are prompting
based on spot checks so that basically
means that you start out with a
rule-based system or a prompting based
system and then you go in and you run it
on some data that you're interested in
you just kind of qualitatively look at
the data and say oh it's messing up here
then you go in and fix your prompt a
little bit or you go in and fix your
rules a little bit or something like
that so uh this is kind of the second
level of difficulty
so the third level of difficulty would
be something like rules are prompting
with rigorous evaluation and so here you
would create a development set with
inputs and outputs uh so you uh create
maybe 200 to 2,000
examples um
and then evaluate your actual accuracy
so you need an evaluation metric you
need other things like this this is the
next level of difficulty but if you're
going to be a serious you know NLP
engineer or something like this you
definitely will be doing this a lot I
feel and
then so that here now you start needing
a depth set and a test set and then
finally fine-tuning you need an
additional training set um and uh this
will generally be a lot bigger than 200
to 2,000 examples and generally the rule
is that every time you
double
every time you double your training set
size you get about a constant
Improvement so if you start
out if you start out down here with
um zero shot accuracy with a language
model you you create a small printing
set and you get you know a pretty big
increase and then every time you double
it it increases by constant fact it's
kind of like just in general in machine
learning this is a trend that we tend to
see so um So based on this
uh there's kind of like you get a big
gain from having a little bit of
training data but the gains very quickly
drop off and you start spending a lot of
time annotating
an so um yeah this is the the general
overview of the different types of
system building uh any any question
questions about this or comments or
things like
this I think one thing that's changed
really drastically from the last time I
taught this class is the fact that
number one and number two are the things
that people are actually doing in
practice uh which was you know people
who actually care about systems are
doing number one and number two is the
main thing it used to be that if you
were actually serious about building a
system uh you really needed to do the
funing and now it's kind of like more
optional
so
yeah
so it's it's definitely an empirical
observation
um in terms of the theoretical
background I am not I can't immediately
point to a
particular paper that does that but I
think if you think about
the I I think I have seen that they do
exist in the past but I I can't think of
it right now I can try to uh try to come
up with an example of
that so yeah I I should take
notes or someone wants to share one on
Piaza uh if you have any ideas and want
to share on Patza I'm sure that would be
great it'd be great to have a discussion
on
Patza um Pi
one cool okay so next I want to try to
make a rule-based system and I'm going
to make a rule-based system for
sentiment
analysis uh and this is a bad idea I
would not encourage you to ever do this
in real life but I want to do it here to
show you why it's a bad idea and like
what are some of the hard problems that
you encounter when trying to create a
system based on rules
and then we'll move into building a
machine learning base system after we
finish this so if we look at the example
test this is review sentiment analysis
it's one of the most valuable uh tasks
uh that people do in NLP nowadays
because it allows people to know how
customers are thinking about products uh
improve their you know their product
development and other things like that
may monitor people's you know
satisfaction with their social media
service other things like this so
basically the way it works is um you
have uh outputs or you have sentences
inputs like I hate this movie I love
this movie I saw this movie and this
gets mapped into positive neutral or
negative so I hate this movie would be
negative I love this movie positive and
I saw this movie is
neutral so um
that that's the task input tax output
labels uh Kary uh sentence
label and in order to do this uh we
would like to build a model um and we're
going to build the model in a rule based
way but it we'll still call it a model
and the way it works is we do feature
extraction um so we extract the Salient
features for making the decision about
what to Output next we do score
calculation calculate a score for one or
more possib ities and we have a decision
function so we choose one of those
several
possibilities and so for feature
extraction uh formally what this looks
like is we have some function and it
extracts a feature
Vector for score calculation um we
calculate the scores based on either a
binary classification uh where we have a
a weight vector and we take the dot
product with our feature vector or we
have multi class classification where we
have a weight Matrix and we take the
product with uh the vector and that
gives us you know squares over multiple
classes and then we have a decision uh
rule so this decision rule tells us what
the output is going to be um does anyone
know what a typical decision rule is
maybe maybe so obvious that you don't
think about it often
but uh a threshold um so like for would
that be for binary a single binary
scaler score or a multiple
class binary yeah so and then you would
pick a threshold and if it's over the
threshold
you say yes and if it's under the
threshold you say no um another option
would be um you have a threshold and you
say
yes no
obain so you know you don't give an
answer and depending on how you're
evaluated what what is a good classifier
you might want to abstain some of the
time also um for multiclass what what's
a standard decision role for
multiclass argmax yeah exactly so um
basically you you find the index that
has the highest score in you output
it we're going to be talking about other
decision rules also um like
self-consistency and minimum based risk
later uh for text generation so you can
just keep that in mind and then we'll
forget about it for like several
classes um so for sentiment
class um I have a Cod
walk
here
and this is pretty simple um but if
you're bored uh of the class and would
like to um try out yourself you can
Challenge and try to get a better score
than I do um over the next few minutes
but we have this rule based classifier
in
here and I will open it up in my vs
code
to try to create a rule-based classifier
and basically the way this
works is
that we have a feature
extraction we have feature extraction we
have scoring and we have um a decision
rle so here for our feature extraction I
have created a list of good words and a
list of bad
words
and what we do is we just count the
number of good words that appeared and
count the number of bad words that
appeared then we also have a bias
feature so the bias feature is a feature
that's always one and so what that
results in is we have a dimension three
feature Vector um where this is like the
number of good words this is the number
of bad words and then you have the
bias and then I also Define the feature
weights that so for every good word we
add one to our score for every bad word
we add uh we subtract one from our score
and for the BIOS we absor and so we then
take the dot product between
these
two and we get minus
0.5 and that gives us uh that gives us
the
squore so let's run
that um and I read in some
data and what this data looks like is
basically we have a
review um which says the rock is
destined to be the 21st Century's new
Conan and that he's going to make a
splash even greater than Arnold
Schwarzenegger jeanclaude vanam or
Steven Seagal um so this seems pretty
positive right I like that's a pretty
high order to be better than Arnold
Schwarzenegger or John Claude vanam uh
if you're familiar with action movies um
and so of course this gets a positive
label and so uh we have run classifier
actually maybe I should call this
decision rule because this is
essentially our decision Rule and here
basically do the thing that I mentioned
here the yes no obstain or in this case
positive negative neutral so if the
score is greater than zero we uh return
one if the score is less than zero we
return negative one which is negative
and otherwise we returns
zero um we have an accuracy calculation
function just calculating the outputs
are good and
um this is uh the overall label count in
the in the output so we can see there
slightly more positives than there are
negatives and then we can run this and
we get a a score of
43 and so one one thing that I have
found um is I I do a lot of kind
of research on how to make NLP systems
better and one of the things I found
really invaluable
is if you're in a situation where you
have a
set task and you just want to make the
system better on the set task doing
comprehensive error analysis and
understanding where your system is
failing is one of the best ways to do
that and I would like to do a very
rudimentary version of this here and
what I'm doing essentially is I'm just
randomly picking uh several examples
that were
correct
um and so like let let's look at the
examples here um here the true label is
zero um in this predicted one um it may
not be as cutting as Woody or as true as
back in the Glory Days of uh weekend and
two or three things that I know about
her but who else engaged in film Mak
today is so cognizant of the cultural
and moral issues involved in the process
so what words in here are a good
indication that this is a neutral
sentence any
ideas little bit tough
huh starting to think maybe we should be
using machine
learning
um even by the intentionally low
standards of fratboy humor sority boys
is a
Bowser I think frat boy is maybe
negative sentiment if you're familiar
with
us us I don't have any negative
sentiment but the people who say it that
way have negative senent maybe so if we
wanted to go in and do that we could
maybe I won't save this but
uh
um oh whoops I'll go back and fix it uh
crass crass is pretty obviously negative
right so I can add
crass actually let me just add
CR and then um I'll go back and have our
train accurate
wa maybe maybe I need to run the whole
thing
again
and that budg the training accuracy a
little um the dev test accuracy not very
much so I could go through and do this
um let me add
unengaging so I could go through and do
this all day and you probably be very
bored on
engage but I won't do that uh because we
have much more important things to be
doing
um and uh so anyway we um we could go
through and design all the features here
but like why is this complicated like
the the reason why it was complicated
became pretty
clear from the uh from the very
beginning uh the very first example I
showed you which was that was a really
complicated sentence like all of us
could see that it wasn't like really
strongly positive it wasn't really
strongly negative it was kind of like in
the middle but it was in the middle and
it said it in a very long way uh you
know not using any clearly positive
sentiment words not using any clearly
negative sentiment
words
um so yeah basically I I
improved um but what are the difficult
cases uh that we saw here so the first
one is low frequency
words so um here's an example the action
switches between past and present but
the material link is too tenuous to
Anchor the emotional connections at
purport to span a 125 year divide so
this is negative um tenuous is kind of a
negative word purport is kind of a
negative word but it doesn't appear very
frequently so I would need to spend all
my time looking for these words and
trying to them in um here's yet another
horse franchise mucking up its storyline
with glitches casual fans could correct
in their sleep negative
again um so the solutions here are keep
working until we get all of them which
is maybe not super fun um or incorporate
external resources such as sentiment
dictionaries that people created uh we
could do that but that's a lot of
engineering effort to make something
work
um another one is conjugation so we saw
unengaging I guess that's an example of
conjugation uh some other ones are
operatic sprawling picture that's
entertainingly acted magnificently shot
and gripping enough to sustain most of
its 170 minute length so here we have
magnificently so even if I added
magnificent this wouldn't have been
clocked
right um it's basically an overlong
episode of tales from the cryp so that's
maybe another
example um so some things that we could
do or what we would have done before the
modern Paradigm of machine learning is
we would run some sort of normalizer
like a stemmer or other things like this
in order to convert this into uh the
root wordss that we already have seen
somewhere in our data or have already
handed so that requires um conjugation
analysis or morphological analysis as we
say it in
technicals
negation this is a tricky one so this
one's not nearly as Dreadful as expected
so Dreadful is a pretty bad word right
but not nearly as Dreadful as expected
is like a solidly neutral um you know or
maybe even
positive I would I would say that's
neutral but you know uh neutral or
positive it's definitely not
negative um serving s doesn't serve up a
whole lot of laughs so laughs is
obviously positive but not serving UPS
is obviously
negative so if negation modifies the
word disregard it now we would probably
need to do some sort of syntactic
analysis or semantic analysis of
some metaphor an analogy so puts a human
face on a land most westerners are
unfamiliar though uh this is
positive green might want to hang on to
that ski mask as robbery may be the only
way to pay for this next project
so this this is saying that the movie
was so bad that the director will have
to rob people in order to get money for
the next project so that's kind of bad I
guess um has all the depth of a waiting
pool this is kind of my favorite one
because it's really short and sweet but
you know you need to know how deep a
waiting pool is um so that's
negative so the solution here I don't
really even know how to handle this with
a rule based system I have no idea how
we would possibly do this yeah machine
learning based models seem to be pretty
adaptive okay and then I start doing
these ones um anyone have a good
idea any any other friends who know
Japanese no okay um so yeah that's
positive um that one's negative uh and
the solution here is learn Japanese I
guess or whatever other language you
want to process so like obviously
rule-based systems don't scale very
well so um we've moved but like rule
based systems don't scale very well
we're not going to be using them for
most of the things we do in this class
but I do think it's sometimes useful to
try to create one for your task maybe
right at the very beginning of a project
because it gives you an idea about
what's really hard about the task in
some cases so um yeah I wouldn't
entirely discount them I'm not
introducing them for no reason
whatsoever
so next is machine learning based anal
and machine learning uh in general uh I
here actually when I say machine
learning I'm going to be talking about
the traditional fine-tuning approach
where we have a training set Dev set
test set and so we take our training set
we run some learning algorithm over it
we have a learned feature extractor F A
possibly learned feature extractor F
possibly learned scoring function W and
uh then we apply our inference algorithm
our decision Rule and make decisions
when I say possibly learned actually the
first example I'm going to give of a
machine learning based technique is uh
doesn't have a learned feature extractor
but most things that we use nowadays do
have learned feature
extractors so our first attempt is going
to be a bag of words model uh and the
way a bag of wordss model works is uh
essentially we start out by looking up a
Vector where one element in the vector
is uh is one and all the other elements
in the vector are zero and so if the
word is different the position in the
vector that's one will be different we
add all of these together and this gives
us a vector where each element is the
frequency of that word in the vector and
then we multiply that by weights and we
get a
score and um here as I said this is not
a learned feature
uh Vector this is basically uh sorry not
a learn feature extractor this is
basically a fixed feature extractor but
the weights themselves are
learned um so my my question is I
mentioned a whole lot of problems before
I mentioned infrequent words I mentioned
conjugation I mentioned uh different
languages I mentioned syntax and
metaphor so which of these do we think
would be fixed by this sort of learning
based
approach
any
ideas maybe not fixed maybe made
significantly
better any Brave uh brave
people maybe maybe
negation okay so maybe doesn't when it
have a negative qu
yeah yeah so for the conjugation if we
had the conjugations of the stems mapped
in the same position that might fix a
conjugation problem but I would say if
you don't do that then this kind of
fixes conjugation a little bit but maybe
not not really yeah kind of fix
conjugation because like they're using
the same there
probably different variations so we
learn how to
classify surrounding
structure yeah if it's a big enough
training set you might have covered the
various conjugations but if you haven't
and you don't have any rule-based
processing it it might still be problems
yeah yeah so in frequent words if you
have a large enough training set yeah
you'll be able to fix it to some extent
so none of the problems are entirely
fixed but a lot of them are made better
different languages is also made better
if you have training data in that
language but if you don't then you're
out of BL so um so now what I'd like to
do is I'd look to like to look at what
our vectors represent so basically um in
uh in binary classification each word um
sorry so the vectors themselves
represent the counts of the words here
I'm talking about what the weight uh
vectors or matrices correspond to and
the weight uh Vector here will be
positive if the word it tends to be
positive if in a binary classification
case in a multiclass classification case
we'll actually have a matrix that looks
like this where um each column or row uh
corresponds to the word and each row or
column corresponds to a label and it
will be higher if that row tends to uh
correlate with that uh that word tends
to correlate that little
bit so
this um training of the bag of words
model is can be done uh so simply that
we uh can put it in a single slide so
basically here uh what we do is we start
out with the feature
weights and for each example in our data
set we extract features um the exact way
I'm extracting features is basically
splitting uh splitting the words using
the python split function and then uh
Counting number of times each word
exists uh we then run the classifier so
actually running the classifier is
exactly the same as what we did for the
uh the rule based system it's just that
we have feature vectors instead and
then if the predicted value is
not value then for each of the
features uh in the feature space we
upweight
the um we upweight The Weight by the
vector
size by or by the amount of the vector
if Y is positive and we downweight the
vector uh by the size of the vector if Y
is negative so this is really really
simple it's uh probably the simplest
possible algorithm for training one of
these models um but I have an
example in this that you can also take a
look at here's a trained bag of words
classifier and we could step through
this is on exactly the same data set as
I did before we're training on the
training set
um and uh evaluating on the dev set um I
also have some extra stuff like I'm
Shuffling the order of the data IDs
which is really important if you're
doing this sort of incremental algorithm
uh because uh what if what if your
creating data set was ordered in this
way where you have all of the positive
labels on
top and then you have all of the
negative labels on the
bottom if you do something like this it
would see only negative labels at the
end of training and you might have
problems because your model would only
predict negatives so we also Shuffle
data um and then step through we run the
classifier and I'm going to run uh five
epochs of training through the data set
uh very
fast and calculate our accuracy
and this got 75% accuracy on the
training data set and uh 56% accuracy on
the Deb data set so uh if you remember
our rule-based classifier had 42 uh 42
accuracy and now our training based
classifier has 56 accuracy but it's
overfitting heavily to the training side
so um basically this is a pretty strong
advertisement for why we should be using
machine learning you know I the amount
of code that we had for this machine
learning model is basically very similar
um it's not using any external libraries
but we're getting better at
this
cool
so cool any any questions
here and so I'm going to talk about the
connection to between this algorithm and
neural networks in the next class um
because this actually is using a very
similar training algorithm to what we
use in neural networks with some uh
particular
assumptions cool um so what's missing in
bag of words um still handling of
conjugation or compound words is not
perfect it we can do it to some extent
to the point where we can uh memorize
things so I love this movie I love this
movie another thing is handling word Ser
uh similarities so I love this movie and
I adore this movie uh these basically
mean the same thing as humans we know
they mean the same thing so we should be
able to take advantage of that fact to
learn better models but we're not doing
that in this model at the moment because
each unit is uh treated as a atomic unit
and there's no idea of
similarity also handling of combination
features so um I love this movie and I
don't love this movie I hate this movie
and I don't hate this movie actually
this is a little bit tricky because
negative words are slightly indicative
of it being negative but actually what
they do is they negate the other things
that you're saying in the
sentence
so um like love is positive hate is
negative but like don't
love it's actually kind of like this
right like um Love is very positive POS
hate is very negative but don't love is
like slightly less positive than don't
hate right so um It's actually kind of
tricky because you need to combine them
together and figure out what's going on
based on that another example that a lot
of people might not think of immediately
but is super super common in sentiment
analysis or any other thing is butt so
basically what but does is it throws
away all the stuff that you said before
um and you can just pay attention to the
stuff that you saw beforehand so like we
could even add this to our um like if
you want to add this to your rule based
classifier you can do that you just
search for butt and delete everything
before it and see if that inputs your
accuracy might be might be a fun very
quick thing
to cool so the better solution which is
what we're going to talk about for every
other class other than uh other than
this one is neural network models and
basically uh what they do is they do a
lookup of uh dense word embeddings so
instead of looking up uh individual uh
sparse uh vectors individual one hot
vectors they look up dense word
embeddings and then throw them into some
complicated function to extract features
and based on the features uh multiply by
weights and get a score um and if you're
doing text classification in the
traditional way this is normally what
you do um if you're doing text
classification with something like
prompting you're still actually doing
this because you're calculating the
score of the next word to predict and
that's done in exactly the same way so
uh even if you're using a large language
model like GPT this is still probably
happening under the hood unless open the
eye invented something that very
different in Alien than anything else
that we know of but I I'm guessing that
that propably hasn't
happen um one nice thing about neural
networks is neural networks
theoretically are powerful enough to
solve any task if you make them uh deep
enough or wide enough uh like if you
make them wide enough and then if you
make them deep it also helps further so
anytime somebody says well you can't
just solve that problem with neural
networks you know that they're lying
basically because they theoretically can
solve every problem uh but you have you
have issues of data you have issues of
other things like that so you know they
don't just necessarily work
outs cool um so the final thing I'd like
to talk about is the road map going
forward some of the things I'm going to
cover in the class and some of the
logistics
issues so um the first thing I'm going
to talk about in the class is language
modeling fun
fundamentals and uh so this could
include language models uh that just
predict the next words it could include
language models that predict the output
given the uh the input or the prompt um
I'm going to be talking about
representing words uh how how we get
word representation subword models other
things like that uh then go kind of
deeper into language modeling uh how do
we do it how do we evaluate it other
things um sequence encoding uh and this
is going to cover things like uh
Transformers uh self attention modals
but also very quickly cnns and rnns
which are useful in some
cases um and then we're going to
specifically go very deep into the
Transformer architecture and also talk a
little bit about some of the modern uh
improvements to the Transformer
architecture so the Transformer we're
using nowadays is very different than
the Transformer that was invented in
2017 uh so we're going to talk well I
wouldn't say very different but
different enough that it's important so
we're going to talk about some of those
things second thing I'd like to talk
about is training and inference methods
so this includes uh generation
algorithms uh so we're going to have a
whole class on how we generate text uh
in different ways uh prompting how uh we
can prompt things I hear uh world class
prompt engineers make a lot of money
nowadays so uh you'll want to pay
attention to that one um and instruction
tuning uh so how do we train models to
handle a lot of different tasks and
reinforcement learning so how do we uh
you know like actually generate outputs
uh kind of Judge them and then learn
from
there also experimental design and
evaluation so experimental design uh so
how do we design an experiment well uh
so that it backs up what we want to be
uh our conclusions that we want to be
backing up how do we do human annotation
of data in a reliable way this is
getting harder and harder as models get
better and better because uh getting
humans who don't care very much about
The annotation task they might do worse
than gp4 so um you need to be careful of
that also debugging and interpretation
technique so what are some of the
automatic techniques that you can do to
quickly figure out what's going wrong
with your models and improve
them and uh bias and fairness
considerations so it's really really
important nowadays uh that models are
being deployed to real people in the
real world and like actually causing
harm to people in some cases that we
need to be worried about
that Advanced Training in architectures
so we're going to talk about distill
distillation and quantization how can we
make small language models uh that
actually still work well like not large
you can run them on your phone you can
run them on your local
laptop um ensembling and mixtures of
experts how can we combine together
multiple models in order to create
models that are better than the sum of
their
parts and um retrieval and retrieval
augmented
generation long sequence models uh so
how do we handle long
outputs um and uh we're going to talk
about applications to complex reasoning
tasks code generation language agents
and knowledge-based QA and information
extraction I picked
these because they seem to be maybe the
most important at least in research
nowadays and also they cover uh the
things that when I talk to people in
Industry are kind of most interested in
so hopefully it'll be useful regardless
of uh whether you plan on doing research
or or plan on doing industry related
things uh by by the way the two things
that when I talk to people in Industry
they're most interested in are Rag and
code generation at the moment for now um
so those are ones that you'll want to
pay attention
to and then finally we have a few
lectures on Linguistics and
multilinguality um I love Linguistics
but uh to be honest at the moment most
of our Cutting Edge models don't
explicitly use linguistic structure um
but I still think it's useful to know
about it especially if you're working on
multilingual things especially if you're
interested in very robust generalization
to new models so we're going to talk a
little bit about that and also
multilingual LP I'm going to have
fure so also if you have any suggestions
um we have two guest lecture slots still
open uh that I'm trying to fill so if
you have any things that you really want
to hear about um I could either add them
to the
existing you know content or I could
invite a guest lecturer who's working on
that topic so you know please feel free
to tell
me um then the class format and
structure uh the class
content my goal is to learn in detail
about building NLP systems from a
research perspective so this is a 700
level course so it's aiming to be for
people who really want to try new and
Innovative things in uh kind of natural
language processing it's not going to
focus solely on reimplementing things
that have been done before including in
the project I'm going to be expecting
everybody to do something something
that's kind of new whether it's coming
up with a new method or applying
existing methods to a place where they
haven't been used before or building out
things for a new language or something
like that so that's kind of one of the
major goals of this
class um learn basic and advanced topics
in machine learning approaches to NLP
and language models learn some basic
linguistic knowledge useful in NLP uh
see case studies of NLP applications and
learn how to identify unique problems
for each um one thing i' like to point
out is I'm not going to cover every NLP
application ever because that would be
absolutely impossible NLP is being used
in so many different areas nowadays but
what I want people to pay attention to
like even if you're not super interested
in code generation for example what you
can do is you can look at code
generation look at how people identify
problems look at the methods that people
have proposed to solve those unique
problems and then kind of map that try
to do some generalization onto your own
problems of Interest so uh that's kind
of the goal of the NLP
applications finally uh learning how to
debug when and where NLP systems fail
and build improvements based on this so
um ever since I was a graduate student
this has been like one of the really
important things that I feel like I've
done well or done better than some other
people and I I feel like it's a really
good way to like even if you're only
interested in improving accuracy knowing
why your system's failing still is the
best way to do that I so I'm going to
put a lot of emphasis on
that in terms of the class format um
before class for some classes there are
recommended reading uh this can be
helpful to read I'm never going to
expect you to definitely have read it
before the class but I would suggest
that maybe you'll get more out of the
class if you do that um during class
we'll have the lecture um in discussion
with
everybody um sometimes we'll have a code
or data walk
um actually this is a a little bit old I
I have this slide we're this year we're
going to be adding more uh code and data
walks during office hours and the way it
will work is one of the Tas we have
seven Tas who I'm going to introduce
very soon but one of the Tas will be
doing this kind of recitation where you
um where we go over a library so if
you're not familiar with the library and
you want to be more familiar with the
library you can join this and uh then
we'll be able to do this and this will
cover things like
um pie torch and sentence piece uh we're
going to start out with hugging face um
inference stuff like
VM uh debugging software like
Xeno um what were the other
ones oh the open AI API and light llm
other stuff like that so we we have lots
of them planned we'll uh uh we'll update
that um and then after class after
almost every class we'll have a question
quiz um and the quiz is intended to just
you know make sure that you uh paid
attention to the material and are able
to answer questions about it we will aim
to release it on the day of the course
the day of the actual lecture and it
will be due at the end of the following
day of the lecture so um it will be
three questions it probably shouldn't
take a whole lot of time but um uh yeah
so we'll H
that in terms of assignments assignment
one is going to be build your own llama
and so what this is going to look like
is we're going to give you a partial
implementation of llama which is kind of
the most popular open source language
model nowadays and ask you to fill in um
ask you to fill in the parts we're going
to train a very small version of llama
on a small data set and get it to work
and the reason why it's very small is
because the smallest actual version of
llama is 7 billion
parameters um and that might be a little
bit difficult to train with
resources um for assignment two we're
going to try to do an NLP task from
scratch and so the way this will work is
we're going to give you an assignment
which we're not going to give you an
actual data set and instead we're going
to ask you to uh perform data creation
modeling and evaluation for a specified
task and so we're going to tell you uh
what to do but we're not going to tell
you exactly how to do it but we're going
to try to give as conrete directions as
we can um
yeah will you be given a parameter limit
on the model so that's a good question
or like a expense limit or something
like that um I maybe actually I should
take a break from the assignments and
talk about compute so right now um for
assignment one we're planning on having
this be able to be done either on a Mac
laptop with an M1 or M2 processor which
I think a lot of people have or Google
collab um so it should be like
sufficient to use free computational
resources that you have for number two
we'll think about that I think that's
important we do have Google cloud
credits for $50 for everybody and I'm
working to get AWS credits for more um
but the cloud providers nowadays are
being very stingy so um so it's uh been
a little bit of a fight to get uh
credits but I I it is very important so
I'm going to try to get as as many as we
can um and so yeah I I think basically
uh there will be some sort of like limit
on the amount of things you can
practically do and so because of that
I'm hoping that people will rely very
heavily on pre-trained models um or uh
yeah pre-trained models
and yeah so that that's the the short
story B um the second thing uh the
assignment three is to do a survey of
some sort of state-ofthe-art research
resarch and do a reimplementation of
this and in doing this again you will
have to think about something that's
feasible within computational
constraints um and so you can discuss
with your Tas about uh about the best
way to do this um and then the final
project is to perform a unique project
that either improves on the state-of-the
art with respect to whatever you would
like to improve with this could be uh
accuracy for sure this could be
efficiency
it could be some sense of
interpretability but if it's going to be
something like interpretability you'll
have to discuss with us what that means
like how we measure that how we can like
actually say that you did a good job
with improving that um another thing
that you can do is take whatever you
implemented for assignment 3 and apply
it to a new task or apply it to a new
language that has never been examined
before so these are also acceptable
final projects but basically the idea is
for the final project you need to do
something something new that hasn't been
done before and create new knowledge
with the respect
toy um so for this the instructor is me
um I'm uh looking forward to you know
discussing and working with all of you
um for TAS we have seven Tas uh two of
them are in transit so they're not here
today um the other ones uh Tas would you
mind coming up uh to introduce
yourself
so um yeah nhir and akshai couldn't be
here today because they're traveling
I'll introduce them later because
they're coming uh next
time cool and what I'd like everybody to
do is say um like you know what your
name is uh what
your like maybe what you're interested
in um and the reason the goal of this is
number one for everybody to know who you
are and number two for everybody to know
who the best person to talk to is if
they're interested in
particular hi uh I'm
Aila second
year I work on language and social
and I'm I'm a second this year PhD
student Grand and Shar with you I search
is like started in the border of MP and
computer interaction with a lot of work
on automating parts of the developer
experience to make it easier for anyone
to
[Music]
orif
everyone first
year
[Music]
I don't like updating primar models I
hope to not update Prim
modelm yeah thanks a lot everyone and
yeah
than and so we will um we'll have people
uh kind of have office hours uh every ta
has office hours at a regular time
during the week uh please feel free to
come to their office hours or my office
hours um I think they are visha are they
posted on the site or okay yeah they
they either are or will be posted on the
site very soon um and come by to talk
about anything uh if there's nobody in
my office hours I'm happy to talk about
things that are unrelated but if there's
lots of people waiting outside or I
might limit it to uh like um just things
about the class so cool and we have
Patza we'll be checking that regularly
uh striving to get you an answer in 24
hours on weekdays over weekends we might
not so um yeah so that's all for today
are there any questions