Grading
[MUSIC]
Well, hello.
Welcome to another
episode of Undercooled.
Today, Tim and I are
gonna talk about grading.
Why?
Because we just graded our
class from the end of the term,
and it's foremost on our minds.
So why don't we start by talking about
what grading is, why it exists,
and maybe some of the
problems we've identified.
So we'll start with what's wrong with
traditional grading,
and let's start with curving.
So Tim, what do you think about curves?
I think they're very
statistically interesting,
and they reflect a few assumptions and
biases in our grading system.
First is this idea that there should even
be a Gaussian distribution
in our populations
that we're working with.
And like a lot of these assumptions about
grading, there's a nugget of truth to it.
If you looked at the entire nation,
there'd probably be something pretty
close to a normal
distribution of proficiency in
lots of different areas.
But at a university, we're performing
this biased selection
from a certain subset of
the population to try to get the kind of
people that we want to get.
So believing that there should be an
underlying normal distribution
is already quite a flawed model.
We tend to have a much more skewed
distribution with the
students that we have,
because we're selecting who we admit,
some of those selection criteria.
Actually, you know, that's
another episode entirely.
We can talk about admissions another day,
but we don't have a Gaussian
distribution to begin with.
So why would you try to enforce one?
I think curving is
pretty silly in that sense.
Yet we're always told that you should fit
your distribution to a bell curve,
which is a Gaussian
distribution, of course.
And that's the standard way that most
faculty just approach it.
And it's a shame because it's really
flawed thinking like
you just pointed out.
Yeah.
And there's another problem there, which
is to a lot of students,
this conveys a very competitive,
anti-collaborative mindset, right?
If they're going to
be stuck onto a curve,
then they have to out-compete their
classmates in order to get a grade,
which maybe they should or should not
actually care about,
but intrinsically a lot of them do.
And as soon as you throw away the curve,
you can get to a space
where everyone can succeed
and everyone can help each
other achieve at a higher level
to actually meet and hopefully exceed
your expectations for the course.
Yeah. That's so true.
Whoever said that only 10%
of the class should get A's.
What does that mean?
You know, if you take teaching
seriously, like I think we do,
you want everybody to
be able to get an A.
But you have to have the bar pretty high
so that everybody actually earns an A.
But our goal should be
to get everybody there.
Why have an arbitrary number that makes
absolutely no sense.
And another thing that makes absolutely
no sense is why we're
so focused on teaching
grading with a zero to 100 scale,
as if we have the precision to say
anything about that.
So if you think about it, when we grade
on a zero to 100 scale,
we normally look at the standard grading,
which says you need a 93% to get an A, a
90% to get an A minus,
an 87% to get a B plus, et cetera.
Why do we have those numbers?
And when we do it like that, what happens
is the entire class gets compressed
between usually 75 and 100.
And then we look at all the scores,
especially in a large class,
and we look for natural breaks because
we're so omniscient
about what these things mean.
And we know that this
moves people to the right area.
So we sit there and we look at grades and
we look at the boundaries
when we do our final grades,
because we usually never publish these
because then the
students will scream at us.
And we look and see,
well, this person had a 89.95.
And okay, you go, well, that's pretty
close to a 90. Let's give them a 90.
Except you realize in your class of 150
students that you've got 10 students,
four of which have 89.95.
And then just below that, you have 89.89.
And you go, ah, that's a big jump.
But think about that.
That is 89.95 versus 89.99 even.
That is 0.04%.
So the fraction is 0.0004.
How in our wildest dreams do we feel that
that is significant?
If we were to put error bars on this and
think thoughtfully of
what those error bars mean,
probably we can only measure learning
through exams and
grades and stuff like that
to at best plus or minus 10%, which is a
lot bigger than 0.04%.
Yeah, there's a great irony there that
especially in a lab context,
I'm always mentioning to students the
importance of significant figures
and thinking about the
precision of your measurement
and how the number of digits you use
conveys the intrinsic precision of the
tools that you're using
and for significant figures on a grade.
That's borderline meaningless.
I could be convinced we have two sig
figs, as you said, to order of 10% error.
Yeah, that's actually pretty achievable.
And hard to do but possible.
But if we were going to grade ourselves
on that kind of
effort, we'd probably fail.
And yet this is done in just about every
class all over the world.
And it's absurd.
And it's not right.
And so unless we can find a way to
magically improve the precision of our
measurement of learning,
which maybe can be done, I mean, you can
have precise measurements that are good
down to 10 to the minus 7.
Delta E over E or
however you want to measure it.
Look at the LIGO project, measuring
gravitational waves.
That was the hero
project of improving precision.
Unbelievable what they did.
But that took billions of dollars and it
was in an area where you actually could
make those measurements and
could improve the precision.
I'd argue it's a lot harder to measure
learning than it is to measure
gravitational fields.
And so why are we even doing it?
So the simplest thing, of course, to do
is not grade on a zero to 100 scale,
but something like zero to 10 or even
zero, one and two or a five point scale
or something like that,
where the data is a more meaningful
reflection of the actual precision that we're trying to measure.
And so, but it still comes back to how do
we measure learning?
And that's a deep subject.
And I'd argue that is the hardest problem
in education is how do
you measure learning?
And I wish more people
studied it, but they really don't.
They just keep giving tests.
So, and tests are a problem.
Maybe you can talk about
the problem with exams.
Oh, goodness.
I, you know, I first have to come from
the research perspective, having done
education research for enough years,
that there is this gap between measuring
learning in a rigorous
research context versus
what you have to do to survive as a
teacher with hundreds of
students in one semester,
developing a tool to precisely and
reliably and validly measure student
learning in one small topic
area is a years long effort.
And that's not something that any of us
have the luxury of time to do when we're
just trying to get through our day
and make sure, hopefully try to make sure
that our students are learning something
rather than spending all of our time
developing measurement tools.
You know, these measurements can be done,
but I don't think it's practical to
expect most practicing teachers to get
into so much of the minutia of the
psychometrics and the statistics and the
validation procedures to say,
"Yes, I'm actually
measuring learning with this test."
That's just not
something we can practically do.
So instead, you have to ask, "How should
we assess our students?
What can we do as regular practitioners
of this craft of teaching to say, "I
think I have a gauge, you know, a certain
level of understanding of which of my
students have learned how much."
So the other problem is that almost all
exams are focused on having a student
demonstrate a certain level of knowledge
on the day and time of that exam.
And very little thought usually goes into
the idea of retention.
And this is famously born out with Father
Guido Sarducci's little comedy skit, The
Five-Minute University.
I have none of you out
there have ever seen that.
Just type into Google, "Five-Minute
University," or in YouTube, and watch it.
And he makes the point that, you know, in
five minutes, he can teach students what
the average college graduate remembers
five years after they leave college.
And sadly, it's kind of true.
And there's a large body of educational
research that bears this out, that by
giving students exams that are largely,
largely factual recall and retrieval,
has caused generations of
students, myself included,
headed to the library two days before the
exam to cram like crazy to remember all
the things that I might be tested on.
Then I go and do a data dump on the test.
I do really well, but they show that even
two hours later, the students couldn't
even do 50% retrieval
of what they just did.
And two months later, it's down to like,
you know, 20% or some crazy low number.
And so that's the whole point of Father
Guido Sarducci's little skit.
And why don't we think about retention?
Retention is what really matters.
Now, in some fields where it's
conceptually based, we
can think about concepts.
And if you can understand a concept, the
chances are you'll retain that
understanding much
longer than just memorization.
And so the physics community has come up
with concept tests, which is really an
excellent way to do that.
But it's not so easy to translate that to
other fields because most of the concepts
in material science, for instance,
are really chemistry
concepts or physics concepts.
And we don't test those.
We try to test the ability to use those
concepts in the context of material
science to be able to answer a question.
Like earlier, we were just playing around
with the new voice model for chat GPT.
And Tim, you asked it about Martin site.
What is it?
And chat GPT was confused.
It just said it's an interesting phase,
but I don't really see it
on any of the phase diagrams.
So I'm not quite sure what it is.
And so chat GPT just scrubbing the
internet didn't get that
it's a metastable phase.
And you don't see metastable phases on
equilibrium phase diagrams.
And that's the whole point.
So there's an underlying concept there.
And so that's why chat GTP didn't get it
because it doesn't
really understand anything.
It just repeats things.
So giving exams is a dangerous way, in my
opinion, to assess learning.
And, you know, there is one type of exam
that's fantastic for actually
understanding whether
your students know something.
That's right.
But that's not a written exam, is it?
No.
Yeah.
I think you know
where I'm going with this.
Go ahead.
Oral exams can be so powerful.
Both on the student side
and on the teacher side.
You know, this ability to interact with
someone in real time to ask them
questions, to give them hints,
to see exactly where in the process of
working out a problem in
real time, where they get stuck,
where they have moments of insight.
That can be very powerful for revealing
learning in our students.
And it's the original
adaptive testing, right?
Yeah.
The student is crushing it.
You make the question harder.
You see how far they can get.
The student is struggling.
You back off and you give them support
and you see what you can
help them with collaboratively.
That's right.
We're trying to develop all this
technology to replicate that experience.
But either way, the oral exam, as good as
it is, is not very scalable.
If you've got a class of 100 students,
how are you going to have 100
half-hour-long conversations
with all of the
individual people in your class?
And that's the key point.
Some people say, "Oh, you only have to
give a 10-minute oral exam
and you can learn everything."
I'm not sure I agree with that.
And then they say, "Well, to grade
somebody's paper takes about 10 minutes,
so it should be scalable."
But I think you said the key thing.
It kind of takes more like a
half an hour for every student.
Because all of us who went and got PhDs,
most of us had a PhD
defense, which is an oral exam.
And that takes at least an hour,
sometimes two hours.
But that's a big body of work.
And I'll never forget, I was a math major
as an undergrad, and I
had a really interesting
professor who he was in the National
Academy of Sciences.
And he used to tell us
about thought process.
He brought up this thing,
the Epicurean model of thought.
And so he thought that when you were
trying to solve, prove a
theorem, very abstract, right?
And you would sit there and have this
experience of pounding your
head on the table and getting
very, very frustrated while you tried to
figure out the problem.
And often the best thing to do would be
to work for a few hours
and then go take a break,
do something else, go
to the bar, whatever.
And maybe in the middle of the earth, of
sleeping that night, you'll
wake up with this aha moment.
And he viewed that as the, he called it
the Epicurean model of thought.
I have no idea if this is
accurate, but it's a good model.
While you're pounding your head on the
table, you are taking
relevant ideas off that pegboard in
your brain and throwing them into a bingo
pot where like the
kinetic theory of gases,
you get random connections of
these ideas coming together.
And when the right idea comes together by
random, the power of the
aesthetics of that beautiful
solution are so powerful, they crash
through to your conscious mind.
So he gave us math exams.
We only had 14 kids in the class because
it was, you know, who was a math major.
And he would give us a piece of chalk and
the chalkboard and he'd
tell us something to prove
that was pretty easy. And we'd start
writing on the board and doing it.
As soon as he realized that we knew what
we were doing, he
said, oh, enough of that.
Let's do this. And you knew you did well.
If you could never finish any of the
things he asked you
because he could tell that you
knew how to do it. So I waste the time
and he wanted to push you
and he'd eventually get you
to a place where you had no idea what to
do. And then he'd start to
drop hints and see how we did
with the hints. And when it was all done,
because this was a very
subjective exam, he gave us two
grades and those two grades were averaged
for our final grade. And
those two grades were how good
was our conscious thinking and how good
the other grade was, how
good was our subconscious
thinking going back to this Epicurean
model that he told us about.
And that was where he gave us
hints. How could we synthesize those to
crash through to our
conscious mind to do the technical
aspects of actually doing the proof. And
so it was pretty amazing.
But that took a lot of time.
I mean, I was in there with him for over
a half an hour and I got a
C for my conscious thinking,
but I got an A for my subconscious
thinking. So I got a B for
the exam and I was pretty happy,
but I learned a lot. It was an amazing
experience, but it's just
not scalable to 150 people.
That's just too much time.
You know, that reminds me of one question
from my PhD qualifying exam. And
the question was very simple. You have a
gas of charged particles,
the box that the gases in
expands, does the temperature go up or
down? And so of course, having done
however many years of
physics at that point, I went straight to
the nuclear option. I'm
like, I'm going to write the
partition function for this thing.
Because if I know that I can calculate
anything. So, you know,
I go through the jumping through the
hoops of writing down the
states of the system. And I
start to write integral of, and then I
have this moment of I'm
never going to be able to do the
math for this. This is completely
intractable what I've set
myself up for. And one of the
people on the committee saw me have that
moment where I had
finally realized that, you know,
integrating this by hand was not the
correct approach. And he
says, Okay, good. So now just
think about it physically. Where is the
energy in the system? And
I'm like, Oh, yeah, huh. And
five minutes later, I was at the answer.
But, you know, it was just
one of those experiences that
sticks with you. Yeah, I wish we could
have more of our students to
have more of those epiphanies
on their exams instead of just grinding
out written calculations.
And speaking of written,
another option is to think more
physically and have them actually write
essays and write papers
about the concepts that underlie it. And
this used to be a pretty good way. Now
that also takes a lot
of time to grade. That's not so easy. But
the whole world has changed
in the last year. First of all,
now chat GPT will help the students write
those without necessarily
showing that they understood
it, but it will also let the instructor
grade them more
accurately. And maybe there's a way
with the oral exam, if you record it,
that we could transcribe
it and, you know, grade it
automatically. But unfortunately, you
still have to sit there
through the whole oral exam and
participate in the interaction. So until
chat GTP voice models can
actually do that interrogation
on a curated set of data, so it's
accurate, and be able to do assessment,
this hasn't happened yet.
Maybe it can. Maybe chat GPT will finally
give us a way to evaluate
the quality of learning.
But I think that's a long way off. But
it's a nice thought.
At any rate, so we've talked a lot about
what's wrong with
grading, what's wrong with exams,
why, you know, oral exams, which are
great, really aren't scalable. But what
do we do? Are there new
methods out there? And when I say new
methods, I've got to put that
in quotes because there are,
but they're really not that new. They've
been around for a while.
And so, for example, you know,
people talk a lot about mastery grading
as one of these new things
that we're going to do. But if
you go back in the literature, the person
who coined the phrase
mastery learning and first wrote
about it was none other than Benjamin
Bloom, who's very famous for his Bloom's
taxonomy back in the
60s. And so, but mastery grading is
starting to gain a lot of traction. So
why don't you tell us
what you think mastery grading means?
Interesting. So this is not
something I use in my classes.
We can get to that. But my understanding
of mastery grading is that
you have your course broken into
many small components. And for each of
those components, you're
assessing the students almost
on a binary scale of have you achieved a
sufficient level of proficiency on this
or have you not? And
then by totaling how many course
components students achieve mastery of
over the course of the
term, the quarter semester year, whatever
it is, then you can assign
a final grade based on how
much of the course content they have
mastered. Is that how you
implement it in your classes?
Sort of, but I'd say that the K through
12 community has been
using this for a long time,
especially with the younger students. And
so I know when my kids
were going to school,
they weren't getting grades until like
after fourth grade or
fifth grade until then they
were just getting one of three things
exceeds expectation meets
expectations or is approaching
expectations because the idea wasn't to
stigmatize the student with something
horrible, but rather
talk about where they are relative to
what the teachers wanted
them to do to give them flags so
that they knew who to give more attention
to. Cause you know, our
public school system doesn't have
that much money. Teachers aren't paid
that well. And most teachers find
themselves in classes
that are way too big for them to
adequately teach. So they do
a triage and that's exactly
what that is. You know, except unlike a
triage, the worst group,
they don't just leave to die.
They actually spend more time on that
worst group, a little more
time with the people who are
just starting to meet those expectations.
And then maybe even enlist
the help of those students who
exceed to help the younger students do a
little bit better. That's kind of the
idea of the one room
schoolroom, the schoolhouse, right? Where
all grades used to be mixed
together. The older students
would help the younger students. And, you
know, it's a pretty
powerful technique and I think it
works really well. So why don't we use
that in higher education?
That's kind of my thought on
the whole thing. Well, since you gave me
the perfect launching
point, I'll have to say that's
very close to what I do in my classes. I
am bringing together a few of
the topics we covered today.
Most of my assignments are scored on a
five point scale because
that's course enough that I can
actually be reliable about it. I can
distinguish between a four
and a five, a four and a three.
And my criteria are exactly what you've
just stated. I have
meets expectations, exceeds,
does not meet. And then descriptive
rubrics that say, here's what I'm
expecting. If you achieve
this, that's a four. If you exceed that,
that's a five. And there is a lot of
student angst around
that, especially at the start of the
semester. Oh my God, I got a
three out of five. I'm failing
out of school. No, it means you're almost
there. You're not quite
there yet. And it's something
that I find makes the grading go a lot
smoother because I can focus on the
qualitative feedback
that I give to the students about where
they succeeded and where
they still need to develop
more as opposed to spending my effort
trying to turn it into a
number. Right. And of course,
by using rubrics, you're moving towards
another form of mastery
grading. You're probably doing
something closer to specifications
grading. Yes, that's
right. So I'm, I'm giving the
specifications for what the different
levels of achievement are for each
assignment. And then
just comparing what the students produce
to those specifications.
So you've put in a lot of transparency to
your grading, even though
the students may be fearful
because it's new, it's actually a lot
less opaque than just, you
know, getting a percentage on an
exam where then the professor arbitrarily
draws the lines later on. There's no
transparency to that.
So that's pretty cool. And then there's
another movement that
has grown up around mastery
grading specifications, grading, and
there was a book published that talks
about this quite a bit,
and it's called ungrading. And the
ungrading movement is also
not a new idea. For example,
the president of University of Chicago
decided that grades were
evil back in the 1930s, and he
banned grades, but that only lasted for
about a year before they
threw the guy out because none of
the faculty could deal with it. But the
truth is that if you go
way back, grades didn't exist
until the late 1800s. And they were
started in England at
Cambridge and Oxford,
not grades, but levels, right? You
achieve some level. And do you
know where grades were? A, B,
C, D were first instituted in the United
States in the whole world.
I don't. I would guess University of
Chicago just for pure irony.
No, for real irony, it started at the
University of Michigan.
Oh, no. Yes. So we were
the leaders there as well.
Yes, we were. And it caught on like
wildfire. And at least that's what the
book ungrading says. I
will trust them. It's a fascinating book.
And so this whole
ungrading movement is to get rid of
all of those concepts. And if you take it
to its logical conclusion,
the logical conclusion of
ungrading means you don't give grades.
And instead, you just ask the students,
what do you think you
deserve? And it's kind of interesting
because, you know, that
will actually work sometimes,
but not at the University of Michigan.
And I watched a podcast that
Eric Mazur did on ungrading
with two faculty members from the Eastern
Kentucky University, which has a very
different demographic
than Michigan or Harvard. And they
actually talked about their students.
When they let their students
give themselves grades, they almost
always give themselves a
lower grade than they actually
deserved. But that's because they have
mostly non-traditional
students who, you know, are mostly
first-generation college students. They
know why they're in school
and they're pretty tough on
themselves. And a great comment, one of
the faculty members said, but this must
be great at Harvard.
Your students are like so highly evolved.
And Eric just shook his head
and said, no, think about how
they got into Harvard. They all got in by
focusing entirely on
grades. And so there's a lot of
dishonesty in that whole process. And
sadly, University of
Michigan is close to that. So it
just won't work to let students give
themselves grades because they'll just
give themselves A's.
I don't want to be that, you know,
cynical, but I'd say for 90%
of the time, it will be cynical.
So we can't quite go that route. It might
work some places, but it's
that to me is a bridge too far.
Yeah. As we're talking about students
grading themselves and the
sort of student obsession
with the A, it really makes me think
about the fact that there's
a very wide gap between how
important our students believe grades are
and how important grades
actually are not. You know,
that as we're sending students to their
employers or to graduate schools, the
large majority of what
we're writing in our recommendations is
not about the grades they
got. It's about what they did
and the interactions that they had. And
certainly when I'm writing
a letter of recommendation,
I'll say about two sentences of this
student got an A and then two
pages about the actual human
being, not the letter. That's right. And
what's sweeping the country
is this whole idea of holistic
evaluation of admissions. And for the
graduate programs, you
know, it's well known that
all grades are as a predictor is whether
or not students will score
well in exams in graduate
school. And it says nothing about how
well they will do with
research, which is what PhDs are all
about. And for research, we know that
we'd rather have a student who's
perseveres, who's fearless,
who embraces failure, learns from it, has
all these other things.
We would much rather have a
student who's overcome adversity in their
undergraduate career
to go from a pretty low
level to a high level that shows an
ability to overcome the kind
of barriers that are actually
important for doing research. Because
that's what research is all
about. Better get used to failing
a lot and embracing it because if we knew
what the answer was, it
wouldn't be called research.
So, you know, with all of that, I, again,
students have this, you
know, how do we educate students
that grades aren't important when all of
society is telling them
that they are. And it's really,
really sad. So anyway, we've talked a lot
about some background
about what you and I believe
is important in grading or not important
in grading. Why don't you
tell me what you actually
do for grading? You started by telling me
you have a multi-point
scale. Can you be a little
more specific about how you do grading in
your class right now? Sure.
Yeah. On the implementation
side for each of these assignments,
whether it's a report or a
lab notebook or a homework or,
no, actually, I guess that's about it.
The rubric will have a description of
what students should be accomplishing in
the assignment. So as a
concrete example, in this
last lab report that my students have
just turned in, I'll have a
line that says, for example,
must compare predictions of a theoretical
model to experimental data
and evaluate the accuracy of
the model. And then at that point, the
different levels of grading
are a five does an excellent
job of providing a convincing explanation
that combines multiple
sources of data. And a four
accurately compares the theoretical model
to the experimental data,
but does not bring in other
sources of information. And there's this
kind of tiered structure of
what will really convince me
that you know what you're doing and then
sort of taking away some of
those pieces of the argument
as you get to lower score levels. And an
important distinction that I
think is worth pointing out
here is that I'm telling students what
they need to have
accomplished at the end of the day. I'm
telling them that here's what will
convince me, but I'm not actually giving
them a list of boxes
to check, right? I'm not saying, okay,
you have to get some error
bars on this thing. And I'm not
telling them, oh, yes, you should run the
model with different model
parameters to see whether
there are numerical issues because that
that's part of what's being
assessed is do they have the
understanding of this process of science
to be able to do a
meaningful comparison. And sometimes
students look at this rubric and they're
like, so you want a meaningful
comparison? What does that
mean? But then that's a great question.
That's something we can have a
conversation about. And
when they're ready to ask that question,
and we can talk about it,
then that's a place where I think
true learning can really happen. So I
would argue that there's value in an
interactive course that
leaving some things not fully written out
explicitly in these
specifications is good
because it gets students to question the
meaning of the words in the
specification to help them
unpack any gaps in their own
understanding of it.
Right. So I think it's important to
mention that you teach a
laboratory class, and that you
typically have, you spend a lot of time
with the students. So
your teaching schedule is
four days a week, every afternoon, you're
in the lab with groups of
students and you typically have
groups of four in each team. Is that
right? Yeah, usually three to four. And
how many teams do you
have because you have them all rotating
on stations around to
utilize the equipment?
Yeah, typically a section will be four
teams and then I'll have however many
sections I have based
on enrollment, usually three. Right. So
you have like up to 16
students at any given time.
So you actually have time to interact
with each student. Because
how long are your labs are like
two hours? They're four hours, four
hours. Yeah. So you have, you
have a lot of time to interact
with each student. And that just you just
have to spend a lot of time. So in a way,
you're able to do oral exams with all
your students almost every day.
Yeah, I just don't call them that. And
that's right. That takes
away a lot of the stress around,
Oh my God, I'm being examined. No, we're
just having a conversation so I can see
what you get and what you don't get. So I
can help you get what you don't yet get.
Right. But you know,
this approach. Go on.
I was just going to say that this
approach is going to hit a
scaling limit where in a 100,
120 person course, there's just no way to
do this unless I had an army of grad
students who were all
trained in pedagogy and able to tease out
students ideas in the same
way. So that kind of leads to
the question of in a bigger class,
because like Steve, your classes are
significantly larger than
mine. What are other approaches that you
can employ for grading
that would work well with
a larger number of students? That's
right. So I have 140
students in my class this term.
We had a 24 teams of six people per team,
roughly. Some people,
some teams were five people.
It was in a very large room. And so, you
know, I've tried to
structure my course really around
the way that I'm going to do grading. And
so I think one thing that
was really important in what
you do, Tim, is a best practice that lots
of people talk about
that if you're going to give
assignments for a grade, it's better to
give lots of low stakes
assignments. So instead of giving
three exams, it's better if you gave 14
exams that were all equally
weighted, because that way,
if you do poorly on one of them, it
doesn't crash your whole
grade. And so by working with them
every single week and interacting with
all of them, you know,
you've kind of created a
large number of evaluations with low
stakes for each one that
conclude with your judgment,
where you can use your rubric to grade
them. And so I do try
to do the same thing. So
with as many students as I have, it's
kind of hard to just redo everything. I
have to do it in steps.
So I looked at what I had been doing. So
I give reading
assignments for every class session.
And I give, we usually have, I meet two
hours, twice a week. And
so each week has two class
sessions. And so I usually give two
reading assignments for
each class session. Each week,
I have a homework assignment. Each week,
not the same week, but I
have a what I call a readiness
assurance activity, which is a more
formative based test, but it's weighted heavily towards the team
participation. I also have outcomes
assessment reflections, where I, at the
end of every module,
I have students reflect on what they
actually learned. So
again, my whole course is set up
to focus on retention, because I think
that's really important.
And one way to get retention
is to have students revisit a concept
multiple times over a
relatively long period of time.
So all of my modules, so a module starts
with a one week effort.
And that module then extends
for three weeks before they actually
finish all the things. So while they're working on other
aspects of that module, a new module is
being initiated the next
week. At any rate, I have
roughly 24 reading assignments in the
term. We have about 12 weeks, 14 weeks,
something like that.
So I have, you know, 24 reading
assignments. I have 14 homework
assignments. I have 14 readiness
assurance assignments. I have 14 outcomes
reflection assignments.
And then I also do projects.
And I've been doing three projects, but
the first one's just for practice. I'll
be changing that next
year, mostly because drop bad deadlines
destroy teams in my class for the first three weeks. So
my first project has pretty much always
been a bust because the
teams got disrupted so much.
It wasn't working, but I finally figured
out how to fix that. I can
now go in the day after class
starts and drop the enrollment number so
nobody can join the
class. People can still drop,
but nobody can add my class after the
deadline. And that's
going to let me have three projects
which will be worth two units each. When
it's all said and done, I
add this all up and I have 72
units make up my course. And what I tell
the students is they have
to get an A on every single
unit. Now I'm a little flexible there and A minus is just as good as an A because I can't really
distinguish. So they pretty much have to
score like a 90%, but it's
not a strict 90%. I just do it
as a zero one or two and a two is an A. A
one is approaching expectations and a zero means
they just didn't do it. And I try to give
each of those units where
I can, I can't always do it
where I can. I give them the ability to
retake them multiple times
until they achieve a two.
All the reading assignments, which are,
you know, 24 out of 72, a big chunk, all of that
are reading assignments on perusal and
perusal gives me the ability
to allow them to continually
add annotations that score highly or to
do high quality interactions.
So they get credit for being
social because I believe social learning
is a powerful tool. But I think that's a really
powerful tool. Social learning is hard
baked into our DNA.
Writing's only been around for 5,000
years. Social learning has been around
for 3 million years. That's
our primal instinct. That's
how we learn. And so by being social, by
interacting with other students on
perusal, I really highly
value that. So they can do that up to a
deadline. They, you know, once the
deadline goes too late.
So that's how, that's how I'm trying to
do mastery grading. I also
have these reflections. So
what I grade is the homework reflection.
I don't grade the
initial homework effort.
I only grade that on effort, not on
accuracy. They have to scan
their work and put it into
canvas. They have to do it, but I pretty
much just, you know,
virtually weigh the packet of
information to see that they tried. And
that's just a check mark.
They have to do that if they
want to get an A on the assignment or
pass that unit. If they
don't do that, it doesn't matter
what they do later on. They're not going
to pass it. Then they have
a homework activity in class
where they make a better homework
solution with their team. And I hire all
of these instructional
aides, our senior level material students
who were just in that class two years
earlier. And so they,
I have one undergrad for every two
tables, maybe every three tables, and
they walk around and
provide help. But it turns out the
students don't really need
help. The help they need is those
students that are introverts and don't
talk need to talk. So I tell the whole
class, we're going to
do this. And I train the instructional
aides to pull those folks
aside and say, Hey, you know,
Johnny, you're not talking enough. How is
anyone in your unit team?
How is anyone in your unit team
realize that you're adding value? And
unless they realize you're
adding value, they won't respect
you. And good teams are teams that
respect each other. So we help the
introverts become less
introverted. We also talk to the
extroverts and explain how important
learning how to listen is.
And it's sort of like an audio, we do
compression on the high
end. And we do, you know,
we boost the levels on the low end, to
try to make teams that
perform better, because ultimately,
they need to teach each other these
concepts, if they want to
have retention. So that's what I
focus on. And does it work? Well, I'm not
there yet. I still have a
lot of problems with it.
But I'm getting there and I'm learning
right now. I have way
too many A's. So although I
want to have everyone get an A, I know in
my heart that the
students who a lot of students
are getting A's who really don't know
anything. And how do I know
that? Because when I walk around
the classroom, I will pick out a few
activities that I give them that I know
are very conceptual.
And I go and make them write stuff on the
whiteboard. And I talk
to every single team,
takes me about a half an hour to get
around the whole room. But
I can get a really good feel
for whether they learned anything from
their reading, whether they
learned anything from what
I talk about. And sadly, sometimes I walk
around the room like I tried
to tell them with extrinsic
semi-conductivity, where the acceptor
level is, where the donor
level is, where the Fermi level
is for an intrinsic semiconductor, and
where it moves when you have
an extrinsic or acceptor or
donor. And I try to get them to draw
these pictures on the
board. And then two days later,
I give them that exact problem on the
readiness assurance activity.
And on the individual round,
30% get it right. So I failed. I thought
I was giving them one-on-one instruction.
I thought they understood what I was
saying, but other factors
always come in. This was on the
second to last day of class. And I'm
sorry, students, by the time they hit the
very last week of the term
are pretty much brain dead because their
other classes are
weighing so heavily on them.
They're cramming like crazy. And it's a
really tough environment
to learn something new that
last week of class. So, you know, we all
suffer from this. I'm sure you've
observed that phenomena
as well. Oh, sure. And so those are my
problems. But what I'm
going to try to do next time,
I'm going to go back to something that's
really like an oral exam. You know,
it's called the Feynman technique. It's
what Richard Feynman told all of us.
If we want to know that we actually
understand something, we
should use the litmus test of,
can you teach this to a five-year-old?
And if you can teach
it to a five-year-old,
you probably really deeply understand it.
And if you deeply understand anything,
chances are you're going to retain that
for a long time. So I'm
going to play around with
things like I do for my projects, which
is whiteboard video. And I'm
going to try to get students to
make short whiteboard videos. My
whiteboard videos for my projects are
like two minutes long.
I think for certain concepts that I'm
going to ask them to do, I'm going to
embody the idea of a
YouTube short or a TikTok video. They
have 30 to 45 seconds to teach the
concept. And we'll see how
that goes. I haven't quite figured out
how to do it. But the truth
is they're all pretty skilled
in editing video. They come to college
knowing how to do that. So
let's exploit those skills.
Let's get stuff up. And maybe they can
help each other enough. And
if I build a library of the
good ones, those can even help students
learn small little chunks. But we'll see.
Yeah, I love that idea of producing short
form content. One trouble
that I have in my course with
the specifications that I'm using is
students come in believing
that more is better. And,
maybe I'll just write 30 pages instead of
20. And then all the
right answers will be in there
somewhere. And they will, but that's not
really helping the
underlying problem. So this is
something that I'm always trying to find
ways to do better in my
classes to give assignments that
really have students distill something
down to just the minimum
essence of the concept or the
problem of the calculation. And do it in
a paragraph. Don't even
do it in a page. If I could
really get them to be more concise and to
think about how short can
I make my product instead?
How long can I make it that would serve
all of us very well, but
we'll get there someday.
We all know it's much harder to write
something that's short
than something that's long.
And the same is true for video or
anything else. The thing I like about
video is that it's way
ahead of where chat GGP can quite get to
right now. I can start to
make videos, but they're not
going to be very good. And so I don't
have a problem with having
the students use chat GPT to
make their script, but to make the script
and make it concise
enough for a 30 second spot.
That's a pretty challenging thing that
requires you actually think
about what chat GPT is telling
you. And as long as you think about it,
that's a really good
activity in my book. Because once
you've thought about it, you've gone
beyond just memorizing and
you've edited what chat GPT says.
And so then chat GPT becomes a tool and
it's a useful tool, but you
must edit it. You must think
about what it's saying in order to make
something very concise. So I
think that's a good strategy.
Evaluate whether it's accurate, decide
how it can be done better, and then
actually execute the
doing it better. And that's a really good
use of time for learning.
And I'm even thinking maybe I
can come up with an activity where they
use their short form video
to teach other people on their
team and have the other people on the
team evaluate the quality of those
videos. So we'll say,
not sure how I'm going to pull this all
off. But that's what I'm
thinking about for next year.
What are you thinking
about for next year?
Ah,
I'm thinking about the biggest change
that I want to make on
this topic of grading is
doing a better job of mapping this five
point scale onto a
numerical scale that will make sense
to students who have grown up in zero to
100 land without getting
to this compression problem
that you were bringing up at the
beginning. If I can actually spread out
the individual numbers
over a wider numerical range so that the
values are distinct and have
separate meanings from each
other without getting students into the
panic of why do I have a C minus
territory? That's really
what I'm focused on right now is to help
students distinguish the
score on the assignment
from the grade you're going to get in the
course. Because as
much as I would like to
just tell them that grades don't matter
and have them actually believe it,
that's an impossible challenge. So have
to work around, work with
what our students believe
instead of what we wish they believed.
Well, that's a good point
to end with. I think we've
been talking for a long time now longer
than we intended. It
sure is hard to be concise.
Yes, it is. Especially when it comes to
something as intractable as grading.
So I guess with that, we'll say goodbye
and looking forward to our
next chat. See you later.
Okay. See you next time.
Creators and Guests
