Undercooled: A Materials Education Podcast

[MUSIC PLAYING]

Hello, and welcome to

another episode of Undercool,

the materials education podcast.

So today, Steve and I are

hanging out in the office.

And we're going to talk

about the acronym that

is no longer an acronym.

It's just a name all by itself.

That's right, ABET.

Maybe once upon a time

it stood for something.

But we'll get into the

history with Steve here.

But today, we're going to talk about

ABET, what it's good for,

what it's useful for, what

it might not be good for,

and how it can inform what we

do in our materials programs

and help us teach better.

So Steve, to kick things

off, why don't you just tell us,

what is ABET and why

should materials departments

care about it?

So ABET used to stand

for the Accreditation

Board of Engineering and Technology.

Now it's just ABET.

And it's because they do

more than just engineering

and technology.

They do applied stuff as well.

But whatever, it's

really the same organization.

It's a federation of 30 to 40

professional societies

that get together and

they make bylaws and set up

all the rules and

regulations for doing accreditation

for engineering schools.

And ultimately, it's about making sure

that the people who are

designing your bridges,

your airplanes, your boats, actually

know something about what

they're supposed to know.

So as a country, as a

society, as the world,

we can actually trust the engineers

that our educational

institutions are putting out.

That's the big reason.

And it's even

sanctioned by the United States

to do accreditation for all US programs,

engineering programs.

But ABET also performs many

international accreditations

at the request of

schools in other countries

because it's become

kind of the gold standard

for engineering accreditation.

And other countries want to have--

I mean, it's kind of funny.

They want to say, we have the same

accreditation as MIT does.

So for whatever

that's good for, but we'll

get into that later on.

So why do MSE programs care about this?

Well, they should care because we usually

have a few students who want

to go into careers where they

must become professional engineers.

And as you know, Tim,

the TMS has a committee

that sets the

professional engineering degree

for materials.

It's really metallurgy mostly.

And although I don't think

more than 20, 30 people a year

take that test, it's

there in case they need to be.

Where would they need to do this?

It would be companies like Exponent.

These are consulting companies that often

do a lot of litigation.

When something goes wrong, in some

company or some person,

who's another company or

another person, goes to court.

And in court, there are all these rules

about who's allowed to

testify as an expert witness.

Anything involving the government,

only people with accredited--

with professional

engineering are going to count.

But it's also to impress the jury.

So the jury is going to want to know

that it's a professional engineer.

And guess what society

is also a member of ABET,

the Society of Professional Engineers.

And a long time ago, they insisted that--

well, hey, they make their own rules.

If you want to be a

professional engineer,

you must have graduated from an

accredited engineering program

or have 11 years of

equivalent experience.

So do we really want our students

to spend 11 years

before they can start a job?

They might want to start

right after in our department.

Maybe 1% or 2% of our

students every year want to do it.

So we do it for them.

And it makes parents and

alums feel really good.

So those are the two main reasons

why we partake in accreditation.

OK, yeah, it definitely makes sense

about wanting to give

students the opportunity

to earn certifications, to have access

to different career

paths, different opportunities

down the road that

ultimately rest on the program

that they graduated

from, having demonstrated

a certain minimum

quality of the education that

is being provided there.

So that seems reasonable.

Now, my understanding

is that to get certified

as an ABET accredited

institution, part of the process

is having ABET, essentially,

inspectors come to the program

and do an on-site physical visit.

What do those visits look like?

And what's the hardest

part of preparing for a visit?

What do you have to do to get ready?

That might not be obvious.

So that's a great question.

And first of all, it's

very important to realize ABET

doesn't accredit

institutions or departments.

They only accredit programs.

So they look at the

degrees that are being granted.

And in our case, we have

one material science degree

that says a bachelor's in material

science and engineering.

That's a program.

And that is what is accredited.

And that's all ABET accredits.

So what you need to do to be accredited,

it's a little different the first time,

but we've been

accredited for so many years.

I wasn't even here when

we first had accredited.

I've been involved in some, but I

think you need to have graduated at least

one student before you

can be accredited.

And when you go for your first

accreditation visit,

you're visited by two

evaluators, not one.

So that's a little different.

But for most of us in most programs,

it's about 125

accredited materials programs

in the country.

And the vast majority of

those have been accredited before.

So what you need to do, your

institution will call up ABET

and say, hey, we're

ready to be accredited.

We're going to start the process.

Here are all the

programs we'd like to do,

because ABET does a

visit for all the programs

at an institution at the same time.

The first thing you need

to do once all that-- that's

all Dean stuff, so

don't worry about that.

First thing a program needs to do

is start six years before that moment,

because they really need to

be preparing for this right

from the day after their last visit.

And I'll get back to that.

So what you need to do is create what's

called a self-study.

And ABET gives you a template to fill in.

ABET is all focused

around their criteria.

They have eight criteria.

I'm not going to go through all of them.

The most important

ones are the first few.

So criteria one is

all about the students.

Where the students come

from, how are they advised,

what kind of tracking do you do, how do

you handle mental health

issues, how do you do

admissions, all that stuff.

And the second

criteria is all about program

educational objectives.

And these are those statements that

talk about what a graduate

should look like a few years

after they graduate.

What's our aspiration for that?

And those don't even need to be measured,

because some of our aspirations are we

want them to be creative.

We want them to solve

the world's problems.

And how do you

actually measure those things?

But it's important to

have high aspirations,

because that kind of

drives the whole thing.

I actually believe that the objectives

are the most valuable part of the ABET

process for a program.

It's also one of the easiest

parts to be compliant with,

because all you need to do is consult

with your constituencies

at least once every six years, and ask

them, are our objectives as written,

meeting your needs as

a constituent group?

Our program has three

constituencies, the students,

the faculty, and our alums.

Some programs get crazy, and they say,

their constituents are the universe.

Well, how are you

going to ask the universe

what their needs are?

You know, we just can't

get to some of those planets.

So that would be a very

unwise thing for a program to do.

It's great to just have three.

It works.

But it's so important

that you meet with them

and document that thing.

That's actually easy to do.

The third criteria is the simplest,

because ABET says, here

are the student outcomes, one

through seven.

And you can add more,

but what crazy programs

can add more and add

more work to their plate?

So just do what ABET asks you to do.

You just list it.

Those are your outcomes.

I think there's a table.

You show how they're

related to your objectives.

That's easy to do.

So that's easy.

It's the fourth criteria

that usually stumps everybody.

And that's the criteria

for continuous improvement.

In that-- and it's very

short if you read the words

the criteria.

It's not much language.

But it says that you have

to have a outcomes assessment

process that is performed regularly.

And in ABET, that

means at least two cycles

during the six years

between when you started

to when you get accredited.

And it must be appropriate.

And that's a big catch word.

That could mean literally anything.

Yes, it can.

So those are the two words that usually

catches most programs.

And so it really all comes down to,

how are you going to assess

the outcomes for the students?

How are you going to do it

in a way that is regular?

And how is it appropriate?

So after being on many ABET visits

and going to many ABET symposiums

and even being on the board of directors

for ABET for a while,

I've gotten a good sense of what they

actually mean by that,

even though it's not

explicitly written down.

So my takeaways for a lot of this

are that what ABET

really cares deeply about--

and you'll get this from

anyone from ABET you talk to--

they care deeply about

continuous improvement.

This all came from ISO 9000.

And it turned into the EC2000, ABET,

all of a sudden, with these words.

But basically, they

changed ABET dramatically

around the year 2000, where they wanted

to be more like what industry does

for their continuous improvement.

And there's a lot of good ideas in there.

I have my own personal beliefs.

I don't believe students in education.

It's like a product

that's on an assembly line.

So it's a little harder to measure

students' achievement

of outcomes than it is

to measure quality control

on the dimensions of a

part, the hardness of the metal,

all those things.

Those are very easy to measure.

Measuring learning is really hard.

And actually, none of us really know how

to do it in a scalable way.

So that's the hardest thing.

But what ABET cares

about is they at least

have a process that makes a

very serious attempt to measure.

And what ABET cares about

is you must, in the criteria,

it says that you must

measure the extent to which

the graduates of your

program have achieved the seven

outcomes.

Now, let's unpack that.

The extent to which ABET doesn't say

anything about that everyone

has to have an A. They

just say your program needs

to know how well your

graduates are doing so they can use

that information if you need to to

improve your program.

It's a diagnostic.

The other thing is the word "graduates."

They're not looking for

learning about what students

at the first or second year are doing,

because as you know, it's hierarchical.

You build on the knowledge.

And the kinds of things

and the outcomes are--

the outcome one is to solve science and

engineering problems,

complex engineering

problems, which usually

means open-ended problems.

That's not something we

expect our freshmen to do.

It's something we expect

our seniors to be able to do.

So it really makes-- because of that,

it makes little sense to do

assessment in the first or second

year if all you care about is

meeting the objectives of ABET.

Now, there's lots of people who might

want to do assessment

earlier for their own purposes,

and that's fine.

And in fact, the mantra--

at least it used to be--

I hope it still is-- at the TMS

Accreditation Committee,

used to be improve your

program for yourself first

and worry about ABET later.

Because if you're doing a good job

improving your program

for yourself, you should have no problem

documenting your processes

and showing ABET that you did it.

The converse is sort of

a fool's errand, right?

To just do it for compliance alone is

kind of wasting your time.

If you're going to do

something, make it meaningful.

So that's what you

should really be doing.

But at any rate, this is

the number one criteria

that shortcomings are

delivered to that

there either is no good process or the

program only did it once

instead of at least twice,

or that it's not appropriate.

And what they mean by

that is either doing it just

running through the paces, but it's not

giving them any information.

The last part of the criteria says

that you must use the results of the

measurements, the assessment,

as input to your

continuous improvement process.

And you must have a continuous

improvement process as well.

The last line of that

criteria is the most valuable.

It says you can also use as input

anything else to improve your program.

Now, this has been

debated by many people at ABET,

but ABET has made it exceedingly clear

that the word input is

there and not output.

So if you do your assessment and you

analyze your results

and you show that everyone is doing great

above whatever threshold

you decide is important,

then you may not be able to use the data

you collected to

actually improve your program.

And that's OK.

Because then you can

bring in those other inputs,

information from your advisory board or

from industry partners

to get that information instead.

And so we have really good students.

They come in, they all take our intro

engineering course, Engin 100,

which is really a fantastic course.

It's really an English course, a

communications course,

but it's cast in a framework of a design,

build, test environment,

teaching freshmen how to do engineering

design right from the get-go.

So our students who take that, they learn

how to work in teams,

they learn about ethics, they learn about

design, they learn

about doing experiments.

They do everything that our

ABET outcomes ask us to do.

So by the time they come in our

department, in Michigan,

they don't join our program

until they've had at least one term.

Wow, they're amazing.

And I remember the days before Engin 100

when it wasn't that way.

I'm sure they were dark times.

Yes. And so things have really gotten

better in the sense of at

least outcomes two through seven.

Outcome one is the toughest outcome.

Outcome one is using engineering, math,

and science to solve

complex engineering problems.

That's difficult.

That's the one we beat

on the students hard.

We have very high expectations for them.

And so the scores for that outcome are

always lower college-wide

for all of these things.

So all that's kind of cool.

So at any rate, you know, that's what's

the most difficult part.

And so we've developed a whole new

approach to doing this.

We started this six years ago.

We actually started in our department

earlier, but we've rolled it out and are

now doing it on a college-wide basis,

automated, trying to follow some basic

principles to make sure our process is

completely sustainable.

And so this is the process for measuring

the student outcomes.

Yes.

OK, so from the instructor side, I can

see my perspective on it.

But I want to actually come back to

something else you said earlier, which is

the priority order of what to do for

improving the program

versus what to do for compliance.

And when I'm thinking about making

changes, hopefully improvements to my

courses, I'm always saying, well, what do

I really want to do with my class?

And then looking at the ABET outcomes

and saying, is there an outcome that this

change I'd like to make

happens to be well tied to?

And if so, to me, that's a good indicator

that it's something worth pursuing

because most of the ABET outcomes are

pretty transparently things that I think

we should want our students to do.

We should want them to consider societal,

economic, environmental implications of

their engineering work.

We should want them to design experiments

that have scientific validity.

These are just good things to do anyway.

So as I'm thinking about my courses,

anytime I can point to an ABET outcome

and say, by the way,

I'm also achieving this.

In addition to just doing what I believe

is good teaching, that's always my anchor

for how to make those changes.

And that's great.

There's kind of two viewpoints of this.

There's the ABET viewpoint that they

believe that they're driving all of our

education by mandating these outcomes.

But, you know, their

outcomes are kind of motherhood.

It's kind of obvious.

And I'd like to think that our program

actually has many more outcomes beyond

just what ABET requires.

And I think any good program will, of

course, do all the things ABET wants.

But the real improvement of the programs,

at least my experience here in Michigan,

has never really come from doing the

assessment of the outcomes.

The real improvement comes from people

just like yourself having that good

attitude of trying to do what's best for

our students in a broad range of

areas.

So we have a very robust undergraduate

committee that reviews

constantly our curriculum.

We do curricular reviews apart from ABET

because we think it's important.

We try to have meetings where we put

together people who teach Thermo and

Kinetics and ask, are you getting the

right kind of

background in your students?

What's missing?

And you know this better than anyone

because you're teaching a math course

because we're not getting students coming

into our Thermo and our

Kinetics courses with enough math.

And so we're going to tailor the math

they need to supplement from what the

math department gives them so they can

perform better in our Thermo

and in our Kinetics courses.

Another example of massive improvements

in our system of our program didn't come

out again from ABET but came by the work

you did that we talked in another podcast

about your alloy

design module in the lab.

You worked with the people who taught

Thermo because while they're taking

Thermo, our students are taking your lab,

whether you're learning how to use tools

like ThermoCalc and design their own

materials, their own alloy.

Using those thermodynamic principles

they're learning in Thermo actually make it

pour dog bone test specimens and test it

and then try to understand why it didn't

work because it rarely works.

But you know it will once you get really

good at it but you've

got to start somewhere.

So how is that captured

in our assessment results?

You need to go way beyond just

assessment. You need to you know be

creative and innovative and you need to

inspire a culture in a program of the

faculty caring about

their undergraduate students.

And I am so happy to say I think we've

got an amazing culture here

at Michigan doing just that.

So the way I try to document that I know

that every faculty member innovates in

every single course they teach.

And so when it comes time to write the

self-study I put a note out to the

faculty asking them to write me half a

page to a page of what they're most proud

of in developing in

the previous six years.

And that's the I think the best part of

our self-study because it's honest it's

from the heart. It's the real actual

improvements that all

these individuals do.

Often with the help and support of the

undergraduate committee or other things.

But what a great way I think to document

all of that and to really demonstrate

that we're doing massive improvements to

our program all the time.

And it's coming from what the students

tell us because we have we have town hall

meetings every year with

our students to hear comments.

We heard a comment last term that the

BioMed course is actually too biological

and not enough material science.

And guess what we're acting on that.

And so Brian Love who also agrees with

that assessment even though he's taught

the course before is actively trying to

change that course now.

We also get comments from our external

advisory board who are people in industry

who are giving us heads

up about things they need.

We get information from our alums with

whatever careers they followed.

So it's you know it's a lot of good stuff

that we use to improve our program.

But I view this outcomes assessment

process which we're mandated to do is

actually a critical part of this process.

And the way I think about it is these are

the diagnostics that if we don't do we

can get ourselves in serious trouble just

like engineers design that little

material inside your brake pads.

To start squealing when

the brake pads get small.

You need that there even if you change

your brake pads

religiously and never hear it.

You want to know before something bad

happens that it's about to happen.

And that's how I view our outcomes

assessment at the very end.

I talk about the

future of what we can do.

I think we can even learn more from our

outcomes assessment.

Well let's get to nuts and bolts for a

minute here because we have this big

picture aspirational vision of here's

what all this assessment accomplishes.

Right. Here's what it helps

our program get better at.

But there is still

that implementation layer.

The actually doing it which might not be

obvious especially if a program is trying

to step up their game in terms of having

an easier more efficient time with

checking the ABET boxes.

While doing great teaching.

So how do you actually

do outcomes assessment.

How is that

implemented at a practical level.

That is absolutely the critical question.

We all know the biggest problem with

doing outcomes assessment is

getting our faculty to do it.

The only people who can really probe the

students in an efficient way where we

distribute the load of the work is to

have every single

instructor of any course.

That we're using to assess our students.

They have to do the work.

So it comes down to some fundamental

concepts to make a build a process that

everyone will participate in in the

easiest possible way so

that it becomes sustainable.

And we do it all the time.

So the first thing is although a bit you

know they don't tell you how to do this.

They just tell you have to do it.

I've seen a lot of suggestions and I

don't like most of them.

Like ABET, symposia.

When they say oh well you could do

outcomes one and two this term and

outcomes three and four the next term and

then new outcomes five and six the term

after that and do that and then cycle and

by the time you're done six

years you've done it two times.

The problem with that is everyone forgets

what they were supposed to be doing

because we're creatures of habit.

So we need to do all of the outcomes

every single term in my opinion and find

a way to make it as easy as possible for

our instructors to actually do the work.

So that's what I first make it easy for

the faculty and instructors

so that actually gets done.

The next thing you know we need to do is

how do we do this and

also make it meaningful.

So what we've done is we built and a lot

of programs do this by the way I think

people have learned over the years.

So it's pretty standard now.

We build a matrix of we look at all of

our required courses and all of our

elective courses and we assign we try to

assign no more than one

outcome to any given course.

Now we can't do this for all courses

because we lean very heavily into our

design and our lab courses because those

are the courses where it's more

appropriate to measure things about

teams, communications, design designing

experiments, ethics.

So those are outcomes two through six,

two, three, four, five and

six. That's five outcomes.

But luckily we've got two lab courses and

we have two design courses.

So we're able to break it up so they only

have three outcomes to do.

Some programs wait until the last year

and dump this all, all seven outcomes on

the capstone design course, which makes

sense because you're measuring with our

graduates the extent to

which they've learned it.

But it makes it

unsustainable in my opinion.

Right. It's even more work for the person

who's already doing the hardest class.

Exactly. And that's just not fair.

So we've broken it up.

So our idea is let's collect a relatively

sparse data set, but do it every single

term so that we end up with a massive

amount of data after six years.

We do this 12 terms.

And I have to say it kind of works well.

So that's the first thing.

And we also only do assessment in our

junior and senior classes.

We assess outcomes one

through six in our required courses.

So regardless of path, every student is

assessed in our required courses because

they all have to take it.

Then our elective courses, because we

want to distribute the load again, we

have outcome seven is done in all of our

which is lifelong learning, learning,

different methods of learning, all that.

And so we have all of our elective

courses measure that outcome.

So regardless of which elective courses

our students take,

they're being assessed.

So we do assessment of all outcomes

across regardless of path

for all of our students.

So once we've done that,

we need one more thing.

How can we make this even easier for our

for our instructors?

And that has to do with the actual

mechanics of how they

first they have to create.

We expect our faculty to create the

assessment measures.

We have a choice.

We could hire assessment professionals

and have them tell us what

the assessment should be.

Or we can ask our faculty.

I happen to think our faculty are better

positioned because

they're teaching the course.

They know what they want

their students to learn.

And after all, don't we want all of our

students to be able to solve complex,

open ended problems?

And all of our faculty do that.

So they should find something within the

context of their course.

So it's meaningful for them for their

course parameters, but use that to

measure the particular outcomes.

So what I've done is I

made a series of videos.

One short video for every single outcome,

explaining to the faculty good, good

approaches to build

an outcome assessment.

And that assessment measure, this measure

for the assessment, it

can be a homework problem.

It can be an exam problem.

It can be an activity that's not even

graded for the course, whatever the

faculty member wants to do.

But if it's outcome one, for instance, it

must involve a complex problem.

And I show them the definition of what a

complex problem is to ABET.

And simply say, if you look at that, it's

really just an open ended problem.

And our faculty are great at

writing open ended problems.

So every faculty member, and they usually

only have to do this once, because if

they teach the same course over and over,

they can keep using the

assessment tool if it's good.

And they can talk to other faculty if

they're inheriting a

course and borrow theirs.

That's OK.

But in each case, we have some faculty

member to make a document where first

they write down what is the

outcome they're assessing.

Because you want it first

and foremost in their minds.

Then what is the actual assessment?

And they write down that problem in great

detail, exactly what the student would

see, what you're

asking the student to do.

And then finally, after that, they write

a short little paragraph explaining why

they believe this is

an excellent outcome,

a measure of the outcome

they're trying to measure.

And again, that's just to make sure that

it's present in their minds and they've

actually thought about it.

Once they do that, they put that in a

document and make a PDF.

And at the end of the term,

they upload that document.

So they'll get a link.

This comes from our college because we've

developed this system

across all of our 12 programs.

And the college has our matrix and the

every program has an ABED coordinator on

the ABED coordinator for our program.

So every term I review the list of who's

teaching what courses that

are being used for assessment.

I confirm that with the college.

They send a special link to each

instructor with everything pre-filled.

So what term it is, what course they're

teaching, their name, all that stuff

that's all in there.

It's done for the faculty member.

All the faculty member has to do is

upload the PDF of what was the metric.

And they have to upload

the scores of the students.

This is where it gets tricky.

So we want to know the actual unique

name, the student

identifier for every score.

This is going to be critical for another

onerous thing that ABED has made us do.

ABED demands that we

disaggregate the data.

How does that mean?

That means that we can only consider

students who are in our program when we

do the analysis of our results.

They don't want to contaminate it by a

graduate student is in the class, by a

student from another department.

I still don't understand

why, but we have to do it.

If we know the unique names of every

student, we know that information.

We have a big data

warehouse that's got all that stuff.

And in the back end, we can easily filter

the students who are in our program and

not making a faculty

member do that is insane.

Because if you have 60 people in your

class, you don't know who they you don't

know what program they're in.

And you don't even want to

know what program they're in.

You want to treat them all equally.

You don't want to have a bias.

Oh, they're graduate students.

They should do more work or oh, these are

students from, you know, from.

Well, I won't say the name, but that

other major that we like to pick on, you

know, that we know we're going to you

don't want to know that you want to know

that there are students in your class.

So and plus, it's really hard.

A faculty member is going to have to

download, you know, look up

what program they're all in.

What a pain.

It's easier to just do your

whole class and dump it in.

And now we're

launching a new way to do this.

So if you have a spreadsheet, you just

highlight the two columns, the unique

names and the score.

And we ask all faculty to put in a score

normalized zero to 100 so we can combine

the scores with other courses to see how

they're doing across an outcome.

They just highlight

those things and copy.

Go to a line. I never knew you could do

this in a Google form right in the line.

You just paste it and it's kind of like

comma separated values.

But in the back end, they use regular

expressions to parse the data to put it

back into a spreadsheet.

We're doing this because these

spreadsheets we were getting from faculty

in the past were all over the map and it

became really ugly for the back end

people to deal with this data.

So this should be

much better at any rate.

The way the reason it works, because the

first thing you might think of is, oh, my

God, that's a FERPA violation.

That's student records. You can't make

that public. And we don't.

But the only way it works is because our

university signed a legal agreement with

Google to make sure that the instance of

Google that we have is FERPA compliant.

So we're allowed to use Google mail from

the university to talk about student

issues and all of that.

If your university doesn't have that

agreement, you'll have

to find another solution.

But it works really well for us. And that

student identifier is incredible, as

you'll see in a few minutes.

So that's all. But from a faculty's point

of view, they just give that assessment,

collect that data, upload a PDF of their

metric and upload the student data.

And they're done. It takes very little

time. And believe me, I remind everybody

to make sure they do that assessment so

they have that data.

And then it works. And after we've done

this now for six and a half years now, we

have 100 percent compliance, except the

very first year, one of our 80 year old

faculty members who didn't understand

Google Forms didn't do it.

But that's pretty good. Yeah, I have to

say on the user side.

Being tasked with three outcomes per

semester due to the lab class, it takes

me an hour, maybe an hour and a half tops

to do this once a semester.

And as you said, I just I

have my assessment items.

They're already in our

learning management system.

So I just grab the student scores,

normalized to 100, grab the student IDs,

paste columns in a sheet,

upload to our collection form.

And I'm done. It's really quite painless.

Yeah. And from the ABET coordinator

point of view, I just go to the Google

spreadsheet that they build and I can

instantly see who's

done it and who hasn't.

I can click on the documents

and check them very quickly.

And if there's a problem with a faculty

member, I just go visit the faculty

member and talk to them

and help them make it better.

So it's great. So the really cool thing

is when it's all done, there's this

program called the A BET.

It's called Tableau that

I'm sure many places use.

It's, you know, a database program that

ingest spreadsheets and you can have you

can build a user UI that does different

filtering on what you see.

But the data that we care about, what we

want to know is we want to be able to see

a histogram of the actual values of the

scores that our students got,

where every single piece of

data is on that histogram.

In the old days, we used to report an

average and we would say if the average

is above this number, we're good.

And of course, A-BET evaluators through

no fault of ABET, but because evaluators

like to come up with new stuff, started

saying, yeah, well, what

about the standard deviation?

And what about the modality? And it's

like, yeah, well, what about it?

And, you know, eventually it's going to

become part of ABET evaluator lore that

if this isn't done, you're going to get a

shortcoming and there's

nothing you can do about it.

So how do you protect yourself against

what I call the ABET virus that rapidly,

you know, because, you know, 13 people,

visitors on a team hear

about some great idea,

which has nothing to

do with the real thing.

And then the next year they go to 13

different programs and they

infect all those new teams.

Well, when I did this, it was and all of

a sudden it's growing

exponentially like a virus.

So how do you protect

yourself against that?

Well, you know, we're scientists.

We know that the best thing to do always

is just to look at all the data.

And so what we plot is the histogram of

all the data so we instantly can see what

the standard deviation is, what the

modality is, if there's skewness, we can

look at the tails, you know, all of that.

We just see in an instant.

And in a way, it makes the analysis of

the data very straightforward because you

just look at it and you can tell.

So what we do is for each outcome, some

way of a spreadsheet and the

spreadsheet has little boxes.

And you first you choose what department.

And when you choose the department, you

can choose which

outcome you want to look at.

Then you choose which terms and you can

check boxes and choose, you know, like

one term or one academic year or two

academic years or

three, whatever you want.

And you'll pull in all the data.

Then you'll see for that particular

outcome, here's this

curve, which is a histogram.

And there's no curve drawn.

It's just the actual data.

And it gives you an average.

We have to filter out zeros sometimes

because when students drop a class and it

still shows up in the

instructors grade books.

So the zeros usually are not meaningful.

Sometimes they are, but whatever.

And then you do the magic because, like I

said, ABIT doesn't accredit departments,

they accredit programs.

And your department might have a course

like we have our structures course MSE

350 that has a huge number of students

who weren't in our program in that course

because we offer a minor.

And one of the requirements for the minor

is our structures course.

So we have students from biomedical

engineering, from aerospace, chemical

engineering, from mechanical engineering

all over the place that

are not in our program.

So when we have another checkbox that

says which program do

you want to filter by?

And we can choose our material science

and engineering undergraduate program.

And you'll want you click that button and

you watch the numbers drop because now

you're only getting the

students from your program.

And it allows us to then prepare these

histograms so that I can go to a faculty

meeting and I do this once a year.

And I show the entire faculty how are our

students doing on outcomes one through

seven for the last academic year of just

students in our program.

And we look at the data,

we discuss it and off we go.

And that's continuous improvement.

We've used it as input if

it's not going to help us.

That's OK. At least we've considered it

as input to our program at

the level that's appropriate.

So that's what we do.

Yeah. Well, the process sounds great.

I love. I actually enjoy that annual

faculty meeting where we look at the data

is we're visually interrogating this

database of student performance in a

variety of different ways.

And it's quite interesting to see where

the program is doing particularly well

and where there are

improvements to be made.

So I'm a fan, but that's just my opinion.

Would you say that this method is working

from the ABET point of view?

Are they satisfied with the process that

you've described here?

Yeah. Well, we just had our ABET review

last fall and we have 12 programs that

are accredited and 10

of those 12 got NGR.

No general. I forgot.

NGR is perfect. No shortcomings.

And so, yeah, I think it

worked really, really well.

Were there problems? Yeah.

That's why the two programs that got

dinged for shortcomings got dinged.

And guess what? They deserve to be dinged

because they just adopted our process.

But forgot a really important part.

They forgot to document how they did

continuous improvement of their program.

They just said, we did this

and it's not going to help.

That's not OK. You must show continuous

improvement of your program.

And they forgot to do

that. Now, they had done it.

They just didn't document it.

So they had to go back.

And of course, both of these programs had

pretty major

curricular reviews and changes.

And they just didn't

document it or write about it.

So it made them complacent because it

worked too well, I would say.

So don't get complacent.

You still have to make sure that you

document and

continuously improve your program.

And this is not going to

get rid of that part at all.

Well, it sounds like the process works as

long as you follow the

process, which sounds almost

tautological. I'll have to

logic that one out for a minute.

Well, how I think about

that, let me ask this.

You've mentioned that we sort of started

this process six years

ago, seven years ago now.

What do next steps look like for our

continuous improvement of our process for

continuous improvement?

How are we going to get better at

inoculating ourselves against ABET?

So when I look at these histograms, I

think when anyone

looks at these histograms,

although we have something like 90% of

our students are above

our threshold for minimal

acceptance, you can graduate from the

University of Michigan with a 2.0 GPA.

That's like a 60% for the

way most of us grade exams.

And so that's our minimum level because

that's just being honest,

because a 2.0 can graduate you.

But when you look at the data, there's

always a few stragglers in

the tails that are below that.

And you have to ask

yourself, what about them?

And so this has actually been a good

process by looking at

these histograms over and over.

It's made me think about what about that

group? Even at our

university level, we are

very, very proud of our graduation rate

at University of Michigan.

Our five-year graduation rate is like

93%. That's the envy of the whole world.

Only a very few schools can say that. I

remember going on college

visits with my daughter and the

people standing up and saying, "We are

very proud of our five-year

graduation rate. It's 65%."

And I'm like, "What? How can you be proud

of that?" And I came

back, and when I talked to

some of the deans about that, they said,

"Yeah, we're unusual."

Even with our amazing 93%, and by the

way, some of the 7% are

accounted for by people who dropped

out of engineering but still ended up

graduating from other

colleges at our university,

there's still the stragglers of the 3% to

5% that do leave our university.

I just went to a second provost seminar

on teaching where the

provost stood up and

talked about this and said, "What can we

do about that 3%? How can we

make sure that they graduate?

That should be our focus." And why

shouldn't that be our focus in our

program? And maybe we can use

the ABET data we're collecting to help us

understand that

because guess what? We know

who those people are. We haven't accessed

that data yet because

it's a little complicated.

You still, I believe, we would need IRB

approval to actually do a

study. I'm not sure. I'm going

to have to find out the rules. But a new

tool just came available

that might let us easily do this

without hiring education researchers or

data specialists. And that tool is MAIZY.

MAIZY is U of M's generative AI bot.

That's a private

generative AI bot system.

So it means we can look at that data

without exposing student

data to the world. We can stay

FERPA compliant. MAIZY has a way to link

to a SQL database, which is what our

whole data warehouse

is built on. And so I've been talking to

those folks and we're

going to see, can we ask the

generative AI bot questions like, the

students in the tails for outcome one,

you know, can we do a longitudinal study?

Because we have several

years of data from when there are

sophomores until they graduate. Do we see

them improving? Do they

graduate? What happens to

those students? Can we talk about race

and gender? Can we talk about first

generation students,

Pell Grant students? What can we learn

from the data of the

students in our tails?

That could significantly help us improve

our program. And I'll

tell you what, if we did it,

I'm not putting this in our self-study

because I'm not doing this

for ABET. I'm doing this for

ourselves because of what I said way

back. First, we want to improve our

program for ourselves and

worry about ABET later. But I think it's

a wonderful exercise, a

wonderful way to use our data.

I should also mention that our Center for

Research on Learning and Teaching,

CRLT, which is solely focused on

improving education in Michigan, when

they saw our system,

they came and asked me, can we use your

system where we put in,

you know, our own questions?

Because they're doing curricular revision

pilots with a few

different programs or departments

because they do care about departments.

And so they're going

to be using our system

for their purposes, even though it has

nothing to do with ABET. So I think it's

really, really cool.

We have a lot of data. We've got data

from, you know, 13,000

students in this last pass.

In any given term, we're looking at 3,000

students. That's a lot

of data. And it's only

growing because I think other programs

have learned how valuable

this is. So that's what we do

for our ABET outcomes assessment. And

it's made getting

through criterion four a breeze.

I really love how this program

essentially started as, let's make sure

bridges don't fall down.

And now we're at a place with it where we

can interrogate this

really rich, robust data set

to ask questions about equity in our

programs and to say, how can

we better serve the students

who we are not currently serving well

enough? I think that's

fantastic. Well, that is all the

time that we have for today, but that was

a pretty good day one deep dive into

ABET. I think we might

have to revisit this topic in the future

now that we're asking interesting

questions about what we

can do with everything that we've learned

from these self-studies and

how we can use it exactly,

as you said, to make our own programs

better for ourselves

and for our students.

And I should mention, so, you know, I

agree, Tim. I'd like to,

you know, maybe bring on,

you know, some people from ABET. I'd like

to see if Jeff Fergus

can talk to us. Jeff Fergus

has probably been one of the most

influential people from the

materials community at ABET.

And I really think his principles and

values align closely with

mine. You know, Jeff was the

head of the EAC, the Engineering

Accreditation Commission. He's been in

charge of training for

ABET. So he's very, very well known at

ABET. And so I think it'd be

great to get his ideas because,

you know, one of the reasons we're doing

this podcast is to try to

share best practices with

our materials community. And to that end,

I hope everybody comes to

the North American Materials

Education Symposium. There's the plug.

It's a plug because on

Friday, I'm going to be doing a

free workshop on, I will show everybody

our outcomes assessment

stuff in detail. You get

to play with our Tableau system and I'm

available to give, you

know, to answer any questions,

not from a official ABET standpoint,

because I don't have any

official standing in ABET,

but just from a colleague in materials

who would love to help all of our

programs get through ABET

with a minimal amount of work and do a

maximal job on their ABET.

So come to our symposium,

get your chair to send you and we'll help

you with ABET for free.

Excellent. Well, thanks for being in the

hot seat today, Steve, and

to everyone else out there in

the world. We'll see you next time on

another episode of

Undercooled. See you later.