DAVID MALAN: Well hello, world.
My name is David Malan, and this is the last of CS50 seminars
here at Harvard in fall of 2022.
As you might know, we've been filming live the class again.
And all of this past semester is lectures
and now seminars will end up on edX and YouTube
and other platforms on January 1 as part of what will be called CS50x 2023.
In this particular seminar, I'm so excited that we're
joined by Andrew Sellergren, an alum of Harvard who actually took CS50
the very first year I taught it, for better or for worse.
I think it's fair to say that surely everything he knows
can be traced back to me, it would seem.
But that's actually not the case because if you read the description for today's
talk on applications of artificial intelligence or AI in medicine,
Andrew has been amazingly self-taught.
In fact, when he was in college here only
took CS50 but then went on nonetheless to work for Google in various roles
since.
Only just a few years ago did he dive into and teach himself
the world of machine learning, which he now
applies to real-world problems in medicine that are so incredibly
impactful that we're so glad that he's here
today to share not only that same trajectory, but a little bit
about what he now does at Google.
So without further ado, here is CS50's own and now Google's Andrew Sellergren.
ANDREW SELLERGREN: Thanks.
Thanks a lot, David.
That was a really nice introduction.
You said you would be funny, but I guess you didn't come through on that.
No I really I know he exaggerates, but I do honestly trace a lot of this back
to CS50 in particular.
Let me go ahead and start sharing, and I'll kind of tell my story as well.
OK, awesome.
Great, yeah, I'll start by welcoming everyone and, thank you for joining.
It's really exciting to be here, and it's really, really cool
to see all the faces from around the world.
That's been a real joy of mine to watch as the course grew.
So as David mentioned, I took the course for the first time.
I guess I should, I am going to date us by saying this, but it was 2007.
It was my senior year in college, and I was a pre-med.
I was getting ready to go to medical school.
And my last year at Harvard, I was taking courses
that I wouldn't normally take just for fun and CS50 was one of them.
I actually already knew David from a previous venture
and was really excited that he was going to be teaching it.
So it was everything I wanted in terms of getting
me excited about a new whole set of knowledge that I didn't have yet.
And I wouldn't say there's not like a lightning strike
there, where all of a sudden I decided I wanted to be a software engineer.
I won't tell the story that way.
But it was the beginning of a lot of lifelong learning.
And that's the advice that I can give that is probably
best for anyone is that as long as you are continuing to learn,
challenge yourself, you're excited about what you're doing,
you don't necessarily need to concern yourself
too much with what that makes you whether that
makes you a software engineer or a scientist or an artist
or what have you.
It's really just about a skill set that you're gathering.
And that's really what the start of it was with CS50.
And I don't exaggerate when I say that that was what gave me the confidence
to learn all of these things.
So CS50 is in large part about learning the fundamentals, the programming
languages that you might begin with and some
of the very foundational concepts of it.
But it empowers you to learn anything you want after that.
You can always trace back and say, well, I remember this concept
from CS50 or yeah, I got stuck at this problem set,
but then I worked my way through.
And that's been invaluable.
And to be honest, I think it continues to serve me even today.
There's just, there isn't anything that I try to limit myself
and say, oh, I can't learn that or oh, that's not
what I'm supposed to be learning now.
It's really just something that I'm always challenging myself with.
And CS50 was a huge part of that to begin with.
So the course itself has grown tremendously
since then, which is also really exciting.
It was still really cool then, but it's gotten a lot bigger.
Oops, OK.
So today, I'm here to talk to you about what I currently do,
which is AI for medical imaging artificial intelligence
for medical imaging and a tool that we just released along with some research
that we did behind it called CXR Foundation, CXR for chest x-ray.
And I'm a member of the Google Health AI team.
So I do promise I used to look like that.
David can attest that I did look like that at some point.
But I guess I, the pandemic hit pretty hard.
So I'm a software engineer.
A little bit more about my journey, too was
that after college when I decided I didn't want to go to medical school,
I wasn't really sure what I wanted to do.
I did end up teaching CS50 a bit.
I taught a few of the other courses that David
offers, really enjoyed it really, used it as an opportunity
to keep learning myself.
And then, I applied for a job at Google as a technical analyst,
which is some kind of data analyst for the ad traffic quality team.
And we were looking for invalid add activity on the AdSense and AdWords
networks.
And that was also a great opportunity because I learned a lot of skills,
and there was a big push to scale yourself.
So if you wanted to have a bigger impact,
but then you could write some scripts and some tools of your own
that would make you scale better than that was a huge benefit.
So that's where I also started learning more of my coding skills.
And that was the time when I decided, hey, I really like this.
And I want to do this more frequently, or I want this to be my core job.
So I transferred into software engineering in 2014.
I worked on the Google Fit app for a couple of years, if you've used that.
Then I, in 2017 I transferred to the Google Surveys team.
In both of those teams, I worked a lot on infrastructure,
so large scale distributed systems, getting data
to flow from here to there, kind of like plumbing.
And it was a fantastic experience, obviously
at Google because of the scale of the data that we were working with.
And it also just hardened my skills in terms of writing good code,
writing good tests, so on.
And then in 2019, I decided that I wanted
to pivot back a little bit toward health and something
that I knew was intrinsically meaningful to me.
So I reached out to the team working on some of the health
AI ventures at the time.
I joined in an infrastructure role, and I was working
on some of the infrastructure there.
I had no modeling experience, but I wanted to learn.
So again, I just kind of picked it up, picked up the mantle
and just said, OK, I'm going to learn this.
I took Andrew Ng's course on machine learning for Coursera
and then just dove in and just really started modeling extensively.
And it's been three years.
I would definitely not say I'm an expert,
but I'm really excited with what I've learned.
And the cool thing about the AI field in general
is that it's advancing so rapidly that pretty much no one can keep pace
with it.
So if you are trying to stay on the cutting edge of it,
you may be ahead of some other people who have studied a lot longer
but haven't been keeping up with the state of the art.
So that's where we find ourselves now.
And I would also be remiss if I didn't show my great team here
at least in one slide.
This-- most of these are my co-authors on the paper.
They run the gamut from pure software engineers who
work on machine learning theory to product managers to we
have radiologists in there.
We have clinicians clinicians, scientists, research scientists.
And this isn't even all of our co-authors.
These are just the ones at Google.
We also collaborated with a few different hospital systems.
So it's really a huge team effort.
And that also hopefully gives you inspiration
that you don't have to be a pure software engineer
to contribute to things like this.
And I'll get into how valuable they were in each of their different roles
and why it's important to come together in this interdisciplinary way.
Because you don't get great results without that kind of teamwork.
So you may know some of our work on Google in terms of AI.
And it was not too many years ago that Sundar put out
a missive basically saying that we would focus on AI first and a lot of things.
So Google has tremendous research experience in this area.
And we can use this AI to improve experience in core Google products
like Google Photos, Google Translate.
So you may have used photos where you can search for personal photos
with tags, or you can even search for objects.
You can search for bike or banana, and it will find it.
And then the Google Translate functionality
is something I've used myself where you can read a menu
in a different language in real time.
It's fantastic.
And we're also using AI to tackle hard problems in the physical world.
So Waymo is taking on computer vision in the realm of self-driving.
And we'll be talking a little bit about computer vision and AI,
and it's not too dissimilar from what they're doing there as well.
And with Google Health and AI, our mission
is really to show what's possible and bring these benefits of AI to everyone.
So we do conduct research that advances the state of the art in the field.
And we're applying AI to new products, new domains.
But one of our larger goals is to make sure that everyone can access that AI.
And that's what I want to talk to you about today
and why I tried to reiterate that no experience is necessary.
I think if some of the topics we talk about today may catch you off guard,
then you can always go back, study a little bit, come back to this.
But the goal is to bring that to everyone both in terms of resources
and in terms of knowledge.
So a few of the examples here, you see there's
some work we're doing on device with Coral.
There's the brain mapping.
There's colonoscopies and prediction of sleep patterns.
So all of these things are in our wheelhouse.
OK, so before we get started, I know that a lot of you
also don't have experience with artificial intelligence
or with ML, machine learning in general.
So I wanted to give a very quick intro to neural networks, which
is the type of machine learning and artificial intelligence
we'll be talking about today.
However, I want to give a big shout out to Brian Yu
and to point, you all to that course.
Because I went through a little bit of the content so far, and it's fantastic.
I'm going to be borrowing a few of his slides
from his lecture on neural networks.
But it's a great way to bootstrap yourself in terms of learning what ML
and what AI can do.
So but let's just talk about neural networks for a minute.
Here, we have an illustration of a neuron.
So these are the fundamental building block of your brain.
And these neurons are connected to each other,
and they receive electrical signals from other neurons.
So there's two important parts there, basically.
One is that they communicate with each other.
So there's billions and billions in your brain.
They process input signals and then they have this notion of being activated.
So that in the brain, that corresponds to an electrical signal.
But we will talk about it in the context of an artificial neural network
in a second.
So here you have a two neurons that are communicating via the synapse.
And that's some kind of electrical signal that's going between them.
And together, the two of them plus billions of others
may combine to do much more complex calculations.
So that's an oversimplification, but it's a useful one.
So when we talk about artificial neural networks,
we're talking about a mathematical model for learning
that's inspired by these biological neural networks.
So you can think of it as just one large math equation.
It takes inputs on one end and gives outputs on the other.
And this is based on the structure and the parameters of the network.
So these neurons, let me go forward for a second.
This neuron, this biological neuron you can report it placed with just this.
We'll have a circle here, a single building
block for this large neural network.
And if we want to do more complex things,
we can connect them together and connect many, many, many, many, many together,
right?
So the-- let's go back for a second to here.
So the other aspect of this neural network that is important to note
and will be good for our sake is that they can learn.
So we start them with basically completely random,
they have no knowledge.
But through a process, which I think I can point,
you again to Brian Yu's course through a process called back propagation.
We give it the outputs.
And then we let it look back on all of its parameters and say, OK, how can
I tweak these so that I predict those outputs a little bit better?
And that is the magic that makes these neural networks work.
These parameters are basically, you hear them called weights.
But they essentially are coefficients in a very large math equation.
So if you were to write that out end to end,
you would be able to put in your inputs.
And then you would get your outputs just like you do in a equation like y
equals mx plus b, except it's many, many millions or billions of these neurons.
So what do we use this for?
So Brian also put together these nice slides about a classification problem.
So a lot of the problems that we face in machine learning
and in health for machine learning are classification problems.
That just means that we're trying to say something is of one class or another,
in this case, red or blue.
In health, it might be diseases present or disease is not present.
So with this distribution of data that you see here,
it's fairly obvious that we could design a classifier just by drawing
a line between the red and the blue.
So in our neural network, then it would be very simple.
It would just be a line.
And it's a nice visual representation of how we can separate that data.
And so then you say anything below the line is blue.
Anything above the line is red, and we go from there.
What about a more complex set of data like this?
So obviously, we as humans can very quickly
draw a circle around the red in there.
That's actually a little bit more complicated
for writing an equation, a math equation for this.
One thing that I'll mention here that you
may hear in other lectures on this topic is that these neural networks
are non-linear.
So that's a huge advantage to them to be able to learn not just
linear relationships but have these so-called activation
functions, which are linear.
And they allow it to learn things like this,
which are a little bit more complicated in their nature.
So we'll put a pin in this for now because I will actually show you
much more complex data that can be even reduced
to something that looks like this.
But very, very briefly and very broadly speaking,
our problem is no different than this.
We're trying to separate the red dots out from the blue dots.
You may have a more complex, maybe there's more than two colors.
There's green there's yellow, et cetera.
But it's fundamentally the same problem.
I guess actually this would be a good time to pause for questions
if there are any, or.
I should just mention that we have plenty of time,
and I will be going fairly quickly.
This will be a big survey through not just ML and AI
but medical imaging as well.
So please don't hesitate to stop me.
And mostly, I want to show you what's capable, what we can do with this
and to get you excited about the field
So let's now talk about AI development for medical imaging.
And as I mentioned earlier, the big focus of Google and Google Health AI
is to reduce barriers to AI development.
So for us to have impact, we want more people to be involved.
We see ourselves as an enabler.
That means both as I said, in terms of resources as well as knowledge
and skills, so we're very well aware that Google has resources for modeling
that most other places don't.
That comes in the form of accelerators for the actual modeling as well
as data.
And we want to share that with the world.
We want to share the benefits of that with the world,
and similarly the knowledge and the skill set.
So we're very lucky to have the best in the world that we can work with.
But we want to make this a little bit more
accessible to those who are just getting started.
So today, we're going to talk the use case or the case study
that I'm going to look at first is chest x-rays.
And that's my main focus.
But it's really just an example of what could be possible.
And this could apply to many different modalities like CT imaging, MRI,
ultrasound, pathology slides, even just natural images
from a camera for things like your skin conditions.
So some of the principles or most of the principles
are pretty applicable through everything.
So let's see.
Oh, I do see some questions.
DAVID MALAN: Andrew, I've been jotting down some questions as we go.
I can leave them in when it finally.
ANDREW SELLERGREN: Yeah, I think it's a good time to pause anyway.
Does this-- Fred says does this mean that in the future,
radiologists will not be human?
I don't know if I would go that far.
I think, and this will be clear from some of how
I describe our collaborations later.
But I think the future is really a combination of the two.
We get our best results when we work collaboratively with radiologists.
They lend their knowledge to us, and then we lend extra tools to them.
And we really want to understand the problem from their perspective.
And from there, we may design a tool that works well with their workflow.
But we are not intending to replace it.
We don't think that that's the best path forward at the moment.
But we do want to help save them time and help bridge
that gap for a short supply.
So let's see.
Yeah, I think we'll have some as far as resources,
we'll definitely have some links that I can provide you
and we can send out that would get you started in this field of machine
learning and in health AI.
With how much precision can medical AI identify disease, e.g.
Cancer cells?
It's a good question.
I think it's getting more and more precise every day.
So and it's maybe scaling even exponentially.
So there-- we have a team for example that
works our genomics or genetics team that works
on deep variant and deep consensus.
And those are things that enable you to do virtually real time sequencing
of either disease or the person.
And that helps with a precision treatment
or the availability of a treatment.
The precision medicine on the scale of say, pathology slides or radiology
images, those are getting more and more real time.
Obviously, they're getting larger.
But then they're still able to run particularly fast.
So it's an exciting time to see how far we can push the field.
Instead of predictions, what type of AI would
run for decision making in health?
That's a really good question.
I don't know.
My field is not necessarily in the NLP or the natural language
processing or conversational AI, but obviously that field
is advancing rapidly as well.
And I think the ability to add reasoning to some of these AI tools
is super compelling.
And I think it's going to get better.
I think there are some data sets like medical question answering data sets
where you test its ability to reason.
And again, I think that's still where humans shine in large part,
but it's getting better every day.
Let's see.
Are there models not based in biological brains?
Yes.
I mean, I think I would actually argue that if you want to dig deep into it
and go down a bit of a rabbit hole, the models don't really
perfectly mimic biological brains.
I think we learn more about how they complement each other
or how they mimic each other every day.
But it's a bit of an oversimplification.
I would also argue that your brain is far, far, far, far more
powerful than the neural networks that we're training.
So that's a pretty cool thing to learn when you maybe
come into this field thinking the opposite that oh, machines
are much better.
They may be better at certain tasks, but we are much better generalists.
And there's no better example of that than my daughter
who watching her learn is amazing.
Because she may not have even seen something before.
But she already has a knowledge of how to interact with it.
And that's a pretty cool thing.
Did Google perform research on EEG, EKG, EMG?
I think we do have some research there.
I'm not sure of the current published status of it.
But yeah.
So let's see.
DAVID MALAN: Andrew, I can take notes on some of the questions
and weave them back in if some of these you'll touch on anyway.
ANDREW SELLERGREN: OK, yeah that's--
wow there's, a lot of--
yeah, we will do our best.
Let's I guess we'll move forward and maybe you
can weave some more in as we go.
DAVID MALAN: Perfect.
ANDREW SELLERGREN: OK, so going back to AI for medical imaging
and as I mentioned, we're going to use chest x-ray here as a case study.
But a lot of it is going to apply to other modalities,
all kinds of medical imaging.
So why chest x-ray?
Well, chest x-rays are ubiquitous.
They're highly accessible, highly available there's about a billion
taken around the world every year.
And they're very inexpensive.
They're also non-invasive meaning they're not requiring
surgery or something like that.
They have a fairly low radiation dose I believe as well.
So it's a great modality for that.
It's also used for a very long tail of rare conditions.
So we'll look today at five or 10 that are fairly common
and that are kind of standard ones that you may see on a lot of chest x-rays.
But there's a huge long tail.
There's a lot of rare conditions which makes chest x-ray very useful,
but it's also very difficult then to interpret them.
And that's where the radiologists shine on the human side of things.
But that high quality interpretation is difficult to do and to train
for, doubly so for training a model to do that.
So that's where this work that we'll talk about will shine.
So there's also a short supply of radiologists.
There are definitely areas of the world where there's far fewer
available than there are images taken.
And there's also variability between the experts and the sites.
So you may have different imaging equipment.
You may have different interpretations of the same image.
So these are all challenges.
And then as I mentioned with Google and where
Google's forte is in building these very, very large models.
So they require very large curated data sets.
So for our purposes for this model we're talking about,
it was about a million chest x-rays, which
even on the scale of machine learning is pretty small.
A lot of the natural language processing ML
that you see today may have been trained on multi-billion or web scale data
sets.
Whereas a million chest x-rays is a very large medical data
set but it's a small data set for natural images.
But for anyone to have a million chest x-rays is pretty difficult.
And it also requires extensive fine tuning.
So some of the very specific models that we developed like our tuberculosis
model, they require six 12, 24 months of software engineering time.
And that's a whole team of people working on it, trying different things,
going back to the drawing board.
We call this fine tuning.
And it's not something that everyone can replicate.
So this is where again, once we've done this in one use case,
how can we help generalize it and let it benefit
everyone and a bunch of other tasks?
So let's-- we can play a little game here if people want to be interactive.
So how challenging is this task, and not just for machines, but for humans?
I wanted to show a few examples of chest x-rays.
You can Google around for things like these.
But here here's one.
Can anyone spot the abnormality here?
I see one person saying pneumonia.
Anyone else have any guesses?
Right chest smaller, left upper lobe, actually,
yeah, I would say left upper lobe is pretty close, so I'll give that credit.
So there's a nodule here, which you can see is very, very difficult to see.
A nodule is a small mass of tissue in the lung.
In the case in this case, it's less than three centimeters.
And it can be benign or malignant.
So it may or may not be even noted on a radiology report.
It could be something that they decide to biopsy and figure out
if it's benign or malignant.
But as you can see, it's not super easy to catch.
And so training an AI classifier to do it is not trivial.
OK, how about this one?
Anyone can spot the abnormality?
Oh, wow.
Salt has got it, pneumothorax.
Yes, so on the right here you have what looks in some ways like more normal
it's a Black space on the chest x-ray.
But this is actually a collapsed lung.
So air has leaked into the space between the lung and the chest wall.
And this would require intubation.
And it's a very serious issue.
I'll pause here just to note one thing about these chest x-rays.
So when we're talking here, we're talking about frontal chest x-rays.
There's two different ways to take those.
There's AP, anteroposterior and PA, posteroanterior.
So this is whether the x-ray machine.
Comes from above or below and there's some interesting implications
for that when patients are sicker, they tend
to have one type because they're laying down,
and they may not be able to have an x-ray standing up.
So they-- this is actually something that confounds our ability to classify.
It may just learn the type, the orientation of the image as opposed
to the actual diagnosis.
Similarly here with the pneumothorax, this one is untreated so far.
So there's no intubation whereas pneumothorax would normally require
intubation and immediate treatment.
So then when you take the chest x-ray, anyone who has a pneumothorax
is going to already have some lines and tubes, which
make it very easy for the AI to learn well as long as there's lines and tubes
then I know it's a pneumothorax.
So that's a big problem.
I think I saw a hand raised, but.
DAVID MALAN: There is a question in the chat.
Amelia is asking Andrew is this the same person, and what about their age group?
ANDREW SELLERGREN: No, this is not the same person as the nodule one,
no it's not.
You can see there's a pretty big difference in their anatomy.
I have the citation here below.
So you should be able to look that up.
I'm not sure about the age group of them.
OK, so this one I'm not even going to ask
you to try because it's very, very difficult,
but it was my example of a very, very difficult case that
may not be easy to catch.
So down there in the bottom right of our screen but on their left
is a rib fracture.
So this is the type of thing that can very easily be missed,
and it's also very hard for AI to pick up on.
So it exemplifies some of the challenges we face here.
Here's a question I can address really quickly.
Would you recommend any specific deep learning structure to the task?
Is Vision Transformer a good choice?
I'll try not to go too much down the rabbit hole
here because it's a very interesting time.
The type of model that we've trained here
is something called a convolutional neural network.
So again I'll point, you to Brian's amazing course,
which talks about convolution.
Convolutions are you can think of as a filter for the image.
So it will progress through the image and in some ways sort of compress
it to that end.
And this has been the standard since about 2012 for vision
anyway for a number of reasons.
Convolutional neural networks have built in what
they call translation invariance.
So it doesn't matter where in the image that it's happening, it can pick it up.
And that's a huge advantage for something like vision tasks.
That being said, so they've been this the go to for 10 or more years
at this point for.
So only recently in the last two years, vision transformers have emerged.
Transformers is another type of architecture
that has been used heavily in natural language processing.
And this just works in a little bit of a different way.
It doesn't have some of the same what we call inductive biases
as the convolutional neural networks.
But it also has advantages like its ability
to using a method called attention, it can basically look at the whole image
And it correlates every block of the image with every other block
of the image, which is a big advantage.
However, they require a lot more data in the case of vision
in order to kind of bootstrap themselves and get used to the task.
So they've definitely proven very performant
on some of the natural language, I'm sorry, natural image tasks.
They are starting to prove even better for the medical image tasks.
I've tried them on some of the same tasks that I'll show here,
and I got slightly worse results than I did
for my convolutional neural networks.
That doesn't mean they are worse or that they always will be that way.
It could just be a matter of I didn't tune them properly.
But that remains to be seen.
So is there a nodule in the right lobe here?
To be honest, I don't know I don't think so.
If you're talking about this, I think that's maybe an artifact of the image,
but I could be wrong.
I'm also not a radiologist I'm doing my best impression here.
But so this last one, we'll see if we can spot the abnormality here, anyone?
Yeah, I'll go ahead and give that I hope I'm saying Myla or Mila, pneumonia.
Yeah, so this one is COVID-19.
So they refer to it technically as peripheral and lower lobe predominant
rounded airspace opacities.
But for the most part, that will amount to pneumonia
in the bottom parts of these lungs.
So this is an important motivator for some of the tasks
that we're going to look at for our model here.
And it's just a reminder of what chest X-rays can do.
So as I mentioned earlier, our mission is
really to enable others to train better custom chest x-ray models.
We know that other outfits don't have the same data, the same setup,
the same computational power.
So our mission is to advance science and deliver impact.
And we define impact as being the end result, actual clinical effects.
In some-- in most cases, those are going to be taken care of
or affected by others, not by Google directly.
And that's what we're looking to do here with our ability to train these models.
So this is a three-fold problem.
We want to decrease the training time for these models.
We want to improve their label efficiency.
That means we just want to be able to use fewer images
to do the same tasks so as opposed to having the hundreds of thousands
or millions of images, can we do it with 1,000, 100, 10 images?
Reducing model complexity, so the convolutional neural networks
that we trained here are on the order of some of them as large as 500 million
or even a billion parameters, a billion weights.
And those are massive.
We're also talking about huge high resolution images.
Original chest x-rays are somewhere between 3,000 by 4,000 pixels.
A lot of the literature will downsample them all the way to 224 by 224.
With our resources, we're able to train on images that are size 1,024 by 1,024.
So that's another big advantage.
So what is our method here in order to try to affect this third party
impact that I mentioned?
What we want to do is just basically do as much as possible on Google's side
as we can before we hand off something resembling a model
or something resembling the training to the third party
to the person who might actually be doing the end training.
So there's a fair amount of text on here,
but let's just let's just break it down fairly simply.
So going back here when I mentioned--
So this is a rough diagram of a neural network.
And you can see we're probably looking at moving left to right,
which is through these neurons.
But you can see let's say you were to turn it on its side,
it resembles something like a pyramid.
The way that we're envisioning this here and a useful analogy
here is to think of it as a pyramid.
So on the bottom, we have our chest x-ray image,
which we feed into the model as pixels, just as numerical values.
And we slowly hone in on our classification.
So the actual output of the model is going to be just one number generally
that will be a number probably between zero and one
that represents our prediction.
So usually, it would be something like if it's closer to one,
then we are predicting that the disease is there or the condition is there.
And if it's zero, it's not there.
So here within our pyramid, these layers in here are artificial neurons.
So we have many, many, many, many layers of them
and again millions billions of them in there.
Somewhere in there toward the top of it, we
have a layer that is not quite the end result,
but it has a lot of what we call features.
So these are still just floating point numbers.
And we can't exactly point to one thing that they represent.
But they are useful for the diagnosis.
So they're not quite as interpretable as say oh, this is a left lung,
and this is a right lung.
But you can think of it as something resembling that.
And we're going to call this the embedding.
This is in our case, an embedding is really just a string of numbers.
It's maybe 1,000 1,300 2,000 numbers in a row.
And these are our features for the chest x-ray.
So another useful way of thinking about it is this
is just a very fancy image compression algorithm.
So we put in a chest x-ray image, which is 3,000 by 4,000 pixels.
And what we got out of it was 1,000 numbers.
So just like you might compress an audio file or a video file on your computer,
this is sort of a compression of that in a way
that we're trying to retain as much of the useful information as possible.
So let's take that definition of embedding and go forward from here.
OK, so what was our technique here?
It's pretty much as simple as it sounds.
So I don't want to complicate this and make
it sound like it was too, too fancy.
What we have is what we call a CXR pre-trained network,
a chest x-ray pre-trained network.
So in a typical setup for machine learning,
you often do what's called transfer learning, which
means that you take a model that was trained on one task,
and you try to train it on another.
You fine tune it on another task by exposing it to new data.
And generally, that is an advantage that tends to get better results.
The story, the truth of it for medical imaging
is that may not be all that useful.
So the state of the art in all kinds of machine learning and computer
vision out there is usually done on a data set
called ImageNet, which is I think 14 million natural images of 1,000
different classes.
And everyone competes to get the best score on this.
Then what we do on in medical imaging is we usually take
a model trained on those, and then we expose it to medical images,
and we get good results.
But the question is, do we get better results
than if we just started from nothing?
And I think the jury is still out on that, actually.
Because it may speed up the time it takes for us to train it.
But I'm not sure that the end result is a better model.
So what we're proposing here is basically
kind of adding in another step where we train things
on more specific data for your task.
So we have the ability to train it on a million chest x-rays.
So then when you take your model, it will already
know what a chest x-ray is.
It will have some way of reasoning about that.
So then on the right side, we're going to take
this pre-trained network, which we're using to generate these embeddings.
And then we can do a bunch of different things.
So the typical setup is on the bottom right.
There's a task specific network, and we can fine tune the whole network.
So this is usually going to get the best performance,
but there's a few concerns here in our case.
One is that we trained on private data from a few different data sets.
And we don't want to expose that data in any way to the outside world.
So releasing the model may be a risk in that sense.
So what can we do instead?
Well, we are able actually to just expose the model as an API.
So you can call this model with your images, and you get back embeddings.
And then from there in these strategy 1 and 2,
you can see you can actually train a very small model on that
and get decent results in any case.
So the way we were visualizing this is, again hopefully very instructive.
If this is the pyramid, we're sort of just chopping off
the head of the pyramid.
And then we're asking you to fill to fill
in the space so as opposed to having to build the whole pyramid yourself.
So this is where this really cool animation comes from.
On the left, you can see the traditional approach
is to build a pyramid for each different task.
So you have to start all the way from scratch or from ImageNet,
and it's going to take time.
On the right, we gave you a pyramid with the head chopped off.
And you can just add your own different heads that represent different tasks.
And what we found is that this approach, you
get really good results even without having started from the very beginning.
So I'll very quickly talk about this is more highly technical side of things,
but this is the type of learning that we did for this.
So when you start your machine learning education,
you will probably learn a lot about cross entropy
loss, which is kind of the typical task or objective for classification
problems.
But there's a lot of other options out there.
Some of them are contrastive learning.
And this is just what it sounds like.
So you're taking to two images, and you're comparing them to each other.
And then you're giving the model some information
about what those two images represent.
So in this case, or in the images we have
here, what we're doing is we're telling the model
that these two images are both of a dog.
And so we want to learn some representation,
some embedding of this image that brings those two things together.
And then we want to push all the cats away.
So this is the same for our medical imaging task.
What we're going to do is we're going to say that these two images are abnormal.
So that means that they as you saw in the previous slides,
maybe they have a pneumothorax, maybe they have a nodule,
maybe they have a fracture, they're abnormal.
So put them together in our embedding space,
and then push away any ones that are normal.
So anything where a chest x-ray doesn't have anything that requires
follow up or is of note.
So the nice thing about this is that most of our radiology classification
tasks can be reformulated in this way.
So you can imagine that if you could identify a chest x-ray as normal,
then we're kind of done with it.
The decision tree is basically you move on.
But the abnormals is where we want to start separating things out.
What's interesting about the loss, and this
is a perfect example of why I love this field is
because even doing something like this, we
get separation of the abnormal images.
They get separated from each other to some extent.
So the fractures get separated from the pneumothorax.
And I'm not really sure why.
It's something that we want to actively consider.
But here's a visualization of that.
So if you remember, I pointed earlier to this red and blue
and how prescient that would be for our classification problem.
So here, we're using a visualization technique
called T distributed stochastic neighbor embedding, or TC.
And this is just a very fancy name for a way of taking our long vector,
are array of 1,000 floats.
And then we're compressing it, or we're projecting it down onto two numbers,
two axes, x and a y so that we can visualize.
It so it's definitely an oversimplification,
but it helps us as humans reason about it a little bit, right?
So what we're seeing here is that on the left,
if we take a model that was trained on ImageNet, and we do this technique,
you can see that there's not really very good separation.
It would be very hard for you as a human to draw a circle around just
the blue dots that excludes.
The red dots and just the same for a machine,
it would be very hard to do that.
On the right, it's a little bit easier.
It's not perfect, but you can see you could
imagine drawing a little circle around the blue,
and that's a separation from the red.
And this on the right is our chest x-ray network.
So you can see that even without having done any work,
we have a pretty good separation.
We already have a pretty good way of classifying this airspace opacity
class.
So that's a pretty cool result.
And I'll say here very quickly just to talk
a little bit about the metrics, what we're visualizing here
or what we're graphing here is something called AUC, area under the curve so
actually, more on that in a minute.
I will say that what we're showing here is
that we're able to get basically the same performance as some
of these pre-trained networks that are fine tuned on the task at hand.
Our model with a small layer trained on top is the red.
So you can see it's best in show up to an order of magnitude
around 10,000 images.
So only there, if you have more than 10,000 images
do you start to get an advantage from training on all of them
and training a large network on them.
Otherwise, you may be better off doing our simple approach, which
is much faster.
And in the case of two really vital diagnoses
here, tuberculosis and COVID-19, we get better results
when we use our small models, and that's because we have so few images.
So I'll bring up this slide so we can talk about some of the metrics
as I mentioned.
So here, we have AUC is on the left.
And that's area under the curve.
So what is that curve?
Here is that curve.
It's called the receiver operating characteristic curve.
So it represents the trade between finding
true positives and false positives.
So once you have your number in a classification model
that's between zero and one, you have to decide
what is your so-called operating point.
So that's the threshold at which you say, OK above this is positive,
below this is negative.
And that's a whole can of worms to try to figure out your operating point.
You might think OK, let's just start with 0.5 well most of the time,
it's not perfectly well distributed.
So maybe 0.8 or 0.7.
So you have a choice to make as to where to pick that operating point
to kind of get the best trade-offs.
So in some cases, you may want to have--
you may want to optimize for more true positives or a higher
true positive rate and then a false positive rate or a lower
false positive rate.
So this graph, this receiver operating characteristic curve just
represents the trade-off between these two.
So everywhere along this line is a possible operating point
that you can pick.
And each one is going to have, again different trade-offs.
So when we talk about what scores we're giving our model,
where we actually just go ahead and add up all the area under this curve.
So that the better this curve is, the more up into the left it is, right?
So this AUC here is on the order of 0.9, 0.92, 0.95.
You can see most of the graph is underneath the curve.
And that's a good thing.
And also here, we have graphed a few different radiologists.
So this is from having them review the same images,
we can show that our performance is at least as good as them.
Because they fall somewhere on the curve or below the curve generally.
But what's fascinating about this is that here, we've
trained on only 100 images.
We trained a very simple model that took five minutes.
And the train and the test sets are from different countries.
So another common practice in machine learning is to train on certain images,
and then you hold out some images so that you
can be sure that you are testing on something sort of fair.
So in this case, our training images came from the US,
and our test images came from China.
So we can be sure that it does pretty well
even if we're using different demographics, different imaging
equipment.
So that's a really good result. And then here, we have even more ROC curves.
And we trained a model on COVID-19 severity.
So our ability to predict whether a COVID-19 patient would need to be
ventilated or go into the ICU.
Again, the interesting thing here is we only have 500 images.
So I'll pause here again to talk quickly about the sum of how
this research came to be.
In March of 2020, we were looking at developing an abnormal normal model,
so as I mentioned, this ability to separate abnormal chest
x-rays from normal chest x-rays, and then when the pandemic hit,
we wondered could we do something with this model
to help with the scarcity of testing that was available at that time.
We talked to our partners at Northwestern,
and we found that they had about 500 images.
And it was March of 2020 so it was very new.
They didn't have a lot of images.
So what can we do with that few images?
Well that's where we started to look at these very low label,
low number of images case studies.
So the result that we showed was that yes, you can actually predict
some of this stuff with chest x-ray.
But I also want to highlight that this conversation, we did not put anything
out there because we, having spoken to our radiologists, our clinicians
on staff the determination was basically that this is not
a part of a normal clinical workflow.
So even though you could diagnose COVID-19 from a chest x-ray,
it would be a strange thing to have people
come in who had symptoms of COVID and then take a chest x-ray of them
and then diagnose them.
That's not typically what would happen.
And so it wasn't super useful off the bat.
However, it was a great way of demonstrating
that we are able to train these models that can very quickly adapt.
And that continues to be true even today because the landscape of COVID-19
is always changing more so than anything we've ever seen before.
So now we have patient populations that are
changing in terms of the vaccine rates, their demographics for who's
being affected, the different variants.
So all of these things are things that are going to challenge our model,
but if we're able to quickly adapt with a small number of images like this,
then we're well suited.
So I'm almost to the good stuff.
I will say one other thing just to set up the--
so another type of-- so I mentioned earlier this idea
of contrastive losses, and we're taking two images and bringing them together.
Another cool thing that you can do is take
not two images, but one image and one piece of text and bring them together.
And you tell the model, these are similar, even though they're different.
One of them's an image, one of them's text.
You can bring them together in this embedding space.
So this was developed originally for the clip paper,
but it's now been applied to chest x-rays with the convert and check zero.
And we'll just hold that thought for now because it will become important
in a minute.
So what can we do with these embeddings?
We can do a few things, and I'm going to do-- help you do all of these right
now.
So let's get to it.
A zero-shot image classifier, what does that mean?
Well zero-shot means that the model has not seen anything
or has not been given a label of something before.
So you could imagine training something on Fracture,
and then you ask it where is the pneumothorax?
That would be an example of a zero-shot?
And it's a very difficult problem, obviously because machine learning
models benefit from having seen labeled images beforehand.
So what if you don't have that?
Well, this is what we'll look at here.
Another case is text image retrieval.
So what if you wanted to look up images based on the text that you input?
So this is what I'm demonstrating here on the right and what we'll build.
What if you wanted to build a radiology report generator?
We can do that, too.
And I want to pause here also to really thank
the folks that put together these two large open source data sets MIMIC-CXR,
it comes out of an MIT.
CheXpert comes out of Stanford, and they're really, really useful
for developing these kinds of models.
So let me go ahead and switch to Colab so we
can get some cool demos going here.
I have taken more time than I expected.
David, do you have any questions you want to feed in right now?
DAVID MALAN: No, I think folks are probably
about to really enjoy the hands-on part.
So we should forge ahead.
ANDREW SELLERGREN: Ok, sounds good.
So if you're not familiar, this is Colab.
It's based on Jupiter, which is also available.
And it's basically just a playground for you to run Python code locally
or you can have it connected to a runtime that's hosted in the cloud.
So I will make some version of this code available.
I need to clean it up because it's embarrassing right now.
But it will serve its purposes now.
So here, there's just some instructions for setting this up to run locally.
And I will I've also prefab some of this.
So here we're just going to go ahead and import a bunch of libraries
that we need, a few of them are TensorFlow, SK Learn, Pandas.
A lot of them are very common data science libraries.
CSR foundation is the name of our tool that we released
and is a library on GitHub and on PIP.
PIP is our library dependency management system.
So I've already installed this.
It also includes a bunch-- has a bunch of dependencies.
So it takes a little longer to install.
So now, I'm connected to my local machine here.
And I'm just going to go into this folder for MIMIC-CXR.
This step unfortunately takes a little bit too long.
But what I'm doing here is just reading in the radiology reports.
Because we're going to use those as inputs to our model.
And here, we're also going to take the radiology reports,
and we're going to just clip into a section of them called
the impressions section.
So that's usually some kind of a summary that the radiologist
has given that is less detailed but more focused
and will be good for our training.
So here, I've also just again, done some more pre-processing of this.
But I'll show you what this looks like in a minute.
OK, so let's examine our embeddings.
So as I mentioned, these are just arrays of numbers.
So to prove that point, here's one of them.
So these range in value.
I don't know that they have a min and a max.
You can normalize them and so on, but they could be negative.
They can be positive.
There's about 1,000 of them in our particular model.
And believe it or not, they contain all of the information that we need.
So here, I have my labels file.
So for every one of my images, I have an identifier.
I want to load up the radiology report that corresponds to this.
So I've again, prefabed this.
But it's really a lot of this is working in Pandas, which is a data science
tool that is fantastic.
And it gives you back these things called
data frames, which is just a table.
It has rows and columns like an Excel spreadsheet.
And it allows you to have things associated with each particular image.
So here, I have our DICOM ID.
DICOM is a type of is a medical image format.
And then I have a bunch of these labels representing
different conditions that are diagnosed on chest x-ray, atelectasis,
cardiomegaly.
Cardiomegaly is an enlarged heart.
A lot of these other ones, edema consolidation,
atelectasis, they pertain to lung conditions.
There's fracture.
So this is the PA versus AP as I mentioned,
is the orientation of the person and the imaging equipment.
And finally over here, so I'll make this a little bit bigger.
Well, actually it's probably easier to just look at the impression section.
So the impression section is again, our boiled down version
of the radiology report.
And you can see, actually one big, let me go ahead and make more.
I'm just going to ask for more examples here.
So you can see this is a good example of one of the problems
that we'll have with our data, which is that it's very templated.
So when a radiology report is normal, they'll
often say something like normal study or no acute intrathoracic process, which
means nothing's going on in the chest and no acute cardiopulmonary process,
nothing's going on with the heart and the lungs.
The other ones that are for abnormals are going to be a lot more involved.
And we're going to, let's see if I can, I have to do a little bit of whoops.
OK this is just to get a little bit wider view of each of them.
So you can see there's a lot of text here.
And we're going to try to get our model to understand most of this text.
So this is not too interesting again, just setting up our frame a little bit.
I'm going to skip this part, skip this part.
So the first thing that we'll do is what I
mentioned earlier, which is we can take our embeddings,
and we can train a very small model.
So you can imagine actually, let me go ahead here's
our GitHub, which I will have links to, and you'll
need to if you want to use this tool, you'll need
to just fill out a very quick form.
But it's just kind of getting to know who you are, nothing too heavyweight.
So if we look at our model here, OK, so the important part is actually here.
So we're putting together just two of these layers that
are called dense layers.
So that's this line of those circles that I mentioned, those neurons.
And there's just two of them.
So that's it.
That's how complicated our model is.
And with Caris, which is a library that works on top of TensorFlow,
it's as simple as just saying give me one layer, give me another layer.
And then I put my model together.
So the rest of this stuff is a little bit.
You don't need to know too much about it right now.
It has to do with things like your learning rate.
But I just wanted to point that out to show you how simple the model is.
OK, so now, I'm going to start this training so that it will oh oh, OK, I'm
going to start at training so that it can run while I'm talking.
So what I've done is I've given it my labels, which are
for the condition called cardiomegaly.
And they're just one for positive, zero for negative.
These are just all of our hyper parameters.
So batch size, how many images are we going to look at once?
512.
How many epoch are we going to train?
20, we're going to go through the whole data set 20 times.
So what's nice about this, you'll see the first epoch,
the first pass through all the data is going to be pretty slow.
But then I'm going to cache the data set, and it will fly.
So as it's training here, you can notice the training loss is going down,
which is good.
The AUC, which is going up it's getting up to 0.81.
This is all for the training set.
So what we mostly care about is our validation set.
We want to know how does this do on data that it hasn't seen yet.
So we'll leave that off for a minute.
I wanted to ask David, are we able to go a little over time if I.
DAVID MALAN: Think we're OK.
Folks don't mind sticking around with us.
Happy to finish up.
ANDREW SELLERGREN: Yeah, sorry.
I underestimated how long this would go.
But I definitely, I'm able to stick around.
So if people have extra questions and so on.
DAVID MALAN: Absolutely, we can leave some
of the questions that were asked in the chat for the very end.
So folks can join or leave as needed.
ANDREW SELLERGREN: OK, sounds good.
OK, let's see.
Other the things I wanted to call out here, let me,
oh I don't have any so two of the things that I didn't--
that I glossed over here were your learning rate and your optimizer.
So these are the things broadly speaking,
when you're doing machine learning, and it's doing back
propagation you're taking steps toward your end result.
And these just determine how big those steps are.
And the learning rate particular defines how big that step is.
You can do things like a learning rate function
where it will get smaller over time.
So that you're taking smaller steps sort of getting closer to your result.
OK, here we go.
So a couple of interesting things here before we move on,
you can notice that over each of the epochs
are this is our training that you see.
And it's getting much, much better, getting almost close to one,
which is the best possible result.
Our training loss here is constantly decreasing.
But our validation loss, so it starts off decreasing,
and then it basically plateaus.
It even starts to increase a little bit at the end.
And our AUC for the validation set stays about the same.
But it's still good, it's 0.86.
So this is an example of overfitting.
When your model, even one that's this small,
but the embedding is being as feature rich as they are,
it learns too well in some ways.
It's picking up on things that don't translate well to other data sets.
So we call this overfitting.
And there's a bunch of different ways to handle it.
A lot of techniques are called regularization
that try to address that.
And that's a lot of what we do as machine learning specialists.
So here, we can just very quickly look at model summary.
We'll give you a quick breakdown of the whole network.
So again, it's very, very small.
It has two dense layers of 512 and 256 neurons.
And the total number of parameters is less than a million.
So that's a pretty small network.
So let's do some more interesting stuff.
So now, as I mentioned before, we often take one data set, we train on it.
We validate on another.
We test on another one to see how it's doing.
So the MIMIC data set comes from MIT.
I can't remember which hospital systems, but it's mostly ICU patients,
so intensive care patients.
CheXpert is a data set that comes from Stanford,
so all the way across the country, different imaging equipment,
different distribution of patients.
So we're going to actually test our model on CheXpert
without having trained on it at all.
We want to see how well it does when it hasn't seen the data yet.
So I'm going to go ahead and load up our train
and our validation sets for CheXpert.
And this is another labels file that I've created.
You can see these numbers correspond to the various conditions,
cardiomegaly among them.
So now, I'm going to gloss over this a little bit
because I need to start the training and then we can come back.
OK, this is even messier than I remember.
So how are we going to feed our radiology report into the model?
Well, we need to convert it to numbers just like the images numbers.
So that's all that this stuff is doing here.
Here, I'm just setting up a little bit of a way for us
to use CheXpert as the evaluation set.
So now, our model is a little bit slightly more complicated,
but it's not too much so.
So we have a couple of layers here that are going to get added onto our image.
And then those a couple of different layers
are going to get added onto our text.
And then we're going to take the outputs of those two and compare them.
And that's going to be our task.
So here, in order to get numbers from our radiology reports,
we're going to pass them through this preprocessor.
So we're going to tokenize them in the language of natural language
processing.
So basically, that's just a dictionary of OK, this word is the number one.
This word is the number two and so on.
In fact they're actually word pieces.
So they're syllables or a few one part of the word.
And this is all a bunch of fancy stuff that just says, OK do that assignment.
Now, I mentioned the clip model.
This is where the magic is happening.
I'm just going to, let's gloss over that part.
All this is doing is passing our image through one side of it,
our text through another.
And again, this loss function here is a cosine similarity.
So it's going to say how close are these together.
The farther apart they are, the more, we want to bring them together.
And that's what our loss function is going to be.
This is just another learning rate function.
So here's our data set in TensorFlow.
You can basically define your data set sort of iteratively
where it reads in these files.
And then it processes them as you go.
And that will be a little bit faster than
if you tried to load it all into memory.
So let's look here.
I'm now initializing our model, our clip loss model.
I'm giving it a cosine decay, which is a type of learning rate.
I'm giving it the AdomW optimizer.
And then I'm just calling model.fit on it.
So each iteration is actually going to go for about 500 steps.
Then it's going to pause.
You can see it's paused right now.
And it's going to evaluate on a validation set of CheXpert.
So but what's interesting here is that, and I maybe
buried the lead a little bit here, but CheXpert, we
don't have the radiology reports for it.
So we have the radiology reports for MIMIC
and we're using that to train the model.
So how do we get a result from CheXpert at the time of evaluating our model?
Well, I glossed over this before.
But let's go back here.
The way that we're defining our labels now for CheXpert
is this so-called zero shot or kind of a soft prompt, right?
So we get to pick whatever we want to define our labels.
So if you want to search for a right pleural effusion
or a left pleural effusion, you can do that instead
of just a label of effusion or not.
You can specify moderate cardiomegaly, severe cardiomegaly.
And we're going to do a little bit of math to turn this into a prediction.
So over here on the right, this set of prompts
is going to say, OK, this is normal.
So as I mentioned earlier, this sort of template
of saying no acute cardiopulmonary process, that means nothing
is going on there.
And there's no pleural effusion.
So we can get our prediction from this.
Let's see how our training is going here.
So we got our first result. I'm actually going to just halt it
there since in the interest of time.
So you can see we're getting really good AUC
scores even though the model has not seen
this data set or these exact phrases.
We're sort of asking a question of the model that hasn't been asked before.
And one cool thing that we can now do with this,
so what if we wanted to write a plain text chest x-ray search engine?
So as I mentioned earlier, we have the ability
to query now our CheXpert data set where we don't have the radiology reports.
What we do is we feed in our query, which is in natural language.
We turn it into numbers, and then we ask the model
how similar is this to the image?
And when we do that, we can then rank them and say here's the top five.
So I have a query here of large bilateral pleural effusions.
And this is going to give me results that have some of those things.
And if we wanted to look for a, let's see, a normal chest x-ray,
we can do that as well.
So as you can see, these are a lot more normal.
So we just built a search engine for our chest x-rays.
You could imagine this would be useful if you were a radiologist
and you want them to look up an example of a case,
but we didn't have radiology reports for it.
So I am going to have to hurry through this last one.
So let's just get to it.
For my next trick, we're going to train a radiology report generator.
So this is not going to be perfect, but it's
just a pretty cool demonstration of what we have,
what the possibilities are here.
So the setup is actually quite similar to our clip model
where we have a text model over here on the side in addition
to our vision side of things.
In this case, it's going to be what we call an encoder decoder.
The encoder is actually not used.
The decoder is a BERT, which is a type of transformer,
which is used a lot in natural language processing as I mentioned.
So we're basically going to give it our image embedding.
We feed that into the text model, and then we
say give us some text for this, which is a crazy thing
to do to be honest to ask it to do something like that.
So you can see it's going to take a couple of minutes to train.
But thankfully, I did pre-fab the results I hope.
One interesting thing to note here, I had
to reduce the batch size considerably.
It no longer fits in my GPU even, which is
why it's also slower, some of the things that you just sort of wrestle
with as an ML practitioner.
And let's just say this is my final trained model I hope.
So now, I'm going to run through our validation data set.
And I'm going to ask it to give me the radiology reports.
And I'm going to compare that to what the actual ones are.
So to be honest, it's surprising even to me how well this works.
So here's a normal one, no acute intrathoracic process,
no evidence of pneumonia.
Here's an example where it's not working,
which is it's always good to start looking at.
So Dobhoff tube is one of a type of tube for I
think it's for feeding, a feeding tube.
And you can see here, instead of Dobhoff, off it says do something.
So it doesn't quite know what to do with that.
Same here, possible cavities, well, it says possible cat.
But it's getting there.
And it's pretty crazy that it can do this with just our embeddings.
And it gives you a sense of what you could do with yours.
So I do I want to quickly, before I wrap up what's so to read more about this
and get more involved, we did publish our paper in Radiology
it's called Simplified Transfer Learning for Chest Radiography
Models Using Less Data.
There's the free API, which I'll have links to where you can,
and we'll have a collab where you can use some of these free data sets.
MIMIC and CheXpert, you can also work with, which for Mimic,
you'll need to go through a training.
For CheXpert, you, I think you need to sign some agreements.
But they're also available to you.
Our goal really is to accelerate research.
We're always looking for collaborations if you
know people, if you are people that want to collaborate and build something
cool with this, please reach out.
What's next?
We want to do this for more modalities.
We want to do it for potentially ultrasound and some others
that I think would benefit a lot from this I also have some links here.
There's a software engineering internship.
Applications are open now through December 1
for undergrad, graduate, PhD students.
There's some connect with Google resources.
So this one references students.
And I think it's focused on the US, but there also is a link.
And maybe I can break them out.
There's links in here for around the world
as well as industry so a few others that are maybe
more specific to whatever group you identify with
and some other resources, Google Careers, OnAir.
And yeah, again, all of these will be linked for you.
So I want to pause there.
Sorry I went even longer than I expected,
but I do want to stick around for questions.
I'm able to say around four 20, 25 minutes
depending on how long people want to stay.
But again, thank you so much.
I really hope this has inspired you, gotten you excited about it,
also giving you the confidence that this is something you can do.
And you shouldn't be overwhelmed by it or feel
like it's beyond your reach you absolutely can do it.
And CS50 will give you the tools to do it, so.
DAVID MALAN: Thank you, Andrew so much for joining us,
and maybe a virtual round of applause with your favorite emoji reaction
or physical hands, so good to reunite.
If I could ask just one question on behalf of the group
and then turn things over to you and Carter
so that I can get to another meeting myself if you don't mind.
But we'd love for folks to stick around for more one-on-one Q&A
if you would like.
I think one question for the group that was recurring in the chat was just
how folks can learn more, not just about these resources
that you have on the screen.
But if they want to learn more about AI, like you did on your own,
if they want to learn more about AI in medicine, are there courses?
Are there universities?
Are there other learning resources you would recommend specifically?
ANDREW SELLERGREN: Yeah, I would definitely
as I said before, I would highly recommend
CS50's AI for artificial intelligence in Python.
The Coursera course is good, too.
I would be remiss to not mention that.
Machine learning was the original Coursera course.
O'Reilly has great books.
I always, when I'm trying to learned something new,
I always go for O'Reilly first.
Those are the ones with the funny images of animals on the front.
Those are usually a good, and honestly YouTube.
There's just so much content these days of people walking you
through things, getting you bootstrapped and those things.
But maybe I can come up with a list of resources, too,
and we can post that with the rest of these slides and so on, David.
That would probably be good.
DAVID MALAN: Absolutely.
I've been sharing the slides in the chat,
and we will post everything at cs50.ly/zoom as well for folks
afterward.