ANITA KHAN: Hi, everyone.
Welcome to Data Science with Python pandas.
This is a CS50 seminar and my name is Ms. Anita Khan.
Just to give you a little bit introduction about myself,
I'm a sophomore here at Harvard and I'm in Pfoho.
This past summer I interned at Booz Allen Hamilton, a tech consulting firm,
and there I was doing server security and data science research.
On campus, I'm involved with the Harvard Open Data Project,
among other things, where we try to aggregate all of Harvard's data
into one central area.
That way students can work with data from all across the university
to create something special, and some applications that improve student life.
I'm also on the school's curling team.
Just to give you a brief introduction about data science,
this is an incredibly evolving field That's? growing so quickly.
Right now on Glassdoor it's rated as the number one best job in America in 2016.
And you can have a median base salary of $116,000.
Harvard Business Review also listed it as the sexiest job of the 21st century,
and it's always growing.
If we look at Indeed, we see the number of job postings has been skyrocketing.
In the past four years alone, the number of postings
have increased eight times, which is pretty incredible, just
because data science is such a growing field and every company now
wants to use it.
If we also look at job seeker interest versus job posting,
we see that there are, at max, there are sometimes 30 times more posts
than there are people to fill them.
And we also have at minimum still almost 20, which is incredible.
And we want to fill that demand.
So here I'll be teaching some data science.
Stephen Few once said that numbers have an important story to tell.
They rely on you to give them a clear and convincing voice.
And today, I'll be helping you to develop
that clear and convincing voice.
If we look at some examples of data science,
we have seen things like how data science has been used
to predict results for the election.
And so we can see here, there is a diagram
about how many different ways Clinton can win,
how many different ways Trump can win, just depending
on the number of different results.
And this results in a very interactive and intuitive visualization.
So if we want to look at the brief article here,
we see here some election results.
And this was just released pretty recently actually,
updated 28 minutes ago.
So we see here there are things like the percentage over time
on the likelihood of winning.
We also have where exactly the race has shifted,
some things about state by state estimates divided by time.
And so this is just a really intuitive way for people all across the world
to be accessing this data.
And so data scientists are always taking this data and these huge spreadsheets
that aren't always accessible for people to see,
and that way people can actually observe what's going on.
You also can see different forecasts, some different outcomes that
are pretty likely, and, again, an interactive visualization
for people to really understand the data that's going on.
Some other ways are that we can see Obama rates are rising here.
Obamacare rates are rising and there is the graph to see how that's changing.
We've also used data to catch people like Osama bin Laden,
and to fight crime.
So data scientists have a lot of different usages
across many different fields.
There are many steps to data science and I'll be going through them today.
So the first thing is you want to ask a question.
Ask a question, then you want to-- it's important to ask
a question because otherwise there's nothing to answer.
Data science is a tool, and so once you have a question
to answer you use data science to answer that.
You can't just use data science on some arbitrary data
set that you don't care too much about.
Next you want to get the data.
And so there are a wide variety of places to get the data,
but you just want to find a data set that you also care about.
After that you can explore the data set a little bit,
get a better sense of what kind of descriptive statistics
you're looking for.
Next, you want to model the data.
So what happens if you are trying to predict
something years into the future?
What happens if this scenario occurs?
Or what happens if this predictor changes a lot?
Then you want to see what could possibly happen based on your model.
And models always improve when you have more data,
and so it's always good to get more data.
Finally, you want to communicate all of your information.
Because while it's great that a data scientist has
all of this information that they found, and all these visualizations,
it's really important to share that with your boss or other colleagues.
That way there can be something actionable about it.
So in the examples we showed before, we've
seen things like how Osama bin Laden was caught using data science.
But if the person who's the data scientist came up with the data
and couldn't present that effectively, then that couldn't have happened.
There are a bunch of different tools to help you get along, help
you find all this information.
So when asking a question, you can think about to your own experiences.
What are some issues that you faced before?
What is something that you want to know more about?
You can also look at websites, like Kaggle,
for example, which presents data challenges pretty frequently.
And so if a company poses a question, you can always answer them yourself.
You can also talk to some experts what kind of things
are they looking to answer that they might not necessarily
have the capability to address.
And so you can help them using the data that you find to answer their question.
As for getting the data, there are many different ways to get data.
You can scrape a web page.
So that you can get information that way.
You can also look at databases if you have access to one.
And finally, a lot of different places have Excel spreadsheets or CVSs,
Comma Separated Values, and text files that are really easy to work with.
After that, you want to explore the data a little bit.
And so we have a couple of different Python libraries, along with others,
but Python seems to be pretty common in the industry.
You have libraries such as pandas, matplotlib,
which is more for visualization, and then NumPy as well,
which works with arrays.
And so after that you want to work with modeling the data.
So, extrapolating essentially.
And so you can also do this with pandas and also
a library that's gaining a lot of traction
is sklearn, which is more for like machine learning.
And finally, you want to communicate your information.
So matplotlib is great for creating graphs and d3 is great for creating
interactive visualizations.
But as we've seen before, pandas is used in both explore and modeling.
And also, matplotlib and NumPy is built into panda.
So that's why pandas is great.
So we're going to be exploring that today.
Just a little bit more information about pandas.
It's a Python library, as I mentioned before.
And it's great for a wide variety of steps in the data science process.
So, things like cleaning, analysis, and visualization.
It's super easy, very quick.
And it's very flexible, so you can work with a bunch of different data types,
often many different types at once.
You could have several different columns with strings, but also numbers,
and even strings within strings, and it's great.
And finally, you can integrate well with other libraries
because it's built off of Python, it works with NumPy tools
and other different libraries as well.
So it's pretty easy to integrate.
Next, we'll also be using Jupityr Notebooks.
So this is kind of similar to the CS50 IDE,
but this is preferential for data science
because you can see the graphs in line and you
don't have to worry about loading things separately.
You also have all of your tools and all your libraries already loaded.
So if you download a package in SAR called Anaconda,
that has all of these tools already.
It also allows over 40 languages.
So today, we'll be focusing on Python but it's great
that you can share notebooks and work with many different languages as well.
So we're going to just launch into pandas.
And so there are two different data types in Python pandas.
So the main one is called series and there's
another great one called DataFrame.
And so series are essentially NumPy arrays.
They're essentially arrays.
So you can index through them, just as you did in CS50,
but one difference is that you can hold a lot of different data types.
So this is kind of similar to a Python array.
So we can work on a couple of different exercises.
So here is going to be our notebook where
we're going to be working with all of our information.
This way you can see everything as it goes.
So you have the code here, and then if you press
Shift-Enter it loads the code for you.
So here in this section, we're going to be exploring different series.
And so first you want to import the library as you did in the last P set,
for CS50.
So if you import pandas as pd, that pd means
that you can access different modules within panda just using the word pd.
So you don't have to type pandas all the time.
So if you want to create a series, you just call pd.Series.
And then this generates this NumPy command.
Import NumPy as np.
This NumPy command generates five random numbers, and then in the series
you'll also have an index.
So let's see what it creates.
As you can see, you have an index here, a, b, c, d, e.
And then you have your five random numbers from just here.
Because this isn't saved inside of a variable,
it's just pd.Series, if you want to save it inside of a variable,
you can also do the same thing.
You also don't need to have an index, you can just have 0
and the default is just 0 through 4.
Next, you can also index through them because there are different arrays.
So can someone tell me what ss[0] would return here?
AUDIENCE: The first value.
ANITA KHAN: Yeah, exactly.
And then do you know what this one is?
AUDIENCE: That's all the values up to the third.
ANITA KHAN: Yup, exactly.
So here you have your first value, as you had here.
And then after, when you are slicing through them, it gets easier.
1 and 2.
So that's a series in a nutshell.
The next type of data structure is called a DataFrame.
And so essentially this is just multiple series added together into one table
that way you can work with many different series at once
and that way you can work with many different data types as well.
You can also index through index and columns,
that way you can just work with many different data types very quickly.
So here we're going to do a couple of exercises with DataFrames.
And so first we create a DataFrame in the same way.
So when we call pd.DataFrame, that means you access the command DataFrame
in pandas.
That means you create a DataFrame out of (s).
So (s), remember, was this series back up here.
So we're going to create a DataFrame out of that series,
and we're going to call that column Column 1.
So as you can see, it's the exact same series
that we had before, these random five numbers put into this DataFrame.
And then its column is named Column 1.
You can also access the column by the name
if you want to have a specific column.
So if you call df, which is the name of the DataFrame,
and then in brackets ["Column 1"], kind of like what we did in the last piece
with accessing like dicts, then you can access that first column.
It's also really easy to work with different functions applied to that.
And so for example, if we wanted to create another column called Column
2, for example, and we want that column to be the same as Column 1
but multiplied by 4, it would just be like adding another element
in that dict.
So then it would be df, and then in that dict
we'd be creating something else called Column 2.
And then that's equal to the Column 1 times 4.
And so as you can see, we've added a second column that's exactly
the same, except it's multiplied by 4.
So it's pretty intuitive.
You can work with many different other functions as well.
And so if you want to add something like df times 5, or like subtracting,
or you can even add or subtract two different columns,
you can add multiple columns, it's pretty flexible with what you can do.
You can also work with other manipulations,
such as a thing like sorting.
So if you want to preserve-- you can do other things such as sorting.
So if you want to sort by Column 2, for example,
you can take this column and you can call df.sort_values and then by Column
2.
And if you want to preserve it, make sure to set it to a variable
because this just sorts it and it doesn't actually affect
how the DataFrame actually looks.
And so if you sort by Column 2, you can see
that the whole DataFrame is just sorted with these indices staying the same.
So, for example, you see that Column 2, this one
has the lowest value so it's going to be at the top,
and then you also have the indices preserved sorted by that Column 2.
You can also do something called Boolean indexing, which
is where-- so if you recall from a Python array, if you just call,
for example, is this array less than 2, then
it should return trues and falses to see whether each element is actually
less than 2.
So this same concept can be applied to a DataFrame.
And so if you call this DataFrame, if you
want to access things that in Column 2 are less than 2,
then you can just do syntax like this and it would
return every column that's less than 2.
As you can see, the first row has been eliminated
because Column 2 is not less than 2.
You can also apply things called anonymous functions.
And so if you have something called lambda x,
is the minimum of the DataFrame plus the maximum of the DataFrame,
then you can apply that to your DataFrame
and then that should return the result of whatever this should be.
So, for example, if you run this you take the minimum of Column 1
and then you add it to the maximum of Column 1.
And this result is negative 1.31966.
And then you do the same thing for Column 2 as well.
So you can run the same thing to another-- you can also
add another anonymous function.
Do you want to try it out?
Give an example?
So it's something like df.apply (lambda x).
AUDIENCE: A mean?
ANITA KHAN: Mean?
OK.
mean(x).
Oh, whoops.
That's why you don't do live coding during seminars.
You can also call on mean(df) and then that should-- np.mean(df).
And then that should return the mean as well.
Finally, you can describe what different characteristics of that DataFrame.
And so if you do something like df.describe,
it returns how many values are inside the DataFrame.
You can also find things like the mean, standard deviation, minimum, quartiles,
and finally, the maximum.
So it's pretty easy once you have all that data loaded into DataFrame if you
call df.describe, then that allows you to access pretty essential variables
about that DataFrame.
That way you can work with different things pretty quickly.
So if you want to subtract and add the mean,
then you have these two values here already.
If you want to access things like the mean exactly, you could call--
if this is the table-- then you want to call table(mean), or ["mean"],
that should access the means as well.
So we're going to go through the data science process together.
So, the first thing we're going to do is ask a question.
So what are some data sets that you're interested in
and what kind of questions do you want to answer with data?
AUDIENCE: Who's going to win the election?
ANITA KHAN: Who's going to win the election?
That's a good one.
AUDIENCE: Anything to do with stock prices.
ANITA KHAN: Stock prices.
What kind of things with stock prices?
Kind of similar to CS50 Finance?
Or like if you want to predict how a stock moves up and down?
AUDIENCE: Yeah.
ANITA KHAN: OK.
All very interesting questions.
And the data is definitely available.
So for something like-- yeah, we can go through that later.
So today we're going to be exploring how have earth's surface
temperatures changed over time.
And this is definitely a relevant issue as global warming is pretty prevalent
and then temperatures definitely are increasing a lot.
We had a very hot summer, a very hot winter.
So this might be something we want to explore,
and there are definitely data sets out there.
So for getting the data in this kind of example,
so where do you think you'd get data about who's going to win the election?
AUDIENCE: I'm sure there's several databases.
Or past results.
ANITA KHAN: Past results of previous elections?
AUDIENCE: Yeah.
And polls.
ANITA KHAN: Where do you think you could get data about elections?
AUDIENCE: Previous polls.
ANITA KHAN: Yeah, definitely.
And as we saw before in the New York Times visualization,
that's how a lot of people predict how the elections are
going to go, just based on aggravating a lot of different polls together.
And we can take maybe the mean and see who's actually
going to win based on all of these.
That way you account for any variance, or where different places are,
and who different polls are targeting, and so on.
So for something like stock prices, what would you look at?
Or where would you get the data?
AUDIENCE: You could start with Google Finance.
ANITA KHAN: Google Finance.
Yeah.
Anything.
AUDIENCE: Like Bloomberg or something like that.
ANITA KHAN: Yeah, for sure.
Same thing?
AUDIENCE: Same places, I guess.
ANITA KHAN: Same places.
Yeah.
And what's really cool is that there are industries
that are predicated off of both of the questions that you're asking.
And so if you can use data science to predict how stocks are going to move,
that's how some companies operate.
That's how they decide what to invest in.
And then for elections, if you can predict the election,
that's life changing.
And so here we're going to get the data from this place called Kaggle.
As I mentioned before, it's where a lot of different companies
pose challenges for data science.
And so if we look here, there is a challenge
for looking at earth's surface temperature data since 1750.
And it was posted by Berkley Earth pretty recently.
What's great about Kaggle is that you can also
look at other people's contributions or discussion about it
if you need help about how do you access different types of data.
So if we look at a description of this data,
we see a brief graph of how things have changed over time.
So we can definitely see this is a relevant issue.
And you can see from this example of data science already,
it's pretty intuitive to see what exactly is happening in this graph.
We see that there is an upward trend of data happening over time,
and we see exactly what are the anomalies over this line of best fit.
We also see that this data set includes other different files,
such as global land and ocean temperature, and so on.
And the raw data comes from the Berkeley Earth data page.
So if we download this-- it might take a little bit to download because it's
a huge data file, because it's containing
every single temperature since 1750 by city, by country, by everything.
So it's a pretty cool data set to work with.
There's a lot of different data sources.
And while this isn't quite like technically big data,
this definitely is a chance to work with a large data set.
So if we look here, we can look at global temperatures.
So here you can see some pretty cool information about the data.
You see that it's organized by timestamp.
You can look at land average temperatures, you can see here.
Might be kind of hard to tell.
Land Average Temperature Uncertainty, that's a pretty interesting field.
Maximum Temperature, Maximum Temperature Uncertainty, Minimum Temperature.
So it's always great to look at a data set, like once you actually have it,
what kinds of fields there are.
And so there's things like date, temperature.
We see that there are a lot of different blanks here, which is kind of curious.
And so maybe this could get resolved later in the data set?
And we see that this goes all the way up to the 1800s so far.
And then we see here that the other fields are populated here.
So it's possible that before 1850, they just
didn't measure this at all, which is why we don't have information before.
So this is something to keep in mind as we work with the data set.
And so we see, there's a lot of information, a lot of really cool data.
And so we want to work with that.
And so we open up our notebook.
You import in all of the libraries you already have.
The great thing about Jupityr Notebook is
that keeps it keeps in memory from things that you've loaded before.
So up here we loaded pandas and NumPy already,
so we don't have to load them again.
And so we just import matplotlib, which is, again, for visualizations,
and graphs, and everything.
And we also import NumPy-- we already imported that-- but it helps
you work with arrays and everything.
This matplotlib inline allows you to look at graphs within Jupityr Notebook.
Otherwise it would just open up a new window, which can get kind of annoying.
And so if you want to see it inline, that way you
can work with things pretty quickly rather than switching between windows,
it's a good thing to use.
And then this is just a style way of preference
for how you want your graphs to look.
And so if you use the default, it's just like blue.
I wanted it to be red and gray, and nice so I changed it.
So if you call pd.read_csv-- again, remember that pd is referencing pandas.
And so this is accessing a module in pandas called read_csv.
So it let's you load in a CSV, just with a single command,
and that way it loads into your DataFrame.
And so if we call that-- yeah.
So this looks exactly the same way we had it before, or had it
in the Excel spreadsheet, just loaded into a DataFrame.
So again, very simple.
If you want to see the rest of the file, you just call df.
I just chose head(), that way head shows the first five elements rather than
every single thing, because it was a pretty long data set.
But it does show the first 30, and then also the last 30 I believe.
And so you can see that there are 3,192 rows and 9 columns,
just from loading it in.
You can also call tail(), and then that should show you the last five elements.
You can also change the number within here to be the last 10 elements.
So you can see things pretty easily.
Next, we want to look at just the land average temperature.
That way we can work with just the temperature for now.
The others are a little bit confusing to work with,
and so we want to just focus on one column for now.
Plus, that's what we're interested in.
We want to see how temperature has changed over time.
So we want to look at just temperature.
And so this is a method to index.
And so this takes the columns from 0 all the way up to 2,
where it stops right before 2.
And then it gets to zeroth column, and the first column.
The zeroth column, remember, is the datetime,
and the first column is the land average temperature.
And then again, we want to take the head().
So as you see, it's just the datetime and the land average temperature.
And we also changed the DataFrame to be updated to this.
That way we can just work with just these rather than the rest of them.
Next, as we saw before, df.describe was a very helpful tool.
And so if we run that again, that will allow
us to see basic information about it.
And so we see that there are in total 3,180.
And then we also have a mean temperature.
We have a standard deviation for temperature.
We have our minimum and maximum as well.
And we also see that we have NaN values, which means it's not a number.
So that's a little bit curious.
We might want to explore that a little bit.
In all likelihood, it probably is just that there are Not a Number
values in there, and so it's hard to find quartiles when some of them
are not valid numbers.
So once we have a description, we can see
we've gained insights already about it, just from those couple lines of code
up here.
And so we see that the mean temperature from 1750 to 2015
was 8.4 degrees, which is interesting.
Next, we want to just plot it, just so we
have a little bit of a sense of how the data is trending.
We just want to plot it, just to see we can explore some of the data.
And plus, it's pretty easy to apply it.
So even if it doesn't look too great, then we aren't losing anything.
And so, plt. Again, we imported matplotlib, which
is the library that helps you plot.
matplotlib.pyplot helps you plot.
And then if import it as plt, you can access all the modules from just
calling plt().
And so we have plt.figure.
plt.figure(figsize), that just defines how big that graph is going to look.
And so we call its going to be 15 by 5.
And so you have the width is a little bit bigger,
and that's to be expected because it should be like a time series graph,
and so there will be more years than there are actual temperatures.
Next, we're going to actually plot the thing.
And so since we have a DataFrame that has all that information,
we can just plug that in.
And this command knows exactly how to sort between the x and y,
so you just need to call that DataFrame.
The only thing is that matplotlib in this case would plot a series.
You can also plot multiple of them.
But as the series, as you remember before,
is a one-dimensional array with an index.
And so in this case that land average temperature, or the temperature itself,
would be what you plot on your y-axis.
And then the x-axis would be the index.
So that would be what year you're in.
You can also plot a whole DataFrame.
And then this, we'd just plot all the different lines all at once.
So if you had a land maximum temperature,
then you can see the differences between that.
We also have plt.title, that changes the title of a whole graph.
You have the x label, year, and y label.
And finally, you want to show the graph.
You also don't have to, but because of Jupityr Notebook,
so then same thing happens.
And so you see from this graph, it's a little bit noisy.
And so we see that there seems to be an upward trend,
but it's kind of unclear because it looks like things are just
skyrocketing back and forth.
Do you have an idea why that might be the case?
AUDIENCE: It's connecting the dots.
ANITA KHAN: Yeah, exactly.
Yeah, that's exactly right.
And so we also see from the table up here,
there are different months located.
And so, of course, the temperature will decrease during the winter
and increase during the summer.
And so as it connects the dots, as you said,
then it'll just be connecting the dots between winter and summer
and it will just be increasing a lot.
So this graph is kind of messy.
We want to think about how exactly we can refine it.
But we do see that there is a general upward trend, which
is a good thing for us to see, probably not good for the world, but it's OK.
We can also pretty clearly see what the ranges are.
And so we see here, you can get from as low as couple of negative degrees
up to almost 20 degrees, which is consistent
with our df.describe findings.
We also see that it goes from the 0 to the 3,000, or almost 3,200,
which is not quite correct because we only had the years from 1750 to 2015.
And so there's something incorrect here.
It's probably referencing the months maybe.
AUDIENCE: I think it's referencing the indexes?
ANITA KHAN: Yeah, exactly.
Referencing the indexes, but each row is a month.
And so it would be like the zeroth month, first month, and so on.
So how do you think we can make this graph a little bit smoother,
so that it doesn't go up and down by month?
AUDIENCE: Make a scatterplot?
ANITA KHAN: Scatterplot.
But if you had the points-- yeah, we can try that.
So plt.plot(kind=scatter).
And then for a scatterplot, you need to specify the x and the y.
So we could have x equals the index, as we said before.
And the y equals the actual thing itself.
plt.scatter.
Scatterplot.
So we still see a couple different-- it's still a little bit messy.
It's still kind of hard to see exactly where everything is.
What else do you think we could do?
So right now we have it indexed by month.
What do you think we could change about that?
AUDIENCE: You can have dates by year.
ANITA KHAN: Yeah, exactly.
So if we ever--
AUDIENCE: Like the max temperature.
ANITA KHAN: Max temperature, yup.
All very good ideas and something to definitely explore.
So for now we can just look at the mean of the year, or average of the year.
That way we can see because each year has all of the months,
it would make sense just to average all of them,
just to see how that's been changing.
However, we notice when we look at the timestamp column, which
is called DT, if we access that and called the type,
it's actually of type str.
So that means all of these dates are recorded inside
of the file as a string rather than a date.
So that would mean if we want to parse through them,
we have to look through every single letter inside of the DT.
So what might be helpful is to convert that to something pandas
has called a DatetimeIndex.
Pandas is very adapted towards time series data.
And so, definitely, there are a lot of tools in their library for this
exactly.
So if we convert it to a DatetimeIndex, we can also group it by a year.
And this is a syntax where we take the year in the index,
and then we also take the mean of every single one.
So if we run that, and then we plot that again, that's a little bit smoother.
So we can definitely see that there is a trend over time.
And as there are a lot of different spikes,
so it's not incredibly uniform, which makes sense
because there are peaks and valleys for years.
But as a whole, this data set is trending upwards.
So this is wrapping up the exploratory phase.
But then we notice there is something pretty anomalous here.
We see right around the 1750, in the beginning with 1750s,
there's a huge dip down.
So before while it was at 8.5 before, it went all the way down to 5.7.
So let's see.
There might be a couple of reasons why this
might be the case, such as maybe there was
an ice age for that one year or something
and then it went back up to 8.5.
But that's probably not what happened.
So let's look into the data a little bit.
Maybe they messed up something, maybe someone mistyped a number.
So that it says negative 40, or negative 20 instead of 20,
or something like that.
And so if we look at the data-- and it's important to check in with yourself,
make sure that what you're getting is reasonable-- we can look in.
And so we want to see what caused these anomalies.
Because it was in the first couple of years,
we can call something like .head(), which shows the first five elements.
And we see here that 1752 is what caused this.
And for whatever reason, even though all of the years previous and after
had 8 degrees and then 9 degrees.
It just goes back down to 6.4 degrees, which
matches what we found in our plot.
So let's look at that data set exactly.
So, as you remember, we can filter by Booleans.
So if we want to see if the year of that grouped DataFrame is equal to 1752,
we can see what happened.
And so we see here, in this case we can see every single temperature
from every single month, and the land average temperature,
as long as that year is 1752.
And because it's a DatetimeIndex, we're allowed
to do something like that, rather than searching
the string for every single thing, looking for 1752.
And so we see here in this exploration that land average temperature,
so while this January makes sense that it's pretty low,
we also have things like Not a Number.
And you have things, like you have a couple of the numbers
but then all these summer months are just gone.
And so what happens is when you average this, where it might not have a number,
it'll just average the existing values.
And so because you're missing those summer months
it'll be low, even though it's not supposed to be.
So what exactly can we do about that?
So there are a lot of null values.
You want to see what exactly we can do.
Also, this might be affecting results in the future.
Because what happens if there are other null values in other years?
It wouldn't be just exclusive to 1752.
And so again, as we tried from that Boolean values,
if we call numpy.isnan(), that can access every single thing and determine
which cells exactly are not a number.
And specifically, land average temperature is not a number.
And so we see here that there are a lot of different values
that are all not a number.
And so this is OK.
It definitely makes sense, because no data set is going to be perfect.
As we saw before when we were looking at the data set,
it was missing all these columns.
And so it's not ever going to be perfect, which is OK.
The thing that you have to do is either work with data that is perfect,
or you have to fill in those null values.
You have to make sure that it has something that's reasonable
that shouldn't affect your data that much,
but you should fill it in with something that makes sense.
So, in order to find out what exactly makes sense,
we want to look at possibly other information around it.
So if we wanted to predict this February of 1752,
how do you think that we could estimate what that should be?
AUDIENCE: Look at the previous and past February's?
ANITA KHAN: Yeah, exactly.
Yeah, previous and past February's are a good way.
Another way to do it might be looking at the January
and the March of that same year.
It should be somewhere around the middle maybe.
Because to get from that January to the March,
you have to be somewhere in the middle.
And so February would make sense that it should be right around the middle.
And then you could do the same thing for these values as well.
It's kind of a little bit more difficult because you
don't have before and after values for where there are a lot in the sequence,
but definitely looking at the year before,
the year after might be helpful.
So what we're going to do today is we're going
to be looking at the month before, or previous thing that's most valid.
So, for example, in February you would look at the month before.
So then this would be that January.
For this May, you would be looking at the April previously.
And then for this June, because the most previous value is that April,
you'll be looking at that April value as well.
So you'd just be filling all of these with this April value.
So, not the most accurate, but it's something that we can at least say it's
reasonable.
So you're going to be changing the value of what that DataFrame column is.
And so we want to set that equal to something else.
And it's going to be exactly the same thing,
but we're going to be calling a command called fillna().
It's another pandas command, but it fills all of the null values.
So these are things like none, NaN, any blank spaces, or anything,
just things that would go under na, that you would classify as na.
And the way we're going to fill this is going to be called something ffill,
or forward fill.
So this is going to be things from before
and then it's just going to fill the things ahead of it.
You can also do backward fill, and there are some other different ways as well.
And so once we call that, it changes.
And then we can graph that again.
And then we see it's a little bit more reasonable.
There still are some dips and everything, but it can't be perfect.
So we might want to try different avenues for the future.
That data set definitely looks a lot cleaner than it was before.
And we know that there are no null values as of right now,
so then we can work with the whole data set
and not have to worry about that at all.
All the syntax for the plots are pretty similar.
So you can always definitely copy it, or even create a function out of it,
that way you don't have to worry too much about styling and everything.
You can also change things like the x-axis, y-axis, font size.
So it's pretty simple.
So that concludes our exploration of our data set.
Next, we want to model our data set a little bit
to predict what would happen based on future conditions
or other variables that could happen.
So in your example of predicting the election, what would you want to model?
AUDIENCE: Who gets electoral votes.
ANITA KHAN: Yes, exactly.
And then for stock price, what might you want to model?
AUDIENCE: Likely [INAUDIBLE].
ANITA KHAN: Yeah, exactly.
And how that all change over time.
And so there are different ways to model.
The model we're going to use today is called linear regression.
So, as you might have learned before in class,
just like creating a line of best fit.
That way you can estimate how that trend is going to change over time.
So we're going to be calling in a library called sklearn.
So this is used for typically machine learning,
but definitely regression models or just seeing
how things will change over time, this is good for, and pretty easy to use.
And so this is just a couple of syntax values,
that way you can set what that x is and what that y is.
You just want to take just the values rather than a series,
and that creates a NumPy array.
And then when you import this as LinReg.
You can just call your regression is equal to this.
And then sklearn has a quirky syntax where you want to fit it to your data
first, and then you can predict the data based on what you had there.
That way if you want to predict a certain value that
wasn't in your data set, you could call that in predict.
And so if you call reg.fit(x, y), that should find the line of best fit
between x and y.
And then if you want to predict something,
then you would call reg.predict(x).
You can also do something called score, which
is where you compare your predicted values against your actual values.
And so here you put in x, which would be your predictors,
and y, which is like your predicted values.
So in this case x would be the year, and then y
would be what exactly that temperature would be.
And so you compare what the actual temperature is against what
your predicted temperature is.
Next, we want to find that accuracy to see how good our model is
and everything.
And so this compares how far the predicted point is
from the actual point, does residual squares,
and r-squared, if you heard that in stats.
And so we see that it's not very accurate, but it's better than nothing.
It would be better than a random point.
And since this was a very basic model, like this is actually not terrible.
It's a good way to start.
And so next we want to plot it to see exactly how accurate is it.
Because while this percentage could mean something as to how accurate it is,
it's not that intuitive, and so we want to graph it.
So again, graph it as we did before.
Scatterplot is good for this.
And we see how all of these points-- you see
that we have our straight line of best fit here, that blue line,
but then we also have all of our points.
And we see that it's not perfect, but it definitely
matches the trend in data, which is what we're looking for.
And so if we wanted to predict something like 2050,
we would just extend that line a little bit further.
Or if you just wanted the number, you could call reg.predict().
And so this is what we did here if you call that reg.predict(2050).
So this predicts that the temperature in 2050
will be 9.15 degrees, which is pretty consistent with what this line is.
Do you have any ideas for a better regression model?
So instead of linear, what might we do?
AUDIENCE: Like a polynomial?
ANITA KHAN: Yeah, exactly.
So it looks like this data set is following a pretty curvy model.
We see while it's pretty straight here, it curves up here.
And so, definitely, polynomial might be something to look for.
There's also another pretty cool method of predicting
called k-nearest neighbors.
And what this is you find the nearest points and then
you predict based on that.
So for example, if you wanted to predict 2016,
you would look at the nearest points, which
are 2015 and 2014, maybe 2013 if you want that.
Average it together and then that would be your prediction.
There are other regression methods as well.
You could do logistic regression, or you can use linear regression
but use a few more parameters.
That way you can decrease the effect a certain predictor has, and so on.
But linear regression is a good start.
You should definitely look at the sklearn library
and there are definitely a lot of different models for you to use there.
And so the next part is communicating our data.
So how do you think we could communicate the information that we have now?
Who would we want to communicate to on global temperature data?
AUDIENCE: [INAUDIBLE]
ANITA KHAN: What do you think?
Same thing?
OK.
If you wanted to communicate something about what your examples are,
once you had data about election predictions,
how do you think you could communicate that?
AUDIENCE: Do something very similar to what the New York Times did.
ANITA KHAN: And what about stock market data?
Who would you communicate to, what would you be sharing?
AUDIENCE: Try to put it in some type of presentation.
ANITA KHAN: Yeah, exactly.
That'd be great.
And you could present to one of these companies,
or you could do it at a stock pitch competition,
or even invest, because maybe you just want to communicate to yourself,
and that's fine too.
But the idea is once you have that data, someone needs to see it.
Once you have that data, it can generate pretty actionable goals, which
is a great thing about data science.
So just talking about some other resources since we've
gone through the pretty simple data science process.
Other resources if you want to continue this further.
I'm a part of the Harvard Open Data Project
where we're trying to aggregate Harvard data sets into one central area.
That way students can work with that kind of data and create something.
So some projects that we're working on are looking at energy consumption data
sets, or food waste data sets, and seeing how exactly
we can make changes in that.
So other than that, again, as I showed you before, Kaggle.
Definitely a great resource if you want to just play with some simple data
sets.
They have a great tutorial on how to predict
who's going to survive the Titanic crash based
on socioeconomic status, or gender, or age.
Can you exactly predict who will survive?
And actually, the best models are pretty accurate.
And so that's really cool that just using a couple regression models
and using exactly the same tools that I showed you, you can predict anything.
Your predictions might not be very correct,
but you can definitely create a model that would be more accurate than if you
shot in the dark.
Some other tools are data.gov and data.cityofboston.gov.
So again, more open data sets that you can play with
and you can create actually meaningful conclusions.
And so in data.gov you could look at a data set on economic trends.
So, how unemployment is changing.
You could predict how unemployment will be in a couple different years.
Or you can definitely get information about how
election races have gone in the past.
You can definitely reach out to organizations like Data Ventures
that works with other organizations, essentially
like consulting for another organization using data science.
There are a lot of classes at Harvard about this.
Definitely CS50 was sentiment analysis.
You can work with that as well.
So if you've got all the tweets of Donald Trump and Hillary Clinton,
and all the other presidential candidates,
and did some sentiment analysis on that, or looked at different words,
you could predict what exactly might happen.
You can also take other classes such as CS109 A and B, which are Data Science
and, I believe, Advanced Topics in Data Science.
CS181 is Machine Learning as well.
There are other classes, I'm sure, that are definitely helping with this.
Also another good resource is if you just Google things.
If you do Python pandas groupby, by, for example, if you forget the syntax,
you can look through great documentation on how exactly to use them.
So it gives you examples, like code examples.
So in case you forget from this presentation, or other tools
that you might want to use as well.
So, for example, if you want to do a tutorial,
or if you want to work with time series, there
are a lot of-- the documentation for pandas is pretty robust.
And same thing for the other libraries as well.
So sklearn linear regression.
Definitely have looked that up before.
And you can do the same thing, where it has parameters that it takes in,
and also what you can call after you've called sklearn in your regression, what
exactly you can get.
So you can get the coefficients, you can get the residuals,
the sum of the residuals.
You can get your intercepts.
There are some other information that you can use.
They probably have examples as well.
They have examples using this, just in case
like you want an example of what exactly yours should look like,
or you want code.
That's definitely helpful.
And finally, just to inspire you a little bit further,
I can talk a little bit about my data science projects that I'm working on.
For one of my final projects for a class I'm
trying to predict the NBA draft order just from college statistics.
So there's a lot of information, I think back up to since the NBA started,
on how exactly draft order is selected, just based
on that college student's statistics.
And so definitely a lot of people are trying--
like there are industries devoted to predicting what will happen
based on those college statistics.
Like exactly what order, how much they'll get paid,
how does this affect their play time while they're on their teams, so on.
Also, over the summer at Booz Allen I was developing an intrusion detection
system in industrial control systems.
Essentially what this entails is industrial
control systems are responsible for our national infrastructure.
And so if we observe different data about them,
we can possibly detect any anomalies in them.
An anomaly might indicate the presence of an attack,
or a virus or something on it.
And so that is a possibly better alternative to current intrusion
detection systems that might be a little bit more complex
rather than just focusing on data.
Something else I'm working on for another final project for class
is looking at Instagram friends based on mutual interactions.
And so each person on Instagram, maybe they like certain people's photos
more often than other people's photos.
Maybe they comment more, maybe they are tagged in more photos.
And so looking at that information, if you look at the Instagram API,
it's pretty cool to see how there is a certain web of influence,
and you have a certain circle that's very condensed
and expands a little bit further.
And what's interesting about that is celebrities, for sure,
they definitely interact with certain people
more or less, definitely get in hate wars, or anything.
For example, Justin Bieber and Selena Gomez.
People found out they broke up because they
unfollowed each other on Instagram.
So I think that's interesting.
Also some other things that I've done are predicting diabetes subtypes
based on biometric data.
So this was in CS109.
First P set, I believe.
And so given biometric data, so it would be information like age and gender,
but also biometric data like presence of certain markers, or blood pressure,
or something.
You can pretty accurately predict what type of diabetes they'll have,
or whether they'll have diabetes or not, like type 1, type 2, or type 3.
And we can also predict things like urban demographic changes.
Because a lot of this information is available online,
you know what socioeconomic status people are in,
but you also know where exactly they're located
based on longitude and latitude.
And so based on how good your regression model is,
if you input in a specific latitude and longitude,
you can predict what exactly socioeconomic status they're in,
which I think is pretty cool.
And over time as well, because their data sets go back many different years.
So those are a couple of ideas.
Any questions about data science?
AUDIENCE: It's pretty cool.
ANITA KHAN: Thank you.
OK.
Well, thank you for coming.
If you have any questions, feel free to let me know.
My information is here if you want any advice or tips or anything.
And also these slides and everything will be posted online
if you want to access that again.
So, thank you.