>> [MUSIC PLAYING]
ROBERT KRABEK: Hello, guys.
My name is Robert Krabek, and I will be teaching you guys
how to scrape the web with Nokogiri, which is a Ruby library,
and Kimono, which is a Chrome extension.
>> So first there's a couple things that you
can do if maybe you've been doing all the psets so far
and your workspace is getting a little full.
We can actually just go and create a new workspace for you
to just do a brand new project in.
So if you do want to continue working in the CS50 template ID
that you currently have, feel free, and you can just
install Nokogiri with CFLAGS equals-- gem install nokogiri.
But otherwise I'll show you how to set a new one up.
And then this is essentially dropping more training wheels.
And you're coding as if you were just coding in Sublime or something.
So if we shift it over.
>> So say this is your current CS 50 ID.
You can just go to Cloud9 here.
You can go to your dashboard.
It should bring up Workspaces tab.
And then you can just click here, Create a New Workspace.
Name your new workspace, maybe test, or scraping.
And then click this custom tab here, instead of the CS50 templates tab.
And then you can just go and create a new workspace.
>> I've already created a workspace here.
So we'll be working with this.
And if you created a new workspace so with the Custom tab,
you can just type gem install nokogiri, which is not going here.
OK, it's a little frozen.
But you can type gem install nokogiri.
And that should be all that there is to the installation.
>> As I said before, if you're still working in your CS50 template ID,
you just need to type CFLAGS equals gem install nokogiri.
And I've already installed it here so I won't do that.
But for those following along, feel free to do so.
>> So once you've got your Nokogiri workspace or library installed,
I'm going to give you a little bit of a crash course in Ruby syntax
because Nokogiri is a Ruby library.
So you'll need to know some basic Ruby syntax for working with Nokogiri.
So some basic differences from what you're used to
perhaps if you've been working so far in just C and PHP,
you declare variables with no type.
You don't use semicolons, which is kind of a relief.
There's no parentheses now around for or while loops, for example.
You just have a block of code, and then you put end at the end of that.
There's no plus plus or minus minus, so just
know that for when you're doing for loops,
just plus equals and minus equals.
And instead of hash include, you'll use require and then
whatever library trying to load into your program.
>> Ruby isn't a compiled language.
So that's another relief.
It's more similar to PHP where it's an interpreted language.
You can run any Ruby script that you write with Ruby followed
by the name of your script or program.
To signify that it's a Ruby program, you just end it with .rb instead of .c.
And there are variable sized arrays in Ruby,
which is super convenient when you're scraping and perhaps want to append
data that you've scraped into an array.
You don't have to malloc a new array and copy the old array into the new array.
You can just append with the two arrow signs.
And there are no chars, there just single letter strings.
So that should be a little easier.
>> So we'll just give you some examples of some basic Ruby syntax.
So here you can see that instead of the slash slash, to comment in Ruby,
you just use the pound sign.
And variable declaration, you just type the variable equals
whatever you want the variable to be.
They can be strings.
You can have array, which you populate with values.
puts and prints are similar.
For our purposes, the only difference is really
that puts, which stands for puts, just puts a new line
character at whatever you're printing.
>> So if we give a small demonstration here,
we can run this with-- open a new terminal.
You can see all of these files that are in my terminal.
And if I just run Ruby, ruby intro.rb, it
puts out five Hello Mather, Quincy, Carrier.
Adams.
So that's all there is to declaring arrays.
AUDIENCE: Robert, can you make your font a little bigger?
ROBERT KRABEK: Yes.
And I can zoom in because you can't zoom in to terminal fonts apparently.
>> So that's how you print variables to your terminal.
You can also use variables inside a string.
So recently in PHP, you might have learned
that there is string interpolation.
So if you take a look here, if I declare three variables, name, library,
and language, and I puts, I write a string, hello my name is.
And then instead of the PHP version of string interpolation
which looks a little more like this, you have a pound sign, and then
a curly brace, and then the name of the variable.
And that's how you'd print, say, whatever the variable name is.
>> And then you can also concatenate strings.
Ruby makes it super easy with the plus sign.
You just have one string on the left plus a variable
or another string plus a string.
So if I print this out, it should just say Hello, my name is Robert.
I will be teaching you nokogiri in Ruby.
>> And let's just confirm that that is indeed the case-- ruby intro.
Hello, my name is Robert.
I will be teaching you nokogiri in Ruby.
>> Moving on, if else statements, it's a little different
from what you might be used to if you've been working in C.
You don't need the parentheses.
You don't need the curly braces.
And instead of else if, it's a concatenated elsif.
So in here, if I've declared x up here, as we can see, x is still 5.
So if x is less than 3, it'll put small.
If it's less than 7, medium, else large.
So 5 is a medium number.
And I end this block of code with end.
>> Here is my for loop.
And this syntax is also slightly different.
The 0 to five just essentially is declaring an arrays of 0 to 5.
So there's five slots in the array.
And then for each slot in that array, I will be incrementing i.
So this should print 0 to 5, or 0 to 4.
And this should print medium.
>> And I'll just blaze through.
You guys will have access to this code later on.
So you guys can run this yourselves.
>> So this is your basic while loop.
This will just be printing j, incrementing by 1 until we hit 5.
>> Super quick Ruby crash course on how to write a function.
Instead of, say, int factorial number, we just have def.
And essentially you're defining a function here.
This is going to be the name of the function,
and this is any variables that you want to pass into the function.
You can have if statements within.
You can return.
In this case, we're defining a recursively
implemented factorial function.
So we just call functions in Ruby like this.
>> So if I've defined this, I can call factorial, pass in 3,
and then 3 will be the number variable that I can use within the function.
And this to_s is just turning the return value of factorial into a string.
Otherwise this will throw an error saying oh, I
can't print a string-- because as you remember,
puts is put string-- because this factorial has returned a number.
So we can convert that to a string like such.
And conversely, you can also convert a string to an integer with to_i.
>> So making everything super simple, if I just comment this out, save
and run the factorial function.
We should be able to see that factorial of 3 is 6.
And that is indeed true.
>> So that's your crash course in Ruby.
And now that you know Ruby, we can go on to the basic Nokogiri scraping set up.
Essentially all you have to do is, in Ruby, require the libraries.
And for our purposes we'll be using the library OpenURI as well as Nokogiri.
And then what you do-- and it'll give you the syntax for this--
is you open the URL much as you would in a cURL request, which stands for C URL.
>> So you take the URL of the website in question.
You store it in a variable.
And then you can search through that variable for unique HTML tags using
the .css command.
And then you can output the content to wherever you want.
You can start in a database.
You can output in a file, or even just print it to the screen.
>> So we'll show you a basic scraper.
So up here you can see we have requiring nokogiri, require open-uri.
Your basic set up, let's call it document or doc,
equals Nokogiri::HTML open, which is the command provided to us by the OpenURI
library.
And we'll be searching, for those of you who might be living in the quad,
for bikes that are in Boston listed on the Boston Craigslist bike section
site.
>> So if you are unfamiliar with cURL, I'll just
show you real quick what cURL will do.
If I wanted to get all of the URL from the Craigslist site, if I type curl,
it just dumps all of the URL from the Craigslist bicycle site
onto my terminal.
That's not particularly useful because I don't
want to manually go through and find the thing I'm looking for.
But just so you can see that I'm actually
using the right code, if you look at the URL for Craigslist in bikes--
for some reason it's not found.
If you look at this page and you look at the URL,
this should be identical to the cURL request that I just send.
And indeed, that's what's being stored in the doc variable.
>> So when you go back to our code, we can then operate on this doc variable
by using .css.
So say I wanted to get all of the tags that are span.txt,
and all the a tags within that tag.
And why might we want to do this, I hear you cry?
>> If we Inspect Element, it gives you a breakdown of how the URL is structured.
If I scroll down through here, you can see
what each of these different elements represents.
So maybe I want to access this particular element.
So I'm using Chrome developer tools to Inspect Element.
I can see down here that this is an a tag within a span
tag with a class of txt.
>> So this gets to our first operation which
is doc.css span, which is the tag that I'm looking for within all this URL.
And then .txt operates much like CSS does when you're just writing CSS
in your HTML files by specifying a class.
So this particular operator will specify a span tag with class of txt.
And then if I leave a space, this will then go within that tag
and then find an a tag within that.
>> So if I just put this to the terminal, I should
be able to see essentially everything that is within this span of class txt.
So we'll give that a go.
ruby craigslist-scraper.
And indeed that gives us all of these tags of the various listings that
are on the Craigslist page.
>> So if we go back, we can turn this into something a little more useful.
Maybe we want just the links.
Because within this tag, I'll also have the hyperlink of the path
that this page goes to.
So if you look at this code here, what I'll do is instead of .css,
I can go at_css.
And this will just get the first element of all of those things.
So if I were to do that up in the code I just previously demonstrated,
instead of returning all of this, it would just
return the first one of those.
So that's how the at_css operator works.
>> So we want to store the path all of the first a tag.
And because a will give us a-- so we're still going to use .css.
But because this is going to give us back an entire array of tags,
we are going to access the first element.
So this is another way that you can access any particular element if you
have an array of elements that is returned,
because you can treat anything that .css returns as an array, essentially.
And then we're going to access the hypertext reference attribute of this.
>> So if you take a look, if you looked really close here,
if you just essentially look at the URL bar,
this is the path that you're going to be scraping.
So if we just run this again, and make sure we've saved it.
You can check at home.
This actually matches up with this link.
>> So why might we want to use this?
If you want to scrape the page and it has
a page of links like Craigslist does, you
might want to go then into each of those links
and then scrape the content of that, which
is exactly what we're going to do.
>> So once you have path as a variable, I no longer really
care about printing it out.
I just need to store it as a variable.
And then I can access another page the same way I access
doc in the first place.
Except with the URL, we're going to use string interpolation
like I was describing in Ruby earlier on to append
the path to the end of the root.
>> So what this is going to do is this is going to put on the path
that I scraped previously and then turn that
into a new item, whatever you want to call it-- first_listing, for example.
But I'm going to leave it on item for now,
because that is what I'm using here.
>> So say I wanted to get the description of the first posting in Craigslist.
So I would go down here.
I would click on Inspect Element again, because this is the description.
I'd go down here and see if I can find how I might
be able to search for this unique tag.
And in this case, it has an ID, which leads us
to our next way of searching for tags, which is with a hashtag.
>> So for classes, you can use the dot operator.
So .txt is specifying a class of txt, whereas the hash specifies an ID.
So in this case, the tag is section, and the ID is postingbody.
>> So this goes and finds the first-- because we're
using at_css-- this goes and finds the first element that
comes up with the tag of section and the ID of postingbody.
And then you can access the text element of that item returned with .text.
And then we can store that in the description.
>> So now that we have a variable description,
we might be able to do, say, file I/O. So file I/O in Ruby
is very similar to file I/O in C where we open a file.
We might write to it.
And then we'll close that file.
>> So here, we're just naming the file, some arbitrary variable.
We could also have just put this here.
We have a variable that we're storing the open file as with File.open.
And we're writing to this file, so we open it with the w operator.
And then we put string into the file with the .puts operator.
And then we put the variable that we want to write to the file within that.
And then we just close the file.
>> So if we go ahead and run this, this should produce a document
with description.txt which will have this description within it.
So if I run it-- no.
It's produced a text file with, hopefully, the same thing.
So there might have been a new posting that's come up while I've been talking.
And indeed it looks like there has been.
So if we go to this classic bike, 1962 to 1966, that seems to match.
And there you go.
>> So that's the most basic functionality of scraping.
We could have instead of just writing to this file,
we can add things to an array.
So if I declare three arrays, title, price, and description.
And we're operating on the doc item now.
We can go through and find all of the span.txt.
And remember, this returns an array of all the items that it finds.
And then in Ruby, you can just use .each to iterate through every item
of the array.
And then for each item, I'm just going to call it
a link, because that's essentially what it is.
>> So if I put each link.css dot a.hdrlnk, this is actually going to the link
and finding within that link another HTML element and corresponding class.
So if we remember what this was, the span.txt,
you can see- let me just go back real quick-- within span.txt
we have a lot of other classes.
So inside span.txt, we're looking for an a tag with a class hdrlnk.
So let me just find that for you guys real quick.
>> So you can see here, this is an a tag that's within the span of class txt
that has the class hdrlnk.
And that's indeed what we're trying to get.
>> So we're now trying to store all of those links inside the title.
And then we're going to print out each of those links.
No, sorry.
We're going to print out the price of each of those.
So let's run this really quick and see what it does.
>> So this just basically went through each of the links
in turn, accessed the tag in question, and then pulled out the price.
And it did that because after you have everything in the title,
we've just stored the title there.
We've just stored the link within the array title.
And in this for loop operation, where instead of going to a.hdrlnk,
we're looking for a span.price.
So if I can just really quickly find the price, if you inspect the element,
you'll see that it is a span with the class of price.
And that's essentially how we're getting the price there.
>> So that's the really basic case of scraping.
That's how you get all the elements on a page
that, say, you already know the URL of.
>> So if we want to get a little more in depth,
we can scrape pages within pages.
And for this example, I'll be outputting to a CSV file.
So I'm requiring csv up here because Ruby doesn't, inside itself,
have the functionality to just output CSV files.
So that's super simple.
Let me just go to the next.
We covered file I/O. So this is similar to how it is in C.
And before we move on to Kimono, I'll just show you really quick how
to scrape sites within sights.
>> So we already learned how to declare arrays in Ruby.
So I'm just declaring a bunch of arbitrary arrays
that I will be storing data within.
doc is operating the same way as it did in the previous file.
We're going in, finding each of the span.txt's.
We already know that.
That is the container within which each link has all of the data that we want.
>> So here what we're doing is for each link of span class txt, we're going in
and we're finding the a tag, finding the first element of that.
Remember, .css returns an array, so you can't just access it as is.
We're going to find the first element.
Even if it's an array of one item, you have to use this syntax,
and then pull out the href attribute.
>> So we did this earlier.
So this should look familiar.
And so now we have an array called paths of all of our links
that we're going to want to use.
So if we have this array of all of the paths that we want to use,
we can then create an item for each of those pages when we open that page.
So as we also saw on the syntax before, where
doing string interpolation with the path here, so the syntax is just for path.
And I could name this variable any arbitrary name.
>> This is the important one.
This is the array that you'll be accessing each element.
But when you say for path in paths, this means for each element in paths,
call it path, and use that.
This is essentially like when you do a for loop and you use int i.
So you can treat the path as the variable that's incrementing.
>> And then for each of those, go into each of those links.
Because we're storing it in item page, so we're creating a new page every time
we access it.
And then within that new page, find span.postingtitletext, span.price,
and then section#postingbody.
We already covered section#postingbody when we looked at the description.
>> So we can go see in the Craigslist post, if you're just looking at the title,
you can see it up here, span postingtitletext.
And that's why it's there.
And then for the price, you can access it with span class of price.
>> So we also perhaps might want to store the URL.
So we'll just run this again, store it in an array,
because if you're looking on Craigslist, you're
probably going to want a way to, if you see something that interests you,
go back to that site.
So you just want to store the URL for references sake.
>> This is just essentially another syntax for the for loop.
I could just do paths.each instead of for path in paths with index.
And this syntax is Ruby for-- path is what we did up here,
declaring a variable for each item.
And index behaves like the i in C for loops.
So you can keep track of what the index is.
>> So here is just a little convenient thing
for when you're running the scraper.
If you're scraping hundreds of pages, to make sure that it's not hanging,
it will just output, I'm accessing this page,
and making sure that it's still continuing.
But for our purposes, because there's a hundred items,
I'm going to access just three of them so that we don't run out of time here.
>> But before we get to that, I'm just going to show you really quick,
I will be outputting the title, price, description, and URL
of each of the links that I've scraped.
And then this is just the syntax for the CSV library.
You open a CSV.
This is what I'm going to call it.
Open it with write do.
And then CSV will be the file that you're inputting everything into.
This is just a sanity check for me to know that it's running.
And this is my sanity check to know that it's completed.
So I'm putting title into a row in the CSV, price, url, description,
all into rows in the CSV.
>> So if we go and run this now-- and I just
make sure that I've saved it-- instead of just outputting it to the terminal,
we should have a CSV file that's produced.
So here we can see the CSV file that's been produced.
This is the output of the scape that I just ran.
As you can see here, accessing page 0, 1, 2, 3.
These are the titles, prices, descriptions.
And if we look at this CSV file that we've generated,
you can see its outputted here.
This isn't Excel, so it's not formatted in rows and columns.
But you can imagine how it might be formatted.
>> CSV stands for comma separated values.
So you can imagine this might be a row.
And each comma would indicate a separate column.
Just a word of caution-- sometimes you're
scraping things with a lot of commas.
So if you're outputting it to a CSV file,
it might not output the way you might think.
>> So that's essentially all there is to scraping basic HTML
pages with Nokogiri.
>> So the internet being innovative as it has come up
with a more automated and GUI version, albeit less robust
version of scraping various websites.
And for our purposes I'll be demonstrating
a Chrome extension called Kimono.
And all you have to do is you navigate to the page that you want to scrape.
You click on a field of interest.
You calibrate the fields, because it will automatically
detect what it thinks you want to be scraping,
and then you just create an API.
>> So if we were to demonstrate it on Craigslist, it actually wouldn't work.
And this is what I was going back to saying about it not being as robust.
It has trouble creating the API.
But as a demonstration of what it would do,
if you install the Chrome extension, all you do is you click on it.
It Kimonofies the page, and then you click on the thing you want to script.
>> So if I were to click on that, it would highlight
what it thinks I want to be scraping off that page.
So maybe I call this listings.
This is how many items I have selected.
And I can just confirm or deny some of the other suggested listings
to get it to add to what will be scraped.
>> So now we can see there's a hundred items selected.
If I want to have another field that I also scrape which is related to this,
say I want to scrape the price as well, then I can do the same.
>> So here's a demonstration of how it's much less robust, because now it's
picking up the city instead of just the price that I want.
And now it's picked up 200 things.
You can go back and delete.
You can try again.
But no guarantees.
This is how this works sometimes.
As you see here, now it says 96 up here.
It's picked up most of the links that you want to scrape, but not
necessarily all of them.
>> Another useful tool of Kimono though is you can go to Advanced Features
here, go to Advanced, and it will show you
the breakdown of the unique way to access the HTML
tags that you want to scrape.
So for listings, if you look at here, if you access div p span span a,
you can actually just use this in your Nokogiri code,
where before we had span.txt to access each of the listings.
If I just want the text within the listings,
I could input div space p space span space span space a,
and it would achieve the same effect.
And for those of you who are interested in using regular expressions,
it happens to also give you the regular expression sort of string to input
to find the things you're trying to find.
>> So there's another cool feature of Kimono where you can paginate,
which is not only can I scrape the results of this page,
I can click on this little button here, Pagination,
specify the button that would take me to the next page,
and then it will just know that it can iterate to the next page,
and then scrape all of the-- as long as it's the same format of course--
scape all of those links as well.
>> So because Kimono doesn't want to work with Craigslist, what we've done
is I've Kimonofied the Harvard Crimson.
I've pulled out some of the sort of top featured articles, confirm here.
Say all of these.
I've compiled this API for you ahead of time.
But otherwise what you would do is you would just click Done.
Enter in your API details.
Set it to either automated or manual crawl.
So you could update your data every 15 minutes,
weekly, daily, whatever you want.
Name your API.
Create the API.
For your benefit, I've created the Crimson front page API already.
>> So you just create an account on Kimono, and it
will store all your APIs for you.
So essentially that's all your separate different scrapes.
>> So if we look here, this is the opinions links that I've collected.
These are the featured links that I've collected.
And these are the most read links that I've collected
from this most recent API scape.
>> So if you can see here, these would be the featured,
these would be the opinions, which in this example,
I've combined them all into one collection.
But if you just play around with it a little bit, you can split it up
and divide it up however you want to as long
as the formatting is slightly different.
>> Just to play around with this, the crawl set up, one of the downsides
is you can only crawl up to 25 pages at a time.
That's one of the limiting factors.
But here, if you set it to manual crawl, this
is how you can tell it to update your data.
And here you can see your crawl history of everything that you've crawled.
And you guys can go back, sign up, play around with all the different ways
that you can modify and use your data.
>> Kimono can be set up to scrape links within links.
And you would do so by first scraping a list of links,
and then using that API as a jump off point for another API
that you create the script.
But that's more complicated than what we're going to get into today.
>> So that's Kimono.
We'll talk about the pros and cons of Nokogiri and Kimono.
>> Nokogiri, it's really fast.
It's easy to test.
You can just puts anything to console, easy to configure.
You can decide exactly what you want to scrape and store.
There are no page limits.
I actually used it to scrape like 1800 South African school websites
for emails for an internship that I did.
>> So that's possible, though best practice would be to split up the script.
Because if it fails, then you don't get anything.
But if you do a hundred, maybe 200 pages at a time,
then you have some chance of at least getting it piecemeal, especially
if you have bad internet.
>> Unfortunately it can only scrape HTML.
So if you have dynamically loaded pages--
and I'll show you an example like Kayak in a second--
Nokogiri unfortunately cannot scrape that.
>> But Kimono is also easy to use.
As you saw, it's essentially a point and click.
It can scrape JavaScript.
Unfortunately, there's a maximum to how many pages you can scrape.
Sometimes it's a little hard to configure.
It gets confused.
But it's definitely something to consider
if you're not trying to have a super robust maintainable scrape.
If you just want to get everything off of a page quickly,
then Kimono is a really good tool to use.
And as I mentioned before, there's the advanced feature of Kimono
that shows you how to access the unique HTML
element, which is super useful even if you are working in Nokogiri.
>> So if we go to the Kayak site, for example, you can see there is--
or maybe you can't see.
But if I show you the URL for Kayak, this actually is just the source URL.
This is the URL prior to being modified by whatever JavaScript scripts
that they have going on.
And it's going to look different from inspecting the element.
>> So if you go through and you match up the Inspect Element
code to the source code, it's actually going to be different.
And this is essentially why Nokogiri can't scrape dynamically loaded sites.
Because Nokogiri is scraping the source URL,
whereas Kimono is actually scraping what you're essentially
seeing in Select Element.
>> So if I go through and I try and Kimonofy Kayak,
I can actually go through and select the price.
It's a little harder, and in this case, it's
actually seeing this price as different from these.
So whereas you can configure-- or if this weren't dynamically loaded,
you could configure Nokogiri to get all of these.
>> Because the formatting is slightly different for this listing
as it is compared to the rest of them, and you can see here
it's actually gone and selected all the flight prices.
Maybe I want to select time of flight as well.
And I can go through and sort of configure that.
I don't want that.
I just want the next flight's time.
And then after a couple of these going through, it gets the picture.
So Kimono's pretty smart.
It's just not quite as robust.
>> There are some other alternatives that you can use.
And I'll show you them here.
If you are more comfortable in Python instead of Ruby maybe,
there is a library called Beautiful Soup.
You can use that.
It's very similar to Nokogiri.
It has a few more features.
You can find an HTML tag and then move up or move sideways.
>> There's PyQt.
This can actually scrape dynamic sites, because it's sort of
is a WebKit that pretends to be a browser without there actually
being a browser.
So it would wait for all the JavaScript to load first, and then
go in and try and scrape the site.
>> If you want to stick with Ruby, you can go one level up from Nokogiri.
You can use Capybara with a Poltergeist wrapper.
And this can actually essentially do the same thing
as PyQt, which is it is a WebKit.
It waits for the JavaScript to load first.
If you fiddle around with it enough, you can even get it to click on things.
>> So if there's a link that isn't a classic href where
the path is easily accessible, and it's some JavaScript thing that detects
a click, you can actually do that.
The more popular library to simulate a user
is in JavaScript, which is PhantomJS.
This can obviously scrape dynamic sites because this is essentially
pretending to be Chrome without the user interface.
>> And then, of course the most robust, but slowest option,
is a Selenium browser automation.
And unfortunately, you're not going to be
able to do this within your CS50 IDE.
Because essentially what it does is it boots up your Chrome,
Firefox, whatever browser that you want to use,
and it tracks maybe your mouse movement, whatever you type in,
and it just sort of automates this process.
So it was developed as a sort of website automation testing tool.
But a lot of people use Selenium to scrape websites
that they otherwise have a lot of difficulty scraping
with some of these other, faster tools.
>> So that's all I've got for web scraping.
Have fun.
>> AUDIENCE: Question.
>> ROBERT KRABEK: Yes.
>> AUDIENCE: Is there a mechanism to hash the website so you could basically
go through it later on.
ROBERT KRABEK: Yeah.
So we put the, in our example, for both of them,
we put the entire website into doc.
And so you could actually just take the variable doc and write it to a file.
So if I wanted to, I could write it out as an HTML file,
and then instead of using OpenURI and a cURL request,
then I could just open up doc HTML and then search for that.
>> AUDIENCE: But can you preserve the sort of online experience
while you do offline.
For example. when you're flying for several hours,
I want to basically archive the whole website. [INAUDIBLE]
>> ROBERT KRABEK: Yeah, that's exactly-- so literally what this is doing
is it's taking everything that would be at this URL.
So if we ran cURL, it's taking all of this HTML,
and it's storing it inside the variable doc.
So then you can do whatever you want to do with doc.
You can output it to a file.
AUDIENCE: But it's not linked up.
It's not dynamic.
It's not recursive, right?
You see what I mean?
I'm trying to basically sort of a hash the whole website on my hard drive
so that I could basically do it for several hours without internet.
>> ROBERT KRABEK: Right.
So if I had-- so where's my file I/O?
So this is the file I/O. So say instead of this, I call this craigslist.html.
I'd open that up.
I'd puts doc into it.
I close the file.
And then just because the CS50 IDE is on the cloud, that's whatever.
I can go here.
I can download the file.
And then that would be on my hard drive.
So you can do it that way.
Or if you're at home, not using the CS50 IDE, like Sublime or something,
this is even easier, because this is all available locally,
not tied to the internet.
>> AUDIENCE: I see.
This is for one particular problem.
Can you do it recursively so that you go several layers deep kind of thing?
>> ROBERT KRABEK: I can download folders as well, if that's what you're asking.
AUDIENCE: Yeah.
>> ROBERT KRABEK: Cool.