Is this possible?

Hey guys,

I have a question for those of you more accustomed to this type of stuff. All I've ever done is create websites based off the Wordpress CMS, however, I have an idea for a new website that will be more complicated and thus will require a more custom and hands-on approach.

To start, I'd need to build some time of script or crawler that would be able to crawl a site like Amazon.com for pretty much all of it's products it has available and retrieve data - would something like this be possible / feasible (or legal?). Note that I'm not asking or looking to hire anyone, I'm just talking hypothetically.

Thanks for any input!

 

Hypothetically if you were to crawl and scape the content of every product on Amazon you would be violating their terms of service and copyright. You don't have the legal right to use their content. However, if you are scraping just basic manufacturer information and pictures and your site is not competing with Amazon I'm sure you could get away with it. But the real matter at hand is the fact that there are hundreds of thousands of products on the site if not millions. Creating such a scraper would be rather difficult.

Elaborate what you are trying to do. There may be an easier way to do this.

 

Thanks for the reply.

I figured that would be the answer I would receive. It doesn't even have to be Amazon, maybe I could utilize something like Google Shopping or something to that extent - or maybe even just grab the top 100 (most popular) products from each category Amazon has available.

What I would like to eventually be able to do is to build a database of products, things people buy on sites like Amazon, and use this database to crawl a list of sites (or maybe utilize Google Shopping) to find the lowest price at the current date. I figure once I have the product, getting a script to find the lowest price shouldn't be too difficult, but how to initially build that list of products (and even potentially have it change dynamically) is the problem I'm trying to solve right now.

 

There are tons of web sites that already do this.

 

Can you give some examples, please? I haven't described my whole "plan", per say, but it is very likely I'm envisioning something that already exists, though I can't seem to find it through searching.

 

There are some open source crawler scripts available you can choose one of them and can use it.

 

Thanks, I come from a primarily web design background, so I don't know too much about the technical side of websites, never really crossed my mind that such things are actually readily available.

Do you (or anyone) have any preferences in these crawler scripts? Anything that could make it easy (easier) to accomplish what I was talking about above?

Thanks again for the replies guys, I appreciate your time!

 

So would anyone have any preferences / insights on a good crawling script / API out there that I should look at, just looking for input from people who know what they're talking about.

Thanks!