Web Scraping Amazon Product Data using ChatGPT and Python #openai #chatgpt

If you ask chat GPT to scrape amazon.com It will show this policy violation error If you write code and try to scrape Amazon you will get this error Amazon Will not let you scrape the data in this Video we will solve both the problems And scrape product data from Amazon Go to amazon.com You will see lots of products listed Here Choose which product category you want To scrape For this video I am selecting computers Here you can see some subcategories Listed let’s select monitors you can Choose any other category the process Will be the same Now here it has two thousand items This is the first page and there are 88 Pages total we will scrape all of them We want to scrape the image URL title Rating and price for each of the items Step number one Just right click here and select inspect In the developer tool find the div Element for this item look at this every Div represents one item all div have the Same class name right click on the div Element and select edit HTML now copy The class name for this did Paste it in some text file for now Right click on the item image and select Inspect Then find the image tag and right-click

Edit as HTML copy the class name for This image tag Paste it here now right click on the Title and select inspect Then find the span tag and our title is Here as a text Right-click and edit as HTML copy the Class name of this span tag paste it Here Now right click on the rating and select Inspect Then find the span tag and our rating is Here as a text Right-click and edit as HTML Copy the class name of this span tag Paste it here Now right click on the price and select Inspect then find the span tag and price Of the item is here as a text Right-click and edit as HTML copy the Class name of this span tag paste it Here Step number two We want to scrape data from this page Just press Ctrl s And save this as index.html this is the Source code of this page with all the Product data Step number three Let’s write our prompt for chat GPT Use python read index.html file and Parse it with beautiful soup Find all the div with this class name Every div will represent data of one

Item this will create a list of div For each div in the above list Find the image tag with this class name And save its URL Find the span tag with this class name And save text value into the variable Title Find the span tag with this class name And save text value into the variable Rating find the span tag with this class Name and save text value into the Variable price Now check on Amazon some products won’t Have a price or rating in order to avoid Error we need to use try and accept Just add a try keyword here We also want to export our data into a Json file open data.json file and right Link title rating price And make the value of each variable an Empty string if you can’t find it Our prompt is ready Copy this Go to chat GPT and paste it now chat GPT Is generating code based on the prompt We have given we didn’t ask the chat GPT To scrape any website We just ask it to write some code for us So we will not get any policy violation Error our code is ready Copy this code and paste it inside the App.py file Add here Encoding equals utf-8

Save this file and run with command Python app.py and here we have a Data.json file as a result Let’s open this file and we successfully Scrape this data from page 1. let’s Scrape page 2. just go to page 2. press Control plus s Save this file as page2.html An app.py file Change file name to page2.html And change this to data2.json Save this file and run this And here we have a data2.json file as a Result Let’s open this file and we successfully Scrape this data from page 2. Let’s scrape page 3. you know the Process now just save this page and run The python script And here we have a data3.json file as a Result Let’s open this file and we successfully Scrape this data from page 3. this is How you can scrape as many pages you Want We are scraping data locally from our HTML file so no error from Amazon you Can download hundreds of pages and then Put all the python code in a for Loop to Automate the process I hope you understand the process and Repeat this on your own thank you for Watching and see you in the next video