Web Scraping Using Python

Web scraping is data scraping used for extracting data from websites. Web scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page. Therefore, web crawling is a main component of web scraping, to fetch pages for later processing. An example would be to find and copy names and telephone numbers, or companies and their URLs, or e-mail addresses to a list.

To extract data using web scraping with python, you need to follow these basic steps:

  1. Find the URL that you want to scrape.

Python Libraries Used For Web Scraping:

  1. Requests is a HTTP library for the Python programming language. The goal of the project is to make HTTP requests simpler and more human-friendly. Allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your POST data.

Import the required python libraries:

import requests
from bs4 import BeautifulSoup
import pandas as pd

Create Empty variables to store scraped data:

Text = []

Now enter the URL from where you want the data. Requests library is used to make html requests to the server.

res = requests.get('https://en.wikipedia.org/wiki/Web_scraping')
soup = BeautifulSoup(res.text, 'html. Parser')
soup.select('mw-headline')
for i in soup.select('.mw-headline'):
print(i.text)
Text.append(i.text)

After extracting the data, you might want to store it in a format. This format varies depending on your requirement. For this, we will store the extracted data in a CSV format.

df = pd.DataFrame({'Text':Text}) 
df.to_csv('Web_Scrap.csv', index=False, encoding='utf-8')

By running the above code. Here is the snapshot of the csv file generated after running the code.

Hence, we get to learn how to scrap data from the internet and format it for further analysis. You can see whole code here:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store