Complete Python Tutorial: Build a Web Scraper with BeautifulSoup

TL;DR: Build a Python web scraper using BeautifulSoup to extract porcelain home decor product names, prices, and descriptions from an online catalog. You’ll fetch HTML with Requests, parse it with BeautifulSoup, and export the data to CSV—no images needed.

Step 1: Set Up Your Environment

Install Python 3.9+ and run pip install requests beautifulsoup4 lxml pandas. Create a new folder named porcelain_scraper and inside it, a file scraper.py. For this guide, we’ll scrape a mock porcelain decor store page (e.g., https://example-porcelain.com/products) that lists vases, figurines, and dinnerware in a simple HTML table.

If you want to dig deeper, check out our guide on Luxury Porcelain Home Decor: Elevate Your Space.

Step 2: Fetch the Page with Requests

Use requests.get() to download the HTML. Add a User-Agent header to avoid being blocked. Wrap the request in a try/except to handle connection errors. Example: response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}). Always check response.status_code == 200 before proceeding.

Step 3: Parse HTML with BeautifulSoup

Create a BeautifulSoup object: soup = BeautifulSoup(response.text, 'lxml'). Inspect the page structure by printing soup.prettify() in your terminal. For porcelain products, you’ll typically find each item inside a div with class product-card or a tr in a table. Use soup.find_all('div', class_='product-card') to isolate each product block.

Step 4: Extract Product Details

Within each block, locate the name (e.g., h3 tag), price (span with class price), and material description (p with class desc). For example:
name = item.find('h3').text.strip()
price = item.find('span', class_='price').text.strip()
desc = item.find('p', class_='desc').text.strip()
Porcelain-specific fields to look for: “hand-painted,” “bone china,” “glaze finish,” and “diameter in cm.” Store these in a dictionary.

Step 5: Handle Pagination and Edge Cases

Many porcelain catalogs have multiple pages. Find the “next” button link via soup.find('a', text='Next') and loop until it’s missing. Also, use try/except for missing fields—some items may lack a description. Replace empty strings with 'N/A'. Respect robots.txt and add a time.sleep(1) between requests to avoid overloading the server.

Step 6: Export to CSV

Collect all product dicts into a list, then use pandas: pd.DataFrame(products).to_csv('porcelain_decor.csv', index=False). Open the CSV in Excel to verify. You’ll see columns: name, price, description, and material. For advanced use, add a column for “category” (vase, figurine, plate) by checking keywords in the name.

Step 7: Run and Debug

Execute python scraper.py. If you get an empty list, print the first 500 characters of soup.text</code