Use Beautiful Soup to write a script that takes an HTML file and writes a version of it with nicer indentation to an output HTML file. Also, if there is no <title> tag in the input HTML file, the script should print a warning.
The input HTML filename should be expected as the first command line argument, and the filename to which the prettified HTML is written is the second command line argument.
For example, if the in.html contains
<!doctype html><html><head></head><body>
<h1>MCS 275 HTML file</h1></body></html>Then running
python3 prettify.py in.html out.htmlshould print a message
Warning: This HTML file has no <title>.and should write the following to out.html:
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<h1>
MCS 275 HTML file
</h1>
</body>
</html># MCS 275 Worksheet 15 Problem 1
# J Vaccaro
# I completed this work myself, in accordance with the syllabus.
from bs4 import BeautifulSoup
import sys
# Create a beautiful soup from the filename in the first command line arg
with open(sys.argv[1],"r") as infile:
soup = BeautifulSoup(infile,"html.parser")
# Check whether the soup has a title in the head section, and print a message
if soup.head.title == None:
print("Warning: No html title. Prettifying anyways!")
else:
print("Prettifying html with title...",soup.head.title.string)
# Write out the prettified soup to the filename in the 2nd command line arg
with open(sys.argv[2],"wt") as outfile:
outfile.write(soup.prettify())
It's also possible to use this code with a live website, as in the following code:
from bs4 import BeautifulSoup
from urllib.request import urlopen
with urlopen("https://example.com/") as response:
soup = BeautifulSoup(response, "html.parser")
print(soup.prettify())
Consider this web page for a graduate complex analysis class that was taught at UIC in 2016:
One section of the page lists weekly homework. Each homework assignment has a number, a title, and a list of problems from various sections of the textbook. Write a scraper that downloads this course web site's HTML, parses it with Beautiful Soup, and creates one dictionary for each homework assignment having the following format
{
"number": 10,
"title": "Harmonic functions",
"problems": "Sec 4.6.2(p166): 1,2\nSec 4.6.4(p171): 1,2,3,4"
}It should then put these dictionaries into a list and save the list to a JSON file called math535spring2016homework.json.
Note: If you finish this problem early, you might find it fun to watch this animation of the UIC logo distortion that appears on the Math 535 course web page, and see if you can figure out what's going on.
# It's good practice to save the html locally during development.
# Here's a short script that saves the html as 'math535.html'
from urllib.request import urlopen
from bs4 import BeautifulSoup
with urlopen("https://www.dumas.io/teaching/2016/spring/math535/") as response:
soup = BeautifulSoup(response,"html.parser")
with open("math535.html", "wt") as fout:
fout.write(soup.prettify())
# MCS 275 Worksheet 15 Problem 2
# J Vaccaro
# I completed this work myself, in accordance with the syllabus.
from urllib.request import urlopen
from bs4 import BeautifulSoup
# First, create a beautiful soup, either from the url or a local copy
## --- Comment this out for the final version ------------
# with open("math535.html", "rt") as infile:
# soup = BeautifulSoup(infile, "html.parser")
## -------------------------------------------------------
# --- Comment this out during development ----------------
with urlopen("https://www.dumas.io/teaching/2016/spring/math535/") as response:
soup = BeautifulSoup(response,"html.parser")
# --------------------------------------------------------
# We want to make a list of dictionaries, so start with an empty list
hw_data = []
# The relevant section is in an unordered list inside the "homework" div.
hw_ul_tag = soup.find("div",id="homework").ul
# Iterate through each bullet item in the homeworks list
for hw in hw_ul_tag.find_all("li"):
# Not every 535 homework assignment fits the expected format.
# If there's an issue parsing, just skip that assignment and continue.
# A sweeping try/except is not always recommended, but neither
# is parsing html.
try:
# The problems are inside the contents, on lines without other tags.
problems = ""
for prob in hw.contents:
# Convert to string and strip out starting/ending white space
prob = str(prob).strip()
#If the content line has a tag or is whitespace, then skip
if "<" in prob or prob == "":
continue
#Otherwise, concatenate to problems
else:
problems += "\n" + prob
# The assignment number and title are all inside the "b" tag
heading = hw.b.string.strip()
words = heading.split()
number = int(words[1])
title = " ".join(words[7:])
# Create a dictionary with the fields we collected
d = {"number":number, "title":title, "problems":problems}
# Append the dictionary to the list of dictionaries
hw_data.append(d)
except Exception as e:
# Skip the homework assignments that don't have the expected format,
# but print the error message.
print(e)
continue
# Dump out the list-of-dictionaries into a json file.
import json
with open("math535spring2016homework.json", "wt") as outf:
json.dump(hw_data, outf)
Here is a link to an HTML file:
If you open it in a browser, you won't see anything. The document contains nothing but <span> tags, and no text. Some of the <span> tags are nested inside other <span> tags. How deeply are they nested?
Every <span> tag in this file has an id attribute. There is exactly one <span> that has greater depth in the the DOM tree than any other. What is its id attribute?
Write a Python script to load the HTML file with Beautiful Soup and tranverse the DOM to answer these questions.
# It's good practice to save the html locally during development.
# Here's a short script that saves the html as 'capture.html'
from urllib.request import urlopen
from bs4 import BeautifulSoup
with urlopen("https://www.dumas.io/teaching/2021/spring/mcs275/data/capture.html") as response:
soup = BeautifulSoup(response,"html.parser")
with open("capture.html", "wt") as fout:
fout.write(soup.prettify())
# MCS 275 Worksheet 15 Problem 3
# J Vaccaro
# I completed this work myself in accordance with the syllabus.
from urllib.request import urlopen
from bs4 import BeautifulSoup
def span_tag_depth(tag):
"""Recursive function for recursing through the span tree and counting the maximum depth.
Returns the depth."""
# Maintain a list of the children's maximum depths
max_span_depth = 0
# Iterate through the child span tags WITHOUT RECURSING
# i.e. only immediate children, not ancestors
for t in tag.find_all("span", recursive=False):
depth = span_tag_depth(t)
# If the child's depth is the deepest so far, then replace.
if depth>max_span_depth:
max_span_depth = depth
# Pass up the maximum depth
return 1 + max_span_depth
def span_tag_depth_id(tag):
"""Recursive function for recursing through the span tree and counting the maximum depth.
Returns the depth and the leaf's span id."""
# Set the default depth
max_span_depth = 0
# If the current tag is <span> and has an id, set it as the default id
# Then, we will "pass up" the leaf id from the longest branch
if tag.name == "span" and tag.has_attr("id"):
max_span_id = tag["id"]
# Iterate through the child span tags WITHOUT RECURSING
# i.e. only immediate children, not ancestors
for t in tag.find_all("span", recursive=False):
# Recurse through t's children for the max branch length and leaf id
t_depth,t_id = span_tag_depth_id(t)
# If t has the deepest depth so far, replace the max depth/id.
if t_depth>max_span_depth:
max_span_depth = t_depth
max_span_id = t_id # leaf id
# Return the augmented max_depth and the id of the leaf.
return 1+max_span_depth,max_span_id
# Create a beautiful soup, either from the url or a local copy
# --- Comment this out for the final version ------------
with open("capture.html", "rt") as infile:
soup = BeautifulSoup(infile, "html.parser")
# -------------------------------------------------------
# # --- Comment this out during development ---------------
# with urlopen("https://www.dumas.io/teaching/2021/spring/mcs275/data/capture.html") as response:
# soup = BeautifulSoup(response,"html.parser")
# # -------------------------------------------------------
print("Maximum depth:",span_tag_depth(soup.span))
depth,span_id = span_tag_depth_id(soup.span)
print("Maximum depth:",depth,"Leaf id:",span_id)
# Here's a bonus solution from Melanie Wertz!
# It uses the indentation properties of prettify()
# and checks for the line with the most whitespace before a tag.
from bs4 import BeautifulSoup
# I already prettified capture.html to make prettycapture.html
with open("capture.html") as fobj:
max_depth = 0
max_id = 0
for line in fobj:
if "id" in line:
opening_whitespace = line.index("<")
opening_slice = line[0:opening_whitespace]
if opening_slice.count(" ") > max_depth:
max_depth = opening_slice.count(" ")
id_start = line.index("=")
id_end = line.index(">")
max_id = line[id_start+1:id_end]
print(max_id)
If you finish the exercises above, this would be a good time to complete your MCS 275 course evaluation. This anonymous survey helps UIC improve its courses and teaching. Every student enrolled in the course received a link to complete such a survey by email to their uic.edu account.
Evaluations for 15-week Spring 2022 courses are due by 11:55pm on Sunday, May 1.