目前在Coursera上自學Python網路爬蟲。
寫作業的時候碰到一個題目,就是要根據使用者輸入的position,
搜尋網頁中使用者指定position的網址,進入該網址後,
再搜尋下一頁面中該指定position的網址,如此重複counter次
原始題目敘述為:
In this assignment you will write a Python program that expands on
http://www.py4e.com/code3/urllinks.py. The program will use urllib to read
the HTML from the data files below, extract the href= vaues from the anchor
tags, scan for a tag that is in a particular position relative to the first
name in the list, follow that link and repeat the process a number of times
and report the last name you find.
下面是我目前寫好的部分,但只能列印出第一層指定位置的網址,不知道該怎麼
依照指定的counter重複進入該網址再列印下幾層的網址,請大家協助解惑了
import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = 'http://py4e-data.dr-chuck.net/known_by_Fikret.html'
counter = input ('Enter counter: ')
position = input ('Enter position: ' )
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')
# Retrieve all of the anchor tags
tags = soup('a')
lst = []
for tag in tags:
link = tag.get('href', None)
lst.append(link)
print(lst[int(position)-1])