要爬取的网站 http://quotes.toscrape.com/
**爬取名言，作者，标签。***
她们的Html为，通过beautiful库的html.parser解析，通过id,class选择器，提取我们需要的东西。
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
“It is our choices, Harry, that show what we truly are, far more than our abilities.”
Tags:
change
deep-thoughts
thinking
world
Tags:
abilities
choices
J.K. Rowling
相关的code如下
import requests
import os
from bs4 import BeautifulSoup
def get_html(url):
try:
header={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331', }
ht=requests.get(url,headers=header,verify=True)
ht.raise_for_status
ht.encoding=ht.apparent_encoding
return ht
except Exception as e:
print("error:",e)
def writeHtml(url):
text=get_html(url).content
path="C:\Users\Administrator\Desktop\python\baidu.html"
with open(path,"wb") as f:
f.write(text)
print("success")
def getInfor(url):
finalSay=[]
html=get_html(url).text
soup=BeautifulSoup(html,"html.parser")
print(" the saying is:")
say_list=soup.select("span.text")
print("-------名言--------------")
print("the length=",len(say_list))
# print(say_list)
for elem in say_list:
te=elem.text
finalSay.append(te);
print(finalSay)
print("-------标签--------------")
finalTag=[]
tag_list=soup.select("div.tags")
print("the length of tag=",len(tag_list))
#print(tag_list)
for tag in tag_list:
a_tag=tag.select("a.tag")
#print(a_tag)
# for elem in a_tag:
# print(elem.text)
tag_list=[elem.text for elem in a_tag]
#print(tag_list)
finalTag.append(tag_list)
print(finalTag)
print("-------作者--------------")
finalAuthor=[]
author_list=soup.select("small.author")
print("the length of authoer=",len(author_list))
for elem in author_list:
finalAuthor.append(elem.text)
print(finalAuthor)
for i in range(len(author_list)):
print("the saying:",finalSay[i],"t","the author:",finalAuthor[i],"t","the tag:",finalTag[i])
结果如下
the saying: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” the author: Albert Einstein the tag: ['change', 'deep-thoughts', 'thinking', 'world'] the saying: “It is our choices, Harry, that show what we truly are, far more than our abilities.” the author: J.K. Rowling the tag: ['abilities', 'choices'] the saying: “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.” the author: Albert Einstein the tag: ['inspirational', 'life', 'live', 'miracle', 'miracles'] the saying: “The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.” the author: Jane Austen the tag: ['aliteracy', 'books', 'classic', 'humor'] the saying: “Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.” the author: Marilyn Monroe the tag: ['be-yourself', 'inspirational'] the saying: “Try not to become a man of success. Rather become a man of value.” the author: Albert Einstein the tag: ['adulthood', 'success', 'value'] the saying: “It is better to be hated for what you are than to be loved for what you are not.” the author: André Gide the tag: ['life', 'love'] the saying: “I have not failed. I've just found 10,000 ways that won't work.” the author: Thomas A. Edison the tag: ['edison', 'failure', 'inspirational', 'paraphrased'] the saying: “A woman is like a tea bag; you never know how strong it is until it's in hot water.” the author: Eleanor Roosevelt the tag: ['misattributed-eleanor-roosevelt'] the saying: “A day without sunshine is like, you know, night.” the author: Steve Martin the tag: ['humor', 'obvious', 'simile']借助bing的翻译，实现英语翻译中文
完整 https://cn.bing.com/dict/search?q=snow
其翻译的信息主要在
n. v. adj. 网络 书；书籍；著作；部 书籍的；书本上的；账面上的
对应的正则表达式为
re0=r'(.*?)' re1=r'(.*?)'
相关code
主要是正则表达式的使用，已经写文件操作。
def getDictionary():
word=input("亲输入要翻译的词语:")
url="https://cn.bing.com/dict/search?q="
hurl=url+word
trant=[]
html=get_html(hurl).text
re0=r'(.*?)'
flag_list=[]
nav_list=re.findall(re0,html)
#print(hurl)
#print(html)
if len(nav_list)==0 and len(nav_list[0])<1:
return
for elem in nav_list:
flag_list.append(elem[1])
print(flag_list)
re1=r'(.*?)'
tran_list=re.findall(re1,html)
#print(tran_list)
for i in range(len(nav_list)):
tra="t".join([flag_list[i],tran_list[i]])
print(tra)
trant.append(tra)
#trant.append("n")
return trant
def writeTet(t):
path="E:\infor.txt"
with open(path,"a+") as f:
for txt in t:
f.write(txt)
f.write("n")
print("success")
结果为
亲输入要翻译的词语:Python ['n.', '网络'] n. 蟒；蚺蛇 网络 蟒蛇；巨蟒；派森 success
参考文章
beautifulsoup之CSS选择器
Beautiful Soup documentation
Python 文件I/O
python对文件的操作读写追加等演示
