使用Find_All函数返回意外结果集

网站加载了

Javascript

requests

Javascript

selenium

requests_html

现在，我们在表上确实还有另一个选项，可以跟踪从哪里渲染数据。我能够找到XHR请求，该请求用于从中检索数据

back-end

API

您可以

XHR

通过打开Developer-Tools来获取请求，然后检查Network并检查

XHR/JS

根据调用类型发出的请求，例如

fetch

您可以在下面实现您的目标：

请注意以下几点：

1. website

   持有3068

   item

2. 我增加了每页要

   120

   使用的项目

   parameter

   "pageSize": "120"

3. 所以

   3068 / 120

   =

   26

   ，这意味着26页每页120个项目。
4. 因此，您将需要循环使用

   (0, 3120, 120)

   哪种方式

   0 > 120 > 240

   ，依此类推，使用参数

   "beginIndex": "0"

   ，您将在

   for

   循环下递增。

由于您没有为我们提供最终目标，因此您可以在下面实现您的目标。但我相信您的目标是

name

price

import requestsfrom bs4 import BeautifulSoupparams = {    "langId": "44",    "storeId": "10151",    "catalogId": "10241",    "categoryId": "12192",    "parent_category_rn": "",    "top_category": "12192",    "pageSize": "120",    "orderBy": "FAVOURITES_FIRST",    "searchTerm": "",    "catSeeAll": "true",    "beginIndex": "0",    "categoryFacetId1": "12192",    "categoryFacetId2": "",    "requesttype": "ajax"}def main(url):    with requests.Session() as req:        r = req.post(url, params=params).json()        for item in r[5]['productLists']: for nest in item['products']:     soup = BeautifulSoup(nest['result'], 'html.parser')     target = soup.find("div", class_="productNameAndPromotions")     name = target.h3.a.text.strip()     url = target.h3.a.get("href")     img = f"https"+target.h3.a.img.get("src")     price = soup.find(         "p", class_="pricePerUnit").get_text(strip=True)     print(name, price, img, url)main("https://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/drinks/AjaxApplyFilterSearchResultView")

名称和价格的简要输出：

Sainsbury's British Semi Skimmed Milk 2.27L (4 pint) £1.10/unitSainsbury's British Semi Skimmed Milk 1.13L (2 pint) 80p/unitSainsbury's British Whole Milk 2.27L (4 pint) £1.10/unitCravendale Purefilter Semi Skimmed Milk 2L £1.90/unitSainsbury's British Skimmed Milk 2.27L (4 pint) £1.10/unitSainsbury's British Semi Skimmed Milk, SO Organic 2.27L (4 pint) £1.80/unitSainsbury's Sparkling Water, Basics 2L 25p/unitSainsbury's British Skimmed Milk 1.13L (2 pint) 80p/unitSainsbury's 100% Pure Squeezed Smooth Orange Juice, Not From Concentrate 1L £1.30/unitSainsbury's Water, Basics 2L 25p/unitSainsbury's British Whole Milk 1.13L (2 pint) 80p/unitSainsbury's Smooth Pure Orange Juice 1L 95p/unitPepsi Max 2L £1.90/unitSainsbury's Caledonian Still Water 4x2L £1.50/unitHighland Spring Still Water 12x500ml £3.00/unitSainsbury's 100% Pressed Apple Juice, Not From Concentrate 1L £1.30/unitSainsbury's British Whole Milk, SO Organic 2.27L (4 Pint) £1.80/unitLactofree Semi Skimmed Lactose Free Fresh Dairy Drink 1L £1.50/unitDiet Coke 8x330ml £4.00/unitAlpro Roasted Almond Unsweetened UHT Drink 1L £1.80/unitRobinsons Orange Squash No Added Sugar 1L £1.65/unitSainsbury's Soda Water 1L 60p/unitSainsbury's Caledonian Sparkling Water 4x2L £1.60/unitTropicana Smooth Orange Juice 950ml £2.45/unitSainsbury's Diet Indian Tonic Water 1L 60p/unitSainsbury's Pure Apple Juice 1L 95p/unitRobinsons Apple & Blackcurrant Squash No Added Sugar 1L £1.65/unitSainsbury's Sparkling Flavoured Water, Lemon & Lime 1L 50p/unitSainsbury's Conegliano Prosecco, Taste the Difference 75cl £8.00/unitSainsbury's Unsweetened Soya Drink 1L 90p/unitSainsbury's British Semi Skimmed Milk, SO Organic 1.13L (2 pint) £1.15/unitSainsbury's Caledonian Sparkling Water 6x500ml £1.50/unitSainsbury's Apple & Blackcurrant Squash, No Added Sugar 1.5L £1.00/unitHighland Spring Still Water 6x1.5L £3.00/unitAlpro Roasted Almond Unsweetened Fresh Drink 1L £1.85/unitSainsbury's Semi Skimmed Long Life Milk 1L 90p/unitTropicana Smooth Orange Juice 1.6L £2.50/unitSainsbury's 100% Pure Squeezed Orange Juice with Bits, Not From Concentrate 1L £1.30/unitCravendale Purefilter Semi Skimmed Milk 1L £1.15/unitSainsbury's Caledonian Still Water Sports Cap 6x500ml £1.50/unitSainsbury's Double Strength Orange Squash, No Added Sugar 1.5L £1.00/unitDiet Coke 18x330ml £7.00/unitSainsbury's Indian Tonic Water 1L 60p/unitSainsbury's Pure Orange Juice 1L 85p/unitSainsbury's Pure Apple Juice 6x200ml £1.50/unitBuxton Still Natural Mineral Water 8x500ml £2.00/unitSainsbury's Whole Long Life Milk 1L £1.05/unitCravendale Purefilter Skimmed Milk 2L £1.90/unitSainsbury's Sparkling Flavoured Water, Blackcurrant & Cherry 1L 50p/unitInnocent Smooth Orange Juice 1.35L £3.00/unitAlpro Original Soya Fresh Drink 1L £1.55/unitSainsbury's Still Flavoured Water, Strawberry & Kiwi 1L 50p/unitSainsbury's British Filtered Semi Skimmed Milk 2L £1.35/unitSainsbury's Sparkling Flavoured Water, Mango & Passionfruit 1L 50p/unitSainsbury's Caledonian Still Water 5L £1.10/unitMcGuigan Estate Merlot 75cl £5.10/unitSchweppes Slimline Tonic Water 1L £1.50/unitPG tips Pyramid Tea Bags x240 696g £4.50/unitSainsbury's Sparkling Flavoured Water, Strawberry & Kiwi 1L 50p/unitSainsbury's Caledonian Sparkling Water 2L 55p/unitSainsbury's Sweetened Soya Drink 1L 90p/unitSainsbury's 100% Pure Squeezed Smooth Orange Juice, Not From Concentrate 1.75L £2.10/unitSainsbury's Diet Lemonade 2L 60p/unitSainsbury's Apple & Mango Juice, Not From Concentrate 1L £1.30/unitRobinsons Summer Fruits Squash No Added Sugar 1L £1.65/unitSainsbury's 100% Pure Squeezed Pineapple Juice, Not From Concentrate 1L £1.30/unitClearsprings Sauvignon Blanc 75cl £5.50/unitPhantom River Sauvignon Blanc 75cl £5.00/unitNestle Pure Life Still Spring Water 12x500ml £2.50/unitBuxton Sparkling Natural Mineral Water 8x500ml £2.10/unitBrancott Estate Sauvignon Blanc 75cl £6.75/unitSchweppes Slimline Lemonade 2L £1.30/unitMcGuigan Estate South Australian Shiraz 75cl £5.10/unitCoca-Cola Zero Sugar 8x330ml £4.00/unitVilla Maria Private Bin Sauvignon Blanc 75cl £9.25/unitDiet Coke Caffeine Free 8x330ml £4.00/unitSainsbury's British Skimmed Milk, SO Organic 1.13L (2 pint) £1.15/unitSainsbury's Kids Caledonian Still Water 6x300ml £1.10/unitCanti Prosecco 75cl £7.50/unitOatly Enriched with Calcium Oat UHT Drink 1L £1.50/unitSainsbury's Pure Orange Juice 6x200ml £1.50/unitSainsbury's Still Flavoured Water, Lemon & Lime 1L 50p/unitValdo Prosecco Marca Oro 75cl £8.50/unitOyster Bay Sauvignon Blanc 75cl £8.00/unitRibena Blackcurrant Squash 850ml £2.30/unitVolvic Mineral Water 6x1.5L £3.40/unitCampo Viejo Rioja Tempranillo 75cl £6.75/unitNescafé Azera Americano Instant Coffee 100g £4.60/unitTropicana Orange Juice Original 950ml £2.45/unitSainsbury's Double Strength Orange & Mango Squash, No Added Sugar 1.5L £1.00/unit  Robinsons Lemon Squash No Added Sugar 1L £1.65/unitSchweppes Lemonade 2L £1.30/unitRobinsons Orange & Pineapple Squash No Added Sugar 1L £1.65/unitSainsbury's Diet Indian Tonic with Lime 1L 60p/unitSt Helen's Farm Semi Skimmed Goats Milk 1L £1.80/unitSainsbury's Double Strength Orange, Lemon & Pineapple Squash, No Added Sugar 1.5L £1.00/unitSainsbury's Double Strength Summerfruits Squash, No Added Sugar 1.5L £1.00/unit    Alpro Oat UHT Drink 1L £1.80/unitInnocent Smooth Orange Juice 900ml £1.50/unitSainsbury's British Whole Milk, SO Organic 1.13L (2 pint) £1.15/unitSainsbury's Skimmed Long Life Milk 1L 80p/unitNescafé Gold Blend Instant Coffee 200g £7.00/unitHighland Spring Still Water Sports Cap 12x330ml £3.00/unitSainsbury's Cava Brut 75cl £6.00/unitAlpro Light Unsweetened Soya Fresh Drink 1L £1.55/unitSainsbury's Caledonian Still Water 2L 50p/unitKoko Coconut UHT Drink 1L £1.50/unitSainsbury's House Pinot Grigio 75cl £4.50/unitSainsbury's Cola Zero 2L 45p/unitSt Helen's Farm Whole Goats Milk 1L £1.80/unitSainsbury's Double Strength Cherries & Berries Squash, No Added Sugar 1.5L £1.00/unitSainsbury's Lemonade 2L 60p/unitSainsbury's Pure Orange Juice With Bits 1L 85p/unitSainsbury's Pinot Grigio, Taste the Difference 75cl £6.00/unitSchweppes Tonic Water 1L £1.50/unitSainsbury's Cranberry Juice Drink 1L 85p/unitNescafé Gold Blend Instant Coffee Refill 150g £3.50/unitSainsbury's Gold Roast Instant Coffee 200g £3.15/unitSainsbury's Pure Orange Juice with Juicy Bits 1L 95p/unitEdizione 789 Di Mondelli Prosecco 75cl £6.25/unit