Mengikis Python 3 dengan Bs4

Saya mencoba mengikis HTML situs web ini: https://www.idealista.com/venta-viviendas/madrid-madrid/ dengan python 3 (menggunakan PyCharm). Saya hanya tertarik pada harga rumah jadi saya mempersempit pencarian saya pada rentang tertentu seperti:

import requests
from bs4 import BeautifulSoup


page = requests.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')

soup = BeautifulSoup(page.text, 'html.parser')


prices=soup.findAll("span", {"class": "item-price h2-simulated"})

print(len(prices))
print(prices)

Ketika saya menjalankannya saya mendapatkan ini: 0 []

Artinya, ia tidak menemukan apa pun. Selain itu, jika saya mencetak semuanya: print(soup) Saya mendapatkan sangat sedikit konten html untuk halaman sebesar itu, jadi jelas tidak mengambil semuanya.


python beautifulsoup
person David García Ballester    schedule 03.10.2018    source sumber
comment
Anda dapat menemukannya hanya di kelas pertama, menurut saya juga find_all   -  person Serge    schedule 03.10.2018

Jawaban (1)


arrow_upward
1
arrow_downward

Situs ini dinamis, oleh karena itu, Anda perlu menggunakan alat manipulasi browser seperti selenium:

from bs4 import BeautifulSoup as soup
from selenium import webdriver
import re, collections, itertools
d = webdriver.Chrome('/Users/jamespetullo/Downloads/chromedriver')
d.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')
homes = soup(d.page_source, 'html.parser').find_all('div', {'class':'item-info-container'})
results = [i.find('div', {'class':re.compile('price\-row')}).text for i in homes]
price = collections.namedtuple('price', ['original', 'current', 'drop'])
new_prices = [list(filter(None, re.split('\s{2,}', i))) for i in results]
final_prices = [price(a, *[a for a, _ in itertools.zip_longest(b, [None, None])]) for a, *b in new_prices]

Keluaran:

[price(original=' 369.000€', current='395.000 €', drop='7%'), price(original=' 1.250.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido', current='1.650.000 €', drop='4%'), price(original=' 1.750.000€ Garaje incluido', current='1.875.000 €', drop='7%'), price(original=' 1.090.000€ Garaje incluido', current='1.195.000 €', drop='9%'), price(original=' 795.000€ ', current=None, drop=None), price(original=' 1.095.000€ Garaje incluido ', current=None, drop=None), price(original=' 355.000€ Garaje incluido ', current=None, drop=None), price(original=' 995.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.130.000€ Garaje incluido', current='1.190.000 €', drop='5%'), price(original=' 850.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.200.000€ Garaje incluido ', current=None, drop=None), price(original=' 990.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.100.000€ ', current=None, drop=None), price(original=' 830.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.390.000€ Garaje incluido ', current=None, drop=None), price(original=' 685.000€ ', current=None, drop=None), price(original=' 1.150.000€ Garaje incluido', current='1.200.000 €', drop='4%'), price(original=' 915.000€ ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido ', current=None, drop=None), price(original=' 625.000€ Garaje incluido', current='640.000 €', drop='2%'), price(original=' 735.000€', current='760.000 €', drop='3%'), price(original=' 890.000€ Garaje incluido', current='950.000 €', drop='6%'), price(original=' 925.000€', current='999.000 €', drop='7%'), price(original=' 975.000€', current='1.100.000 €', drop='11%'), price(original=' 850.000€ Garaje incluido', current='870.000 €', drop='2%'), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.500.000€ ', current=None, drop=None), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.359.000€ ', current=None, drop=None)]
person Ajax1234    schedule 03.10.2018