PhantomJS acts differently than Firefox webdriver

Two key things that helped me to solve it:

• do not use that custom wait I’ve helped you with before
• set the

  window.document.body.scrollTop

  first to 0 and then to

  document.body.scrollHeight

  in a row

Working pre:

results = []while len(results) < 200:    results = driver.find_elements_by_css_selector("div.flightbox")    print len(results)    # scroll    driver.execute_script("arguments[0].scrollIntoView();", results[0])    driver.execute_script("window.document.body.scrollTop = 0;")    driver.execute_script("window.document.body.scrollTop = document.body.scrollHeight;")    driver.execute_script("arguments[0].scrollIntoView();", results[-1])

Version 2 (endless loop, stop if there is nothing loaded on scroll
anymore):

results = []while True:    try:        wait.until(wait_for_more_than_n_elements((By.CSS_SELECTOR, "div.flightbox"), len(results)))    except TimeoutException:        break    results = self.driver.find_elements_by_css_selector("div.flightbox")    print len(results)    # scroll    for _ in xrange(5):        try: self.driver.execute_script("""     arguments[0].scrollIntoView();     window.document.body.scrollTop = 0;     window.document.body.scrollTop = document.body.scrollHeight;     arguments[1].scrollIntoView(); """, results[0], results[-1])        except StaleElementReferenceException: break  # here it means more results were loadedprint "DONE. Result count: %d" % len(results)

Note that I’ve changed the comparison in the

wait_for_more_than_n_elements

return count >= self.count

with:

return count > self.count

Version 3 (scrolling from header to footer multiple times):

header = wait.until(EC.visibility_of_element_located((By.TAG_NAME, 'header')))footer = wait.until(EC.visibility_of_element_located((By.TAG_NAME, 'footer')))results = []while True:    try:        wait.until(wait_for_more_than_n_elements((By.CSS_SELECTOR, "div.flightbox"), len(results)))    except TimeoutException:        break    results = self.driver.find_elements_by_css_selector("div.flightbox")    print len(results)    # scroll    for _ in xrange(5):        self.driver.execute_script(""" arguments[0].scrollIntoView(); arguments[1].scrollIntoView();        """, header, footer)        sleep(1)