Jan-25-2019, 09:07 PM
When scraping a new site, I like to download each page so that I can work on it offline until I perfect the code.
If I save for example browser.page_source, I get the page and all of the links, etc. which is helpful.
But what I'd really like to have is what is stored when from firefox, you use the 'Save Page As' from file menu,
which not only saves the page, but all of the supporting images, css files, javascript, etc. in a separate directory.
I could write code to do this, but not sure exactly what I need to download to be 'not too little', or 'not too much'
With selenium, when the page is brought up using:
the question: Does anyone know how to do this?
If not, does anyone know exactly what to download to be 'just enough'?
I found a package 'pywebcopy' which does a great job of downloading a page, and it's peripheral files,
but all of the links are missing in the html.
If I save for example browser.page_source, I get the page and all of the links, etc. which is helpful.
But what I'd really like to have is what is stored when from firefox, you use the 'Save Page As' from file menu,
which not only saves the page, but all of the supporting images, css files, javascript, etc. in a separate directory.
I could write code to do this, but not sure exactly what I need to download to be 'not too little', or 'not too much'
With selenium, when the page is brought up using:
caps = webdriver.DesiredCapabilities().FIREFOX
caps["marionette"] = True
browser.get(url)the firefox menu is not shown, so clicking on 'Save Page As' is not an option.the question: Does anyone know how to do this?
If not, does anyone know exactly what to download to be 'just enough'?
I found a package 'pywebcopy' which does a great job of downloading a page, and it's peripheral files,
but all of the links are missing in the html.