Store Screenshot Selenium + MongoDB

[Nuwan16]

I just need to take screenshot of a webpage and store / retrieve that image in mongodb. Below code I'm using to take screenshot of webpage and How I store it in a mongodb collection?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

from selenium import webdriver from PIL import Image from io import BytesIO   fox = webdriver.Firefox() fox.get('http://example.com/')   # now that we have the preliminary stuff out of the way time to get that image :D element = fox.find_element_by_id('hlogo') # find part of the page you want image of location = element.location size = element.size png = fox.get_screenshot_as_png() # saves screenshot of entire page fox.quit()   im = Image.open(BytesIO(png)) # uses PIL library to open image in memory   left = location['x'] top = location['y'] right = location['x'] + size['width'] bottom = location['y'] + size['height']     im = im.crop((left, top, right, bottom)) # defines crop points im.save('screenshot.png') # saves new cropped image

[ndc85430]

Why do you want to store the image in the database in the first place, rather than some file storage (S3, Google Cloud Storage, ...) and storing the URL in the database? Did you look at the docs on Python drivers for MongoDB?

[Nuwan16]

actually I need do that with scrapy. In my program when scrapy spider crawl webpage and store some data like title,description,url... of webpage in mongodb. So I need to when spider crawl a web page, take screenshot of webpage and store it in mongodb. And after that I need retrieve it from mongodb. ? This is my spider ?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90

import scrapy from scrapy.selector import Selector from search.models import * import lxml from lxml.html.clean import Cleaner import re from urllib.parse import urlparse import json     class DSSpider(scrapy.Spider):     name = "ds_spider"       def __init__(self, recrawl='no'):         if app.config['SPIDER_ALLOWED_DOMAINS'] != None:             self.allowed_domains = app.config['SPIDER_ALLOWED_DOMAINS']           self.start_urls = ['https://www.imdb.com']         is_crawled = recrawl.lower() in ['y', 'yes', 't', 'true', '1']         crawl_list = Crawllist \             .objects(is_crawled=is_crawled) \             .limit(app.config['CLOSESPIDER_PAGECOUNT']) \             .order_by('updated_at')         for link in crawl_list:             self.start_urls.append(link.url)       def parse(self, response):         schemas = response.xpath('//script[@type="application/ld+json"]//text()').extract()           for schema in schemas:             data = json.loads(schema, cls=json.JSONDecoder)             page_markup = data.get('@type')           selector = Selector(response)         # get page title         page_title = selector.xpath('//title/text()').extract()[0]         # get page content         cleaner = Cleaner()         cleaner.javascript = True         cleaner.style = True         page_html = selector.xpath('//body').extract()[0]         # remove js and css code         page_html = cleaner.clean_html(page_html)         # extract text         html_doc = lxml.html.document_fromstring(page_html)         page_content = ' '.join(lxml.etree.XPath("//text()")(html_doc))         page_content += ' ' + page_title         # remove line breaks tabs and extra spaces         page_content = re.sub('\n', ' ', page_content)         page_content = re.sub('\r', ' ', page_content)         page_content = re.sub('\t', ' ', page_content)         page_content = re.sub(' +', ' ', page_content)         page_content = page_content.strip()         # get page links         page_hrefs = response.xpath('//a/@href').extract()         page_urls = []           # filter links with unallowed domains         for link in page_hrefs:             # convert links to absolute urls             url = response.urljoin(link)             # extract domain from url             parsed_url = urlparse(url)             url_domain = parsed_url.netloc             if url_domain in self.allowed_domains:                 page_urls.append(url)         # log out some info         self.log('Page: %s (%s)' % (response.url, page_title))             # save the page         if Page.objects(url=response.url).count() == 0:             page = Page(url=response.url, title=page_title, content=page_content, markup=page_markup).save()             for url in page_urls:                 page.update(add_to_set__links=Pagelink(url=url).save())                 # add url to crawl list                 if Crawllist.objects(url=url).count() == 0:                     Crawllist(url=url).save()             # update crawl list             Crawllist.objects(url=response.url).update(is_crawled=True)         else:             page = Page.objects.get(url=response.url)             page.update(title=page_title, content=page_content, markup=page_markup)           for next_pages in  response.css('a::attr(href)'):             next_page = next_pages.extract()             print(next_page)               if next_page is not None:                 yield response.follow(next_page, callback= self.parse)

[ndc85430]

You didn't actually answer my question: why do you think you should store the images in the database and retrieve them from there? It's probably going to be less efficient for the database for a start. What exactly is your end goal? It's also a bit strange that you switched from Selenium to Scrapy; why didn't you just say you were using Scrapy to begin with?

[Nuwan16]

actually I'm developing a scrapy spider and I need take screenshot of each crawled url with other data. Then It should retrieve all scraped data with screenshot when search by title. something like little search engine. So I study internet for this screenshot purpose. I found selenium or splash can do that. But I don't familiar with both. And also there are lot of examples I found which store screenshots in local drive. But I want to store it in mongodb collection. And also I don't have idea that possible to use google cloud or S3 for achieve final goal. Because already I'm using mongodb.

[ndc85430]

If you're just doing this locally, then store the images on the file system and the paths in the database. There's no reason to store the images in the database.

---

Databases aren't really meant for storing binaries like images. File systems and object storage facilities (like those I mentioned) are.

[BitPythoner]

I see some options that don't involve MongoDB,
1) Sqlite3 which is much easier and quicker, as it stores all info in 1 file.
2) Just a file on your disk, easier, better, quicker.

[ndc85430]

I don't think there's an issue using MongoDB. I'd disagree that using SQLite or any relational database is easier. It depends how the data are being used. Plus, if they're already using Mongo, they'd have to do more work to switch - introducing a schema, normalising the data, etc.

[Nuwan16]

Is it possible to store and retrieve large size images in mongodb? If we can, which is the data type should give for image file?

[ndc85430]

Sigh. We're going round in circles here. Don't store the images in the database. Store them in something that was designed for that and store the paths in the database.