Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Catch all cookies from any website
#1
Hey,

first of all sorry for my bad english but i trials it.

I try to create a cookie scanner that get all cookies (first and third party) from a website.
My first idea was to open a website with the selenium chromedriver and read the sqlite3 database which the chrome browser creates. I would do that because selenium itself can't read third party cookies. My problem is that the database is sometimes empty and sometimes not. If i open the database with python i get actually always an empty result but sometimes i open the database after my code is finished with a sqlite browser i get the cookies. I don't know why...

For example i open a mozilla website.

from selenium import webdriver
import os, shutil, sqlite3


browser_list_place = 0
browser_list = []
profiles_folder = "profiles"


def getcookies(url):
    if os.path.isdir(profiles_folder):
        shutil.rmtree(profiles_folder)

    co = webdriver.ChromeOptions()
    co.add_argument("--no-sandbox")
    co.add_argument("--user-data-dir=" + profiles_folder + "/" + str(browser_list_place))

    browser_list.append(webdriver.Chrome('D:\crawler\chromedriver.exe', options=co))
    browser_list[browser_list_place].set_page_load_timeout(30)

    browser_list[browser_list_place].get(url)

    #browser_list[browser_list_place].quit()

    for folder in range(0, browser_list_place + 1):
        con = sqlite3.connect(profiles_folder + "/" + str(folder) + "/Default/Cookies")
        cur = con.cursor()
        cur.execute("SELECT * FROM cookies")
        rows = cur.fetchall()
        for row in rows:
            print(row)


getcookies('https://developer.mozilla.org/de/')
In the chrome browser i can see 7 cookies.
İmage



Now after much problems i asking me if there a more efficient way to get all cookies from a website (first and third party) without selenium?

Thanks for your help!
Quote
#2
Can't anybody help me? :/
Quote
#3
Today's web sites are using a lot of JS so the short answer is - no. You can't catch all cookies from a web site without selenium.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Quote
#4
Okay thanks, that is a good to know.

Do you know why my database is sometimes empty although i can see the cookies in browser? It's very curious...
Quote
#5
(Aug-14-2019, 01:14 PM)Chuky Wrote: If i open the database with python i get actually always an empty result but sometimes i open the database after my code is finished with a sqlite browser i get the cookies. I don't know why...
Try give user-data-dir a real path to a drive,and don't use single \ in path use r or turn it around /.
Test.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
import sqlite3

#--| Setup
options = Options()
options.add_argument("--headless")
#options.add_argument('--disable-gpu')
#options.add_argument('--log-level=3')
options.add_argument(r"user-data-dir=C:\selenium")
browser = webdriver.Chrome(executable_path=r'chromedriver.exe', options=options)
#--| Parse or automation
browser.get('https://developer.mozilla.org/de/')
browser.implicitly_wait(2)
#print(browser.get_cookies())

#--| DB
con = sqlite3.connect(r'C:\selenium\Default\Cookies')
cur = con.cursor()
cur.execute("SELECT * FROM cookies")
rows = cur.fetchall()
#print(len(rows))
print(f'Live get {len(browser.get_cookies())} cookies,From DB get {len(rows)} cookies')
Output:
Live get 5 cookies,From DB get 7 cookies
Quote
#6
Why it not works for me :(

My result is:
Quote:Live get 5 cookies,From DB get 0 cookies

And if i open the cookie database with a sqlite3 browser, it shows 5 cookies:
İmage
Quote
#7
Try copy my code,make new folder for user-data eg C:\selenuim.
Then get DB for this folder as shown.
Two 3-party cookies i get in DB is from Google.
İmage
Quote
#8
I try it. I modfied the code because the script is believed to run on a debian server.
Now it looks like:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
import sqlite3
 
#--| Setup
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--disable-cookie-encryption")
options.add_argument(r"user-data-dir=/bin/crawler/test")
browser = webdriver.Chrome(executable_path=r'/bin/crawler/chromedriver', options=options)
#--| Parse or automation
browser.get('https://developer.mozilla.org/de/')
browser.implicitly_wait(2)
 
#--| DB
con = sqlite3.connect(r'/bin/crawler/test/Default/Cookies')
cur = con.cursor()
cur.execute("SELECT * FROM cookies")
rows = cur.fetchall()

for row in rows:
	print(row)
And my result is 5 cookies...
(13210772067650212, u'.mozilla.org', u'_ga', u'GA1.2.1914013376.1566298468', u'/', 13273844067000000, 0, 0, 13210772067650212, 1, 1, 1, <read-write buffer ptr 0x7f0ba68001f0, size 0 at 0x7f0ba68001b0>, -1)
(13210772067654690, u'.mozilla.org', u'_gat', u'1', u'/', 13210772127000000, 0, 0, 13210772067654690, 1, 1, 1, <read-write buffer ptr 0x7f0ba68001b0, size 0 at 0x7f0ba6800170>, -1)
(13210772067651648, u'.mozilla.org', u'_gid', u'GA1.2.1265142963.1566298468', u'/', 13210858467000000, 0, 0, 13210772067651648, 1, 1, 1, <read-write buffer ptr 0x7f0ba6800270, size 0 at 0x7f0ba6800230>, -1)
(13210772059377725, u'.developer.mozilla.org', u'dwf_sg_task_completion', u'False', u'/', 13213364059377725, 1, 0, 13210772059377725, 1, 1, 1, <read-write buffer ptr 0x7f0ba68002b0, size 0 at 0x7f0ba6800270>, -1)
(13210772059500023, u'developer.mozilla.org', u'lux_uid', u'156629845949992799', u'/', 13210773859500023, 0, 0, 13210772059500023, 1, 1, 1, <read-write buffer ptr 0x7f0ba68002f0, size 0 at 0x7f0ba68002b0>, -1)

Quote
#9
I tested on Linux Mint 19 and to get 5 there.
Not sure why there is difference between with same code on Windows and Linux.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Login to website kapibara 4 236 Jul-27-2019, 02:40 PM
Last Post: kapibara
  Extracting links from website with selenium bs4 and python M1ck0 1 111 Jul-20-2019, 10:29 PM
Last Post: Larz60+
  download pdf file from website m_annur2001 1 202 Jun-21-2019, 05:03 AM
Last Post: j.crater
  website development masoud_da 9 398 Jun-08-2019, 06:54 PM
Last Post: masoud_da
  python scraping all cookies halberdd 1 512 Feb-26-2019, 12:10 PM
Last Post: halberdd
  Python + request from specific website - please help hoff1022 8 608 Feb-14-2019, 06:52 PM
Last Post: buran
  using webbot for website login fails loeten 2 1,022 Jan-03-2019, 07:31 AM
Last Post: loeten
  Taking screenshots with http authentication with username and password from website onenessboy 3 463 Nov-30-2018, 04:17 AM
Last Post: onenessboy
  Flask: Cookies for Saving User Input ? jomonetta 2 595 Nov-03-2018, 10:47 AM
Last Post: j.crater
  I wan't to Download all .zip Files From A Website (Project AI) eddywinch82 68 5,981 Oct-28-2018, 02:13 PM
Last Post: eddywinch82

Forum Jump:


Users browsing this thread: 1 Guest(s)