Trying to automate tweets without Twitter API

[HowardHarkness]

Background: I am a C++ programmer learning Python. I'm to the stage where I can write simple utilities with some tutorial pages open

I thought I would try writing a script that would post Tweets to my Twitter account, with attached images. I initially thought I would use the Twitter API, which would be simpler, but I decided I wanted to learn more general browser automation, and I didn't want to go through the 5-day process of getting developer access to the Twitter API.

Well, looks like the general non-API way of doing things is harder... However, I want some friends to be able to use my completed script without having to go through the developer application process.

I figured out how to install selenium and the Chrome web driver.

Currently, I have figured out how to log in to my account, and make a tweet. But I am stuck on the problem of how to attach an image. It's easy enough to do interactively, but I want the script to run unattended. On the "Compose New Tweet" dialog, there is an "insert image" button. I can get to that with Python, but when I click on it, it creates a Windows file choice dialog, which I have no idea how to control.

Here's what I have worked out so far (BTW, any suggestions on how to do this better or simpler are welcome):

#####################################################
# Local variables
tweettext = "A tweet from my list of tweets"
imagefile = "<Full pathname of an image file on my PC>"
chromedriverpath = "<Full pathname of the downloaded Chrome driver>"
accountname = '<MyTwitterAccountHandle>'
pw = '<MyLongRandomPassword>'
#####################################################

from selenium import webdriver
browser = webdriver.Chrome(chromedriverpath)
browser.get('https://twitter.com/login')
# find username field for login
elem = browser.find_element_by_class_name('js-username-field')
elem.send_keys(accountname)
# find password field
elem = browser.find_element_by_class_name('js-password-field')
elem.send_keys(pw)
# find login button & complete the login
elem = browser.find_element_by_xpath('//*[@id="page-container"]/div/div[1]/form/div[2]/button')
elem.click()
# click new tweet
elem = browser.find_element_by_id('global-new-tweet-button')
elem.click()
# find tweet box & enter the next tweet
tweetbox = browser.find_element_by_xpath('//*[@id="Tweetstorm-tweet-box-0"]/div[2]/div[1]/div[2]/div[2]/div[2]/div[1]')
tweetbox.send_keys(tweettext)
# Add image. First find the image insertion button & click it.
imagebutton = browser.find_element_by_xpath('//*[@id="Tweetstorm-tweet-box-0"]/div[2]/div[2]/div[1]/span[1]/div/div/label/input')
imagebutton.click()
#####################################################
# TODO
# At this point, a Windows file dialog comes up. Now what?

[HowardHarkness]

Looks like some help might be available from
https://sjohannes.wordpress.com/2012/03/...ow-titles/

Still working on this using mostly trial-and-error.

---

Another useful reference (I think): https://pywinauto.github.io/

[HowardHarkness]

I've improved the overall application to the point where I can do automated tweets ok, and I have simplified & refactored it into functions. It can get an account, pw from a file, then get a list of image file names and tweets from another file, and then post all of the tweets.

I'm still stuck on how to attach images to the tweets. Is there anybody here who could help me?

Here is the current interation:

import time
import random
from selenium import webdriver

#globals (file names)
# accounts.csv has "accountname","pw" pairs (currently only using 1 account)
accounts_file = 'path_to_my_accounts_csv_file'
# tweetfile.csv has "image_name.jpg","tweet text" pairs
tweetfile = 'path_to_my_tweets_csv_file'

#####################################################
# read in the file of paired strings
def read_pairs(input_file):
    with open(input_file, 'r') as f:
        firstlist = []
        secondlist = []
        for line in f:
            firsttag, secondtag = line.split(",", 1)
            firstlist.append(firsttag)
            secondlist.append(secondtag)
    return firstlist, secondlist

# start Chrome browser
def start_chrome():
    return webdriver.Chrome('C:\chromedriver_win32\chromedriver.exe')

# login to Twitter
def login_twitter(browser, account, pw):
    browser.get('https://twitter.com/login')
    # find username field for login
    elem = browser.find_element_by_class_name('js-username-field')
    elem.send_keys(account)
    # find password field
    elem = browser.find_element_by_class_name('js-password-field')
    elem.send_keys(pw)
    # find login button
    # This is not needed if the pw field terminated with a newline!
    #elem = browser.find_element_by_xpath('//*[@id="page-container"]/div/div[1]/form/div[2]/button')
    #elem.click()
    return browser

# tweet (no image handling yet)
def tweet(browser, tweettext):
    print(tweettext)
    elem = browser.find_element_by_id('global-new-tweet-button')
    elem.click()
    time.sleep(1)
    # find tweet box
    tweetbox = browser.find_element_by_id("Tweetstorm-tweet-box-0")
    tweetbox.send_keys(tweettext)
    time.sleep(1)
    # find the "tweet" button
    elem = browser.find_element_by_class_name('SendTweetsButton')
    elem.click()
    # wait long enough to satisfy Twitter. 2.5 minutes average seems to work without
    # getting a Twitter lockout. May want to add some other activity in this pause
    time.sleep(120)
    time.sleep(60 * random.random())
    return

def logout_twitter(browser):
    elem = browser.find_element_by_id('user-dropdown-toggle')
    elem.click()
    elem = browser.find_element_by_id('signout-button')
    elem.click()
    return

def main():
    browser = start_chrome()
    accounts, passwords = read_pairs(accounts_file)
    number_of_accounts = len(accounts)
    images, tweets = read_pairs(tweetfile)

	# for now, just take the first entry in the accounts file
    index = 0
    account = accounts[index]
    pw = passwords[index]
    login_twitter(browser, account, pw)
    print(account)
    print(' logged in')
    # wait long enough to satisfy Twitter
    time.sleep(15)

    for t in tweets:
        # I want to be able to attach images to the tweets
        tweet(browser, t)

    logout_twitter(browser)
    # wait for the logout to complete
    time.sleep(5)
    browser.close()

    return

[micseydel]

I recommend using mechanize or something like it instead of selenium, unless you really need Javascript. For your use case, I suspect the browser is a middle-man that you should skip. You can just use developer tools in your browser to look at what an image upload looks like, then recreate that in Python using mechanize.

[HowardHarkness]

Thank you for the mechanize suggestion. I will check that out, along with pyautogui and pywinauto. Where would I go to see a list of available modules for Python 3.7?

[micseydel]

If you use mechanize, there should be no need for any GUI / platform-specific (namely Windows) components of your code.

Pypi (https://pypi.org/) is the biggest source of modules.

[HowardHarkness]

Mechanize looks interesting, but when I tried to install it with pip3, I got an error saying

"mechanize only works on Python 2.x"

As for pypi.org... there are SO MANY modules available that I completely despair of finding an optimal set... I think pyautogui may do what I need. If I can't figure out how to do what I want with that, I'll try pywinauto. Failing that, I will be looking for suggestions from more experienced Python developers.

[micseydel]

Oof, I haven't used it in a while, that's not good.

Honestly, I use Python 3 fairly religiously but if I felt like mechanize was the right tool for the job, I would just use Python 2 for that particular script (I might go so far as to orchestrate using both versions with a shell script). It's definitely your call if you prefer Python 3 with different modules.

As for being overwhelmed - well-crafted Google searches can go a long way. Forums and sites like Stackoverflow tend to have good answers to these kinds of questions. If you want example Google search queries that I think would help, feel free to post back (but I want to make sure you think about it yourself first).

[metulburr]

I'm still stuck on how to attach images to the tweets. Is there anybody here who could help me?

this seem to work

        photo = self.browser.find_element_by_name('media[]')
        photo.send_keys('/home/metulburr/Pictures/unkown_squash.jpg') #fullpath to your image
        time.sleep(1)

# wait long enough to satisfy Twitter. 2.5 minutes average seems to work without
# getting a Twitter lockout. May want to add some other activity in this pause
time.sleep(120)
time.sleep(60 * random.random())

I doubt you have to wait that long.

You can speed up your program by using WebDriverWait instead of time.sleep

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
...
WebDriverWait(browser, 3).until(EC.presence_of_element_located((By.ID, 'global-new-tweet-button')))

Then you are not waiting longer than the browser takes to actually load it. However there are exceptions where time.sleep seems better, but the more you minimize the faster your script runs.

selenium waits
locating elements
By locators

[HowardHarkness]

I'm reasonably sure you *do* have to wait at least a couple of minutes, especially if you have as many tweets to post as I have in my list. I experimented with 20 seconds, and got a Twitter security lockout after about 20 tweets. Apparently, they really don't like automated systems. I suspect that I might be able to do some other operations at random intervals (like go to the followers page, then back) to make Twitter think there is a person doing it, but since my intent is to spread about 200 (or more) tweets overnight, an average of 3-5 minutes is about right. And I probably need to avoid making any tweets that were already posted less than 2 days prior.

Right now, I am working on another problem having to do with loading the tweets into a list. Around tweet #162, I'm getting an error on read saying that there is an illegal character in the file (0x9d), but I have carefully checked the input file, and there is no such character there. Just for grins, I moved about 10 lines from the area where the error was encountered up to the top of the file, and it still bombed around line 163.

That leads me to suspect that I'm running into some file-size limit, or maybe a list-size or buffer-size limit, so I went back and uninstalled EVERYTHING related to Python, and re-installed only the 64-bit version of 3.7.2. No joy, still bombs at the same place.

Here is the section of code that is crashing:

# read in the file of paired strings
def read_pairs(input_file):
    with open(input_file, 'r') as f:
        firstlist = []
        secondlist = []
        linenumber = 1
        for line in f: # crashes here with linenumber == 163
            print(linenumber)
            firsttag, secondtag = line.split(",", 1)
            firstlist.append(firsttag)
            secondlist.append(secondtag)
            print(firsttag)
            linenumber += 1
    return firstlist, secondlist

Error message is:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 21, in read_pairs
File "C:\Program Files\Python37\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 7360: character maps to <undefined>

---

More testing. Tweet file consists of a couple hundred lines in the following format:
image_file_name1.jpg,This is a tweet1 [+ about 90-110 more characters]
image_file_name2.jpg,This is a tweet2 [+ about 90-110 more characters]
etc.

Just for grins, I isolated just the part that reads the file and returns the lists:

#globals (file names)
tweetfile = 'd:\\_websites\\EBookCoverage\\EbookCovers.csv'

# read in the file of paired strings
def read_pairs(input_file):
    with open(input_file, 'r') as f:
        firstlist = []
        secondlist = []
        for line in f:
            firsttag, secondtag = line.split(",", 1)
            firstlist.append(firsttag)
            secondlist.append(secondtag)
            print(firsttag)
    return firstlist, secondlist

tweetpairlist = []
tweetpairlist = read_pairs(tweetfile)

And ran that by itself, which still crashes after reading 162 lines.
Traceback (most recent call last):
File "C:/Users/Howard/.PyCharmCE2018.3/config/scratches/scratch.py", line 17, in <module>
tweetpairlist = read_pairs(tweetfile)
File "C:/Users/Howard/.PyCharmCE2018.3/config/scratches/scratch.py", line 9, in read_pairs
for line in f:
File "C:\Program Files\Python37\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 7360: character maps to <undefined>

Any hints on better ways to get meaningful debug output would be greatly appreciated.