pytrends problem

[Karen]

I've referred to the pytrends, but I still can't understand that the system always shows invalid syntax in the 44th line. Can anyone help me? Thanks a million.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

# -*- coding: utf-8 -*- import time import codecs import random import glob   from pytrends.request import TrendReq   google_username = "XXXXXXXX@gmail.com" google_password = "XXXXXXX"   f = open('stocks_tw.txt', 'r') stocks_no_name = [] for line in f.readlines():     data = line.split('\t')     stock_no = data[0].strip()     stock_name = data[1].strip()     stocks_no_name.append([stock_no, stock_name]) f.close()     files=glob.glob('*.csv')   downloaded_files = [fd.title().lower()[0:4] for fd in files]   stocks_no_name_new = []  for stock_no_name in stocks_no_name:     if not stock_no_name[0] in downloaded_files:         stocks_no_name_new.append(stock_no_name) stocks_no_name = stocks_no_name_new              print(len(stocks_no_name))   # connect to Google pytrends = TrendReq(google_username, google_password, custom_useragent='My Pytrends Script')   while stocks_no_name:     stock_index = random.randint(0,len(stocks_no_name)-1)     stock_no_name = stocks_no_name[stock_index]     stock_no = stock_no_name[0]     stock_name = stock_no_name[1]     print(stock_no, stock_name)       try:         one_stock_data = []             trend_payload = {'q': stock_name, 'date': ''2013-12-29 2016-12-31', 'geo': 'TW','tz': 'Etc/GMT+8'}         # trend         trend = pytrend.trend(trend_payload)         time.sleep(random.randint(120, 360))               table = trend['table']         rows = table['rows']         for i in range(len(rows)):             row_data = []             for j in range(len(rows[0]['c'])):                 row_data.append(rows[i]['c'][j]['v'])             one_stock_data.append(row_data)                            # output one_stock_data to a file         filename = unicode(stock_no, errors='ignore') + '.csv'         outfile = codecs.open(filename, "wb", "utf-8")         for i in range(len(one_stock_data)):             one_stock_data_str =  str(one_stock_data[i][0]) + ", " + str(one_stock_data[i][1])             if i != len(one_stock_data) - 1:                 one_stock_data_str =  one_stock_data_str + "\r\n"             outfile.write(one_stock_data_str)         outfile.close()                   stocks_no_name.pop(stock_index)           except:         time.sleep(random.randint(120, 360))         continue

Moderator zivoni: removed login informations

[zivoni]

There are two single quotes on start of 2013-.. string ''2013-12-29 2016-12-31', should be only one.

[Karen]

After handling the line, I have another problem.
Below is the IPython console shows. I use the anaconda python. It runs, but no file downloads, why?

debugfile('C:/Users/user/Desktop/論文/資料來源/download_trend_past3years.py', wdir='C:/Users/user/Desktop/論文/資料來源')
Traceback (most recent call last):

File "<ipython-input-1-3457bb3b7f77>", line 1, in <module>
debugfile('C:/Users/user/Desktop/論文/資料來源/download_trend_past3years.py', wdir='C:/Users/user/Desktop/論文/資料來源')

File "C:\Users\user\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 888, in debugfile
debugger.run("runfile(%r, args=%r, wdir=%r)" % (filename, args, wdir))

File "C:\Users\user\Anaconda2\lib\bdb.py", line 400, in run
exec cmd in globals, locals

File "<string>", line 1, in <module>

File "C:\Users\user\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
execfile(filename, namespace)

File "C:\Users\user\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 80, in execfile
scripttext = builtins.open(fname).read()+ '\n'

IOError: [Errno 22] invalid mode ('r') or filename: 'c:/users/user/desktop/\xe8\xab\x96\xe6?/\xe8\xb3\x87\xe6?\xe4\xbe\x86\xe6?/download_trend_past3years.py'


runfile('C:/Users/user/Desktop/論文/資料來源/download_trend_past3years.py', wdir='C:/Users/user/Desktop/論文/資料來源')
849
('1718', '\xa4\xa4\xc5\xd6')
('5608', '\xa5|\xba\xfb\xaf\xe8')
('2395', '\xac\xe3\xb5\xd8')
('2103', '\xa5x\xbe\xf3')

('2430', '\xc0\xe9\xa9[\xb9\xea\xb7~')

[buran]

That's not related to your problem, but perhaps you want to edit your first post and remove your password from the script. And change it! :-)

[wavic]

That's not related to your problem, but perhaps you want to edit your first post and remove your password from the script. And change it! :-)

Wow    I'm coming, I'm coming

[zivoni]

From that error output it seems it could be a  spyder problem, maybe spyder has problem to run file with Chinese characters in path? Try to run your file directly from command prompt from your working directory with python download_trend_past3years.py

[Karen]

Can anyone tell me how to use it? I haven't used .py in command prompt before.

---

I've found the solution to the command prompt. But it still can't work.

[Karen]

what is the meaning in the module?

files=glob.glob('*.csv')
downloaded_files = [fd.title().lower()[0:4] for fd in files]

[zivoni]

glob.glob(pattern) returns list of paths matching given pattern. So if you have in working directory files

Output:
second.csv  test_file.csv  Tracker.csv  test.txt  boo.doc

then files will be list containing names of csv files.

Output:
>>> files = glob.glob("*.csv")
>>> files
['Tracker.csv', 'second.csv', 'test_file.csv']

1	downloaded_files = [fd.title().lower()[0:4] for fd in files]

Right side is a list comprehension - this one "takes" names from files list, for each name it uppercases first characters for words in name, then lowercase all, then truncates it to first four characters.

Output:
>>> downloaded_files = [fd.title().lower()[0:4] for fd in files] 
>>> downloaded_files
['trac', 'seco', 'test']

That .title() part is totally obsolete (due to following .lower()) and  zero can be omitted too.

[Karen]

Before the 42nd line, the code seems to be workable. But the pytrends can't work. And I've written the line from 44th to 46rd as

try:
one_stock_data = []
# trend
trend = pytrends.build_payload(kw_list=[stock_name],timeframe='2013-12-29 2016-12-31', geo='TW')
time.sleep(random.randint(120, 360))

But it still can't work.