Python Forum

The Scrapy application just works fine when run from the command line and within the same directory. e.x

When we navigate to the folder and run a spider then that just works fine. While the same command is being used from another directory/folder then it just seems to throw error.

Following is the command that we use from a PHP application

exec('scrapy runspider G:/thevillages/thevillages/spiders/village_spider.py 2>&1', $output);

The errors are as follows

Quote:
0 => "2018-02-12 12:29:08 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)"
1 => "2018-02-12 12:29:08 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}"
2 => "Usage"
3 => "====="
4 => " scrapy runspider [options] <spider_file>"
5 => ""
6 => "runspider: error: Unable to load 'G:/thevillages/thevillages/spiders/village_spider.py': No module named thevillages.items"

However I've confirmed the path

>>> os.path.exists('G:/thevillages/thevillages/spiders')
True
>>> os.path.exists('G:/thevillages/thevillages')
True
>>> os.path.exists('G:/thevillages')
True
>>> os.path.exists('G:/thevillages/thevillages/spiders/village_spider.py')
True

I don't really have much experince with python but I do asked on most of the forums and they did advice to ask question on the python-fourm.

Thank you

---

Ah just solved it by

exec('cd G:/thevillages/thevillages/spiders/ && scrapy runspider village_spider.py &', $output);

The 2>&1 at end redirected stdout and stderr to file named in $output