Sep-06-2018, 07:44 PM
Let's say I'm making a CLI script with dependencies in Python that I want to distribute. I have the following options:
I don't like the first option because it makes dependency handling for end-users (in my case developers) a hassle. The second option, on the other hand, is a lot of hassle to setup and to manage the binary file distribution for just a simple script. Given this case, I want to move to a setup.py file to distribute things using pip. While this has its own problems, it seems the best option for me. Now to the actual problem.
When specifying dependencies for a project in setup.py, one has to add the dependencies to a list parameter called
Now the usual fashion in fixing such problems is resorting to a requirements.txt file (or even Pipenv with its Pipfiles) with which one runs
But what about people deploying applications to PyPI? Whether its the cheese shop itself, or just a private PyPI server, there doesn't seem a agreed-upon format for specifying dependencies and 'locking' them to get reproducible builds.
In other words, the situation in such a case is a mess.
Now I imagine that I am not alone in this dilemma, and that many others have faced similar problems when distributing scripts/apps through PyPI. How do you guys manage dependencies in such cases without having to worry about sub-dependencies of packages breaking?
- Distribute the python file standalone.
- 'Freeze' the code and compile to an executable, then distribute this.
- Turn it into a python package and then distribute it.
I don't like the first option because it makes dependency handling for end-users (in my case developers) a hassle. The second option, on the other hand, is a lot of hassle to setup and to manage the binary file distribution for just a simple script. Given this case, I want to move to a setup.py file to distribute things using pip. While this has its own problems, it seems the best option for me. Now to the actual problem.
When specifying dependencies for a project in setup.py, one has to add the dependencies to a list parameter called
install_requires. Suppose I add requests==2.19.1 as a dependency in this fashion. Looking at the sup-dependencies for requests itself, one of the lines present is certifi>=2017.4.17. Now this presents a problem for me. Even by pinning the package version of requests in setup.py, I still hold the risk of a backwards-incompatible version of certifi being released and breaking requests, which could then break my code.Now the usual fashion in fixing such problems is resorting to a requirements.txt file (or even Pipenv with its Pipfiles) with which one runs
pip freeze > requirements.txt to pin package versions. However, since PyPI ignores files other than setup.py when installing with pip, this doesn't work too well. An alternative seems to be to parse the requirements.txt into the setup.py. However, according to resources such as this Stack Overflow answer and this famous article, doing so is a negative practice. The distinction seems to be between package authoring and deployment authoring, in which the latter uses one of the aforementioned files.But what about people deploying applications to PyPI? Whether its the cheese shop itself, or just a private PyPI server, there doesn't seem a agreed-upon format for specifying dependencies and 'locking' them to get reproducible builds.
In other words, the situation in such a case is a mess.
Now I imagine that I am not alone in this dilemma, and that many others have faced similar problems when distributing scripts/apps through PyPI. How do you guys manage dependencies in such cases without having to worry about sub-dependencies of packages breaking?