When does Python need to use a compiler?

[JanOlvegg]

Recently, I attempted to install chromadb using pip and got an error that MS Visual Studio C++ is needed. This raised concern that this could be an issue in production because the machine might not have a compiler and requesting that it be installed might be a bureacratic issue. In what case does Python need a compiler?

I was also thinking of workarounds.
1. I could just use a linux instance for production, which usually comes default with a compiler.
2. Is there a way to precompile everything and ship it as a package that would include the python and all of the specific libraries rather than using PIP.

Thanks

[jefsummers]

No one else answering so I will take a stab, though I have not used Chroma. There is something odd here, in that Chroma can pip install in a Google Colab environment without doing any compiling. I suspect you are not installing correctly. What specifically are you doing?

[JanOlvegg]

so after a bit of research, I realized now that python packages can contain non-python code. For packages with non-python code, some compile might be needed when you run pip install. Just about every Linux distribution comes with a compiler by default, but most windows do not. As a result when you install packages that contain non-python code, it will probably install properly on Linux but fail on a windows with a c++ error unless you have a compiler installed already.

I don't have any experience with Google colab, but I am guessing that due to licensing issues the colab will be based on Linux and not windows.

[snippsat]

The problem is in the dependency of hnswlib
Two simpler solutions rather than setup a complier for C++,is to use eg Miniconda og Colab.
hnswlib for Conda,this will install a needed complier vs2015_runtime.
Example.

(tom_env) G:\div_code
λ conda install -c conda-forge hnswlib
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done

  added / updated specs:
    - hnswlib

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    hnswlib-0.7.0              |  py310h220cb41_0         129 KB  conda-forge
    numpy-1.24.2               |  py310hd02465a_0         5.6 MB  conda-forge
    openssl-3.0.8              |       hcfcfb64_0         6.8 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        12.6 MB

The following NEW packages will be INSTALLED:

  hnswlib            conda-forge/win-64::hnswlib-0.7.0-py310h220cb41_0

The following packages will be UPDATED:

  ca-certificates                      2022.9.24-h5b45459_0 --> 2022.12.7-h5b45459_0
  certifi                            2022.9.24-pyhd8ed1ab_0 --> 2022.12.7-pyhd8ed1ab_0
  numpy                              1.22.0-py310hcae7c84_0 --> 1.24.2-py310hd02465a_0
  openssl                                  3.0.5-h8ffe710_1 --> 3.0.8-hcfcfb64_0
  vs2015_runtime                     14.29.30037-h902a5da_6 --> 14.34.31931-h4c5c07a_10


Proceed ([y]/n)? y


Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done 

# Test
(tom_env) G:\div_code
λ python
Python 3.10.1 | packaged by conda-forge | (main, Dec 22 2021, 01:34:40) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hnswlib
>>>
>>> dir(hnswlib)
['BFIndex', 'Index', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']

---

Example Colab NoteBook install test ok.
Codespaces also ok.

[JanOlvegg]

OK, after reading your post, I did some digging. If I research this properly, a package in python are not compiled and third party non-python code are setup as "wheels" (I was wondering what wheels were). While python code do not need to be compile, the wheels probably might need compiiling.

Packages like Conda appears to precompile their code, so you would have different packages for different platform (ARM and x86 for example).

Does this sound right?
If conda doesn't have the package, can you compile your own? Thanks.

[snippsat]

While python code do not need to be compile, the wheels probably might need compiiling.

No, wheel(.whl) file is essentially a ZIP (.zip) archive,
with a specially crafted filename that tells installers what Python versions and platforms the wheel will support.
There is possible to pre-compiler C/C++ code then add to it wheel,so end user do not need a C/C++ compiler.

The problem when you install chromadb is the dependency of hnswlib.
So the maker of this package has not pre-compile to a wheel and rather as you can read:

Highlights:
Lightweight, header-only, no dependencies other than C++ 11

So need a C++ complier installed on OS Windows has not this as default.
That's why in this case using Conda can be easier as it dos this automatic.
Can of course setup a C++ 11 in Windows usually need install of Visual Studio and Microsoft Visual C++ Redistributable.