Python Forum
Linking with multiple python libraries
Thread Rating:
  • 2 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Linking with multiple python libraries
#1
I have a somewhat esoteric question, but hopefully someone with knowledge of the Python internals (CPython in particular) can answer this.

I'm working in a very large codebase, developed by a very large company, in which many "apps" which are developed independently by teams in different groups and different brands of the company will all run inside a single process space and a single main thread. Only one app at a time, but the user can switch between them. There is an existing app which is embedding python 2.7, and my team also is developing an app which will embed some version of python 3.

I'm looking primarily at Windows here, but I may face the same issue in Unix.

What will happen if two different versions of python are embedded and initialized within the same process? Will they conflict? Will they use different GILs? Is the GIL or any other resource acquired by name, where the two libs will interfere with each other?

Ideally, and moving forward with the new code my team is developing, the initialization of python will be managed globally and each app will obtain a sub-interpreter. But I don't think I can modify the existing code which uses 2.7 and a single interpreter.

Thanks in advance, I greatly appreciate any thoughts (aside from restructuring the company)!
Reply
#2
(Oct-13-2017, 01:14 AM)poijuggler Wrote: What will happen if two different versions of python are embedded and initialized within the same process?

Not sure what you mean by this. If you have two (or more) versions of Python, they each have their own respective 'home' directories. If you import a module, Python will search the directory of the version that called it, if it is not there, you will get an error, regardless if the module exists in the other versions directory.

If you mean can you import, let's say a module written in Python 3 into a script written in Python 2, I don't see why not, provided the Python 2 executable knows where to find it (for instance, both scripts are in the same directory). Of course it may not work entirely as you expect. You could also use 'subprocess' to run your Python 3 script then return to the Python 2 script.


(Oct-13-2017, 01:14 AM)poijuggler Wrote: the initialization of python will be managed globally and each app will obtain a sub-interpreter.

Again, it's unclear as to what you mean by 'sub-interpreter'.

DISCLAIMER: This is just me thinking out loud here as I've not tried to mix and match versions. Just seems like to much work for to little gain.

As to restructuring the company, that shouldn't be necessary, but they should know that Python 2 is a dead horse, so stop beating it. :-)
If it ain't broke, I just haven't gotten to it yet.
OS: Windows 10, openSuse 42.3, freeBSD 11, Raspian "Stretch"
Python 3.6.5, IDE: PyCharm 2018 Community Edition
Reply
#3
So to be more clear about what I'm facing, let's say we have A.dll, which was written by some other team, which links with python2.7/libs/python27.lib, and calls Py_Initialize. I'm writing B.dll, which links with python3.x/libs/python3x.lib and calls Py_Initialize. Is anything bad going to happen here if the application runs the code from both A.dll and B.dll, in one order or the other?

I'm not talking about the scripts - scripts written by the A team will be designed for 2.7 and will run in the 2.7 interpreter, and scripts my team writes will be designed for 3.x and will run in the 3.x interpreter. The python modules will be completely separate and the paths will be set up to keep things straight.

By sub-interpreter, I mean using Py_NewInterpreter, so if B.dll and C.dll are both using python 3, they will each obtain a separate interpreter environment.

Thanks!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  python convert multiple files to multiple lists MCL169 6 1,436 Nov-25-2023, 05:31 AM
Last Post: Iqratech
  Adding libraries to embedded Python as a ZIP The_Oman 0 1,193 May-05-2023, 04:05 PM
Last Post: The_Oman
  Python script with local libraries zazas321 0 944 Jul-05-2022, 04:55 AM
Last Post: zazas321
  error in Linking two coroutines shankar 3 5,513 Jun-19-2020, 06:50 PM
Last Post: gogawaleankita
  linking with excel mhmdbarazi 1 2,009 Apr-23-2020, 04:18 AM
Last Post: buran
  Linking python to Google Sheets Charliefish1311 1 2,248 Jul-09-2019, 12:31 AM
Last Post: micseydel
  Installing Python and pointing it to required libraries hooiberg 2 4,265 May-13-2019, 05:55 PM
Last Post: ebolisa
  Package python script which has different libraries as a single executable or script tej7gandhi 1 2,583 May-11-2019, 08:12 PM
Last Post: keames
  Libraries needed for python install? burvil 7 20,009 Oct-18-2017, 04:55 AM
Last Post: burvil
  Adding libraries to Python 3.6.2 embedded hynek_compelson 1 4,786 Aug-10-2017, 09:01 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020