Overview
I've created a makefile intended to build Python 3.7.3 and all of its dependencies in a standalone, non-standard path. The default prefix is /opt/utils/python3/1 but you can put it anywhere you want.The makefile will automatically retrieve and unpack the source for the various dependencies, and usually one simply needs to change the version number in the file to use a different version of whatever software. Except sqlite, who annoyingly put their releases in subfolders by year released.
Every binary in this package should use proper rpaths, pointing to whatever
$PREFIX
value you set. This means that you can symlink to the binaries, and they should still be able to find all of their modules and libraries. This makes it very easy to atomically maintain multiple versions of Python (and its dependencies) in a single system.Things to Improve
A number of files are linking to system libraries, rather than the ones I am building. I'm not totally sure why, but I suspect it's a version compatibility issue. For example, even though I build ncurses 6.1, there are 73 files that link to the system
libncurses.so.5
. More investigation is required. I'm open to suggestions here. I'll see if I can work out some of the details myself and keep this post updated as I learn more.A lot of the compilation switches I'm using are kind of a best-guess effort. I am honestly not totally sure what the "best" choices are for a lot of this is. If anyone can point out ways I can improve functionality or stability by using (or not using) various compile-time options, I would appreciate it.
Requirements
This is intended to be used on a CentOS 7 system. It may work on other versions or distributions but I won't promise anything. At the very least, the yum packages
gcc
, gcc-c++
and make
are required for this to build. Other things may be as well, but I can't specifically point to them. If you run into issues let me know and I'll see if I can help figure it out. A ridiculously powerful tool for debugging that is yum provides
, which can usually point you in the right direction. That's how I discovered I was missing gcc-c++
, for example.Usage
If you run
make
or make help
it spits out a short usage screen. If it finds an executable file named find_deps.sh
, the makefile will run it at the very end of building python3
.Output:makefile for Python 3.7.3 and dependancies.
Will build artifacts and install them to PREFIX=/opt/utils/python3/1
If PREFIX is set as an environment variable this makefile will use it.
Any source code needed will be retrieved.
Any target will automatically build and install dependencies.
Note: OpenSSL is compiled without compression support for additional security
TARGET VERSION DEPENDENCIES
expat 2.2.6 none
ffi 3.2.1 none
mpdec 2.4.2 none
ncurses 6.1 none
openssl 1.0.2r none
python3 3.7.3 expat ffi mpdec openssl sqlite3
readline 8.0 ncurses
sqlite3 3.28.0 readline zlib
zlib 1.2.11 none
Special build targets:
all - Synonym for python3
help - Displays this message
clean - Removes temp directories
Set VERBOSE to any non-null value to see debug messages
The Makefile
# Initialize env vars ifeq ($(strip $(PREFIX)),) PREFIX=/opt/utils/python3/$(revision) endif LDFLAGS=-Wl,-rpath,$(PREFIX)/lib:$(PREFIX)/lib64 -L$(PREFIX)/lib:$(PREFIX)/lib64 CFLAGS=-Wl,-rpath,$(PREFIX)/lib:$(PREFIX)/lib64 -I$(PREFIX)/include CPPFLAGS=$(CFLAGS) export LDFLAGS CFLAGS CPPFLAGS PREFIX # Software Package Versions revision=1 expat=2.2.6 ffi=3.2.1 mpdec=2.4.2 ncurses=6.1 openssl=1.0.2r python3=3.7.3 readline=8.0 sqlite3=3.28.0 zlib=1.2.11 #Global vars PYTHON_LIBS=-ltinfo -lncurses #Software Sources expatsource=https://github.com/libexpat/libexpat/releases/download/R_$(subst .,_,$(expat))/$(expatfile) ffisource=ftp://sourceware.org/pub/libffi/$(ffifile) mpdecsource=https://www.bytereef.org/software/mpdecimal/releases/$(mpdecfile) ncursessource=ftp://ftp.gnu.org/gnu/ncurses/$(ncursesfile) opensslsource=https://www.openssl.org/source/$(opensslfile) python3source=https://www.python.org/ftp/python/$(python3)/$(python3file) readlinesource=ftp://ftp.gnu.org/gnu/readline/$(readlinefile) sqlite3source=https://www.sqlite.org/2019/$(sqlite3file) zlibsource=https://zlib.net/$(zlibfile) # Derived global vars expatfile=expat-$(expat).tar.bz2 ffifile=libffi-$(ffi).tar.gz mpdecfile=mpdecimal-$(mpdec).tar.gz ncursesfile=ncurses-$(ncurses).tar.gz opensslfile=openssl-$(openssl).tar.gz readlinefile=readline-$(readline).tar.gz sqlite3file=sqlite-autoconf-$(shell printf '%d%02d%02d%02d' $(subst ., ,$(sqlite3))).tar.gz python3file=Python-$(python3).tgz zlibfile=zlib-$(zlib).tar.gz expatdir=$(basename $(basename $(expatfile))) ffidir=$(basename $(basename $(ffifile))) mpdecdir=$(basename $(basename $(mpdecfile))) ncursesdir=$(basename $(basename $(ncursesfile))) openssldir=$(basename $(basename $(opensslfile))) readlinedir=$(basename $(basename $(readlinefile))) sqlite3dir=$(basename $(basename $(sqlite3file))) python3dir=$(basename $(python3file)) zlibdir=$(basename $(basename $(zlibfile))) cleanexpat=$(wildcard $(expatdir)/.) cleanffi=$(wildcard $(ffidir)/.) cleanmpdec=$(wildcard $(mpdecdir)/.) cleanncurses=$(wildcard $(ncursesdir)/.) cleanopenssl=$(wildcard $(openssldir)/.) cleanreadline=$(wildcard $(readlinedir)/.) cleansqlite3=$(wildcard $(sqlite3dir)/.) cleanpython3=$(wildcard $(python3dir)/.) cleanzlib=$(wildcard $(zlibdir)/.) help: echo "makefile for Python $(python3) and dependancies." echo "Will build artifacts and install them to PREFIX=$(PREFIX)" echo "If PREFIX is set as an environment variable this makefile will use it." echo echo "Any source code needed will be retrieved." echo "Any target will automatically build and install dependancies." echo "Note: OpenSSL is compiled without compression support for additional security" echo printf "%-8s %-7s %s\n" "TARGET" "VERSION" "DEPENDENCIES" printf "%-8s %-7s %s\n" "expat" "$(expat)" "none" printf "%-8s %-7s %s\n" "ffi" "$(ffi)" "none" printf "%-8s %-7s %s\n" "mpdec" "$(mpdec)" "none" printf "%-8s %-7s %s\n" "ncurses" "$(ncurses)" "none" printf "%-8s %-7s %s\n" "openssl" "$(openssl)" "none" printf "%-8s %-7s %s\n" "python3" "$(python3)" "expat ffi mpdec openssl sqlite3" printf "%-8s %-7s %s\n" "readline" "$(readline)" "ncurses" printf "%-8s %-7s %s\n" "sqlite3" "$(sqlite3)" "readline zlib" printf "%-8s %-7s %s\n" "zlib" "$(zlib)" "none" echo echo "Special build targets:" echo "all - Synonym for python3" echo "help - Displays this message" echo "clean - Removes temp directories" echo echo "Set VERBOSE to any non-null value to see debug messages" $(VERBOSE).SILENT: .PHONY: \ help \ expat \ ffi \ mpdec \ ncurses \ openssl \ python3 \ readline \ sqlite3 \ zlib \ $(cleanexpat) \ $(cleanffi) \ $(cleanmpdec) \ $(cleanncurses) \ $(cleanopenssl) \ $(cleanpython3) \ $(cleanreadline) \ $(cleansqlite3) \ $(cleanzlib) all: python3 expat: echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: expatfile=$(expatfile)" echo "INFO: expatsource=$(expatsource)" test -f "$(expatfile)" \ && echo "INFO: Found $(expatfile)" \ || (echo "INFO: Retrieving $(expatsource)"; curl -Lso "$(expatfile)" "$(expatsource)") tar -xf "$(expatfile)" cd "$(expatdir)" && \ ./configure \ --prefix="$(PREFIX)" \ --enable-shared \ --enable-static \ --without-docbook && \ make && \ make install echo "expat $(expat)" >> $(PREFIX)/version ffi: echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: ffifile=$(ffifile)" echo "INFO: ffisource=$(ffisource)" test -f "$(ffifile)" \ && echo "INFO: Found $(ffifile)" \ || (echo "INFO: Retrieving $(ffisource)"; curl -Lso "$(ffifile)" "$(ffisource)") tar -xf "$(ffifile)" cd "$(ffidir)" && \ ./configure \ --prefix="$(PREFIX)" \ --enable-shared \ --enable-static && \ make && \ make install echo "ffi $(ffi)" >> $(PREFIX)/version mpdec: echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: mpdecfile=$(mpdecfile)" echo "INFO: mpdecsource=$(mpdecsource)" test -f "$(mpdecfile)" \ && echo "INFO: Found $(mpdecfile)" \ || (echo "INFO: Retrieving $(mpdecsource)"; curl -Lso "$(mpdecfile)" "$(mpdecsource)") tar -xf "$(mpdecfile)" cd "$(mpdecdir)" && \ ./configure \ --prefix="$(PREFIX)" && \ make && \ make install echo "mpdec $(mpdec)" >> $(PREFIX)/version ncurses: echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: ncursesfile=$(ncursesfile)" echo "INFO: ncursessource=$(ncursessource)" test -f "$(ncursesfile)" \ && echo "INFO: Found $(ncursesfile)" \ || (echo "INFO: Retrieving $(ncursessource)"; curl -Lso "$(ncursesfile)" "$(ncursessource)") tar -xf "$(ncursesfile)" cd "$(ncursesdir)" && \ ./configure \ --prefix="$(PREFIX)" \ --with-cxx-shared \ --with-profile \ --with-pthread \ --with-shared \ --with-termlib \ --enable-broken_linker \ --enable-ext-colors \ --enable-ext-mouse \ --enable-ext-putwin \ --enable-hard-tabs \ --enable-no-padding \ --enable-opaque-curses \ --enable-opaque-form \ --enable-opaque-menu \ --enable-opaque-panel \ --enable-pthreads-eintr \ --enable-reentrant \ --enable-rpath \ --enable-signed-char \ --enable-sigwinch \ --enable-symlinks \ --enable-tcap-names \ --enable-termcap \ --enable-weak-symbols \ --enable-widec \ --enable-xmc-glitch && \ make && \ make install echo "ncurses $(ncurses)" >> $(PREFIX)/version openssl: echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: opensslfile=$(opensslfile)" echo "INFO: opensslsource=$(opensslsource)" test -f "$(opensslfile)" \ && echo "INFO: Found $(opensslfile)" \ || (echo "INFO: Retrieving $(opensslsource)"; curl -Lso "$(opensslfile)" "$(opensslsource)") tar -xf "$(opensslfile)" cd "$(openssldir)" && \ ./config \ --prefix="$(PREFIX)" \ --openssldir="$(PREFIX)" \ threads \ shared \ no-comp \ "$(CFLAGS)" && \ make depend && \ make && \ make install echo "OpenSSL $(openssl)" >> $(PREFIX)/version python3: expat ffi mpdec openssl sqlite3 echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: python3file=$(python3file)" echo "INFO: python3source=$(python3source)" test -f "$(python3file)" \ && echo "INFO: Found $(python3file)" \ || (echo "INFO: Retreiving $(python3source)"; curl -Lso "$(python3file)" "$(python3source)") tar -xf "$(python3file)" cd "$(python3dir)" && \ ./configure \ --prefix="$(PREFIX)" \ --enable-shared \ --enable-optimizations \ --enable-profiling \ --enable-loadable-sqlite-extensions \ --enable-ipv6 \ --with-system-expat=$(PREFIX) \ --with-system-ffi=$(PREFIX) \ --with-system-libmpdec=$(PREFIX) \ --with-openssl=$(PREFIX) \ --with-ensurepip=upgrade \ "CFLAGS=$(CFLAGS) $(PYTHON_LIBS)" \ "CPPFLAGS=$(CPPFLAGS) $(PYTHON_LIBS)" \ "LDFLAGS=$(LDFLAGS) $(PYTHON_LIBS)" && \ make \ "CFLAGS=$(CFLAGS) $(PYTHON_LIBS)" \ "CPPFLAGS=$(CPPFLAGS) $(PYTHON_LIBS)" \ "LDFLAGS=$(LDFLAGS) $(PYTHON_LIBS)" && \ make install echo -n "INFO: Testing if Python runs: " $(PREFIX)/bin/python3 --version [[ -x find_deps.sh ]] && echo "Testing binary dependancies:" && ./find_deps.sh echo "python3 $(python3)" >> $(PREFIX)/version readline: ncurses echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: readlinefile=$(readlinefile)" echo "INFO: readlinesource=$(readlinesource)" test -f "$(readlinefile)" \ && echo "INFO: Found $(readlinefile)" \ || (echo "INFO: Retrieving $(readlinesource)"; curl -Lso "$(readlinefile)" "$(readlinesource)") tar -xf "$(readlinefile)" cd "$(readlinedir)" && \ ./configure \ --prefix="$(PREFIX)" \ --with-curses="$(PREFIX)" \ --enable-shared \ --enable-static && \ make && \ make install echo "readline $(readline)" >> $(PREFIX)/version sqlite3: readline zlib echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: sqlite3file=$(sqlite3file)" echo "INFO: sqlite3source=$(sqlite3source)" test -f "$(sqlite3file)" \ && echo "INFO: Found $(sqlite3file)" \ || (echo "INFO: Retreiving $(sqlite3source)"; curl -Lso "$(sqlite3file)" "$(sqlite3source)") tar -xf "$(sqlite3file)" cd "$(sqlite3dir)" && \ ./configure \ --prefix="$(PREFIX)" \ --enable-shared \ --enable-static \ --enable-readline \ --enable-threadsafe \ --enable-dynamic-extensions && \ make && \ make install echo "sqlite3 $(sqlite3)" >> $(PREFIX)/version zlib: echo "INFO: CFLAGS=$(CFLAGS)" echo "INFO: CPPFLAGS=$(CPPFLAGS)" echo "INFO: PREFIX=$(PREFIX)" echo "INFO: zlibfile=$(zlibfile)" echo "INFO: zlibsource=$(zlibsource)" test -f "$(zlibfile)" \ && echo "INFO: Found $(zlibfile)" \ || (echo "INFO: Retrieving $(zlibsource)"; curl -Lso "$(zlibfile)" "$(zlibsource)") tar -xf "$(zlibfile)" cd "$(zlibdir)" && \ ./configure \ --prefix="$(PREFIX)" \ --64 \ --shared \ --libdir="$(PREFIX)/lib" \ --sharedlibdir="$(PREFIX)/lib" && \ make && \ make install echo "zlib $(zlib)" >> $(PREFIX)/version clean: | \ $(cleanexpat) \ $(cleanffi) \ $(cleanmpdec) \ $(cleanncurses) \ $(cleanopenssl) \ $(cleanpython3) \ $(cleanreadline) \ $(cleansqlite3) \ $(cleanzlib) @echo "INFO: Done cleaning" $(cleanexpat): @echo "INFO: Removing $(expatdir)" rm -rf "$(expatdir)" $(cleanffi): @echo "INFO: Removing $(ffidir)" rm -rf "$(ffidir)" $(cleanmpdec): @echo "INFO: Removing $(mpdecdir)" rm -rf "$(mpdecdir)" $(cleanncurses): @echo "INFO: Removing $(ncursesdir)" rm -rf "$(ncursesdir)" $(cleanopenssl): @echo "INFO: Removing $(openssldir)" rm -rf "$(openssldir)" $(cleanpython3): @echo "INFO: Removing $(python3dir)" rm -rf "$(python3dir)" $(cleanreadline): @echo "INFO: Removing $(readlinedir)" rm -rf "$(readlinedir)" $(cleansqlite3): @echo "INFO: Removing $(sqlite3dir)" rm -rf "$(sqlite3dir)" $(cleanzlib): @echo "INFO: Removing $(zlibdir)" rm -rf "$(zlibdir)"Pastebin has better syntax highlighting support for Make, you can find that >>here<<.
find_deps.sh
The script
find_deps.sh
is a short Bash script I threw together to make sure that everything is linked correctly. It searches through $PREFIX
for all ELF binaries and uses env -i ldd
to get a list of all libraries the file would pull in a vacuum, and lists any files that can't find one or more libraries using the baked-in rpath.#!/usr/bin/env bash [[ -z "$PREFIX" ]] && export PREFIX=/opt/utils/python3 pushd "$(dirname $0)" &>/dev/null MYDIR=$(pwd) popd &>/dev/null declare -a MISSING function lddd { env -i ldd "$1" 2>/dev/null } echo "Enumerating imported libraries:" echo " Count - Library" while read file; do if [[ $(file $file) =~ ELF ]]; then lddd "${file}" if [[ ! $file =~ ${MYDIR} && $(lddd "${file}") =~ "not found" ]]; then MISSING+=($file) fi fi done < <(find ${PREFIX} -type f -name '*.so*' | sort) \ | awk '$3 ~ "^/"{print $3}' \ | sort \ | uniq -c \ | sort -k 2 echo "Missing libs:" for file in "${MISSING[@]}"; do echo "${file}" lddd "${file}" done