Python Forum
Makefile to build Python 3.7.3 on CentOS 7
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Makefile to build Python 3.7.3 on CentOS 7
#1
Overview
I've created a makefile intended to build Python 3.7.3 and all of its dependencies in a standalone, non-standard path. The default prefix is /opt/utils/python3/1 but you can put it anywhere you want.

The makefile will automatically retrieve and unpack the source for the various dependencies, and usually one simply needs to change the version number in the file to use a different version of whatever software. Except sqlite, who annoyingly put their releases in subfolders by year released.

Every binary in this package should use proper rpaths, pointing to whatever $PREFIX value you set. This means that you can symlink to the binaries, and they should still be able to find all of their modules and libraries. This makes it very easy to atomically maintain multiple versions of Python (and its dependencies) in a single system.



Things to Improve

A number of files are linking to system libraries, rather than the ones I am building. I'm not totally sure why, but I suspect it's a version compatibility issue. For example, even though I build ncurses 6.1, there are 73 files that link to the system libncurses.so.5. More investigation is required. I'm open to suggestions here. I'll see if I can work out some of the details myself and keep this post updated as I learn more.

A lot of the compilation switches I'm using are kind of a best-guess effort. I am honestly not totally sure what the "best" choices are for a lot of this is. If anyone can point out ways I can improve functionality or stability by using (or not using) various compile-time options, I would appreciate it.



Requirements

This is intended to be used on a CentOS 7 system. It may work on other versions or distributions but I won't promise anything. At the very least, the yum packages gcc, gcc-c++ and make are required for this to build. Other things may be as well, but I can't specifically point to them. If you run into issues let me know and I'll see if I can help figure it out. A ridiculously powerful tool for debugging that is yum provides, which can usually point you in the right direction. That's how I discovered I was missing gcc-c++, for example.



Usage

If you run make or make help it spits out a short usage screen. If it finds an executable file named find_deps.sh, the makefile will run it at the very end of building python3.

Output:
makefile for Python 3.7.3 and dependancies. Will build artifacts and install them to PREFIX=/opt/utils/python3/1 If PREFIX is set as an environment variable this makefile will use it. Any source code needed will be retrieved. Any target will automatically build and install dependencies. Note: OpenSSL is compiled without compression support for additional security TARGET VERSION DEPENDENCIES expat 2.2.6 none ffi 3.2.1 none mpdec 2.4.2 none ncurses 6.1 none openssl 1.0.2r none python3 3.7.3 expat ffi mpdec openssl sqlite3 readline 8.0 ncurses sqlite3 3.28.0 readline zlib zlib 1.2.11 none Special build targets: all - Synonym for python3 help - Displays this message clean - Removes temp directories Set VERBOSE to any non-null value to see debug messages


The Makefile

# Initialize env vars
ifeq ($(strip $(PREFIX)),)
PREFIX=/opt/utils/python3/$(revision)
endif
LDFLAGS=-Wl,-rpath,$(PREFIX)/lib:$(PREFIX)/lib64 -L$(PREFIX)/lib:$(PREFIX)/lib64
CFLAGS=-Wl,-rpath,$(PREFIX)/lib:$(PREFIX)/lib64 -I$(PREFIX)/include
CPPFLAGS=$(CFLAGS)

export LDFLAGS CFLAGS CPPFLAGS PREFIX

# Software Package Versions
revision=1
expat=2.2.6
ffi=3.2.1
mpdec=2.4.2
ncurses=6.1
openssl=1.0.2r
python3=3.7.3
readline=8.0
sqlite3=3.28.0
zlib=1.2.11

#Global vars
PYTHON_LIBS=-ltinfo -lncurses

#Software Sources
expatsource=https://github.com/libexpat/libexpat/releases/download/R_$(subst .,_,$(expat))/$(expatfile)
ffisource=ftp://sourceware.org/pub/libffi/$(ffifile)
mpdecsource=https://www.bytereef.org/software/mpdecimal/releases/$(mpdecfile)
ncursessource=ftp://ftp.gnu.org/gnu/ncurses/$(ncursesfile)
opensslsource=https://www.openssl.org/source/$(opensslfile)
python3source=https://www.python.org/ftp/python/$(python3)/$(python3file)
readlinesource=ftp://ftp.gnu.org/gnu/readline/$(readlinefile)
sqlite3source=https://www.sqlite.org/2019/$(sqlite3file)
zlibsource=https://zlib.net/$(zlibfile)

# Derived global vars
expatfile=expat-$(expat).tar.bz2
ffifile=libffi-$(ffi).tar.gz
mpdecfile=mpdecimal-$(mpdec).tar.gz
ncursesfile=ncurses-$(ncurses).tar.gz
opensslfile=openssl-$(openssl).tar.gz
readlinefile=readline-$(readline).tar.gz
sqlite3file=sqlite-autoconf-$(shell printf '%d%02d%02d%02d' $(subst ., ,$(sqlite3))).tar.gz
python3file=Python-$(python3).tgz
zlibfile=zlib-$(zlib).tar.gz

expatdir=$(basename $(basename $(expatfile)))
ffidir=$(basename $(basename $(ffifile)))
mpdecdir=$(basename $(basename $(mpdecfile)))
ncursesdir=$(basename $(basename $(ncursesfile)))
openssldir=$(basename $(basename $(opensslfile)))
readlinedir=$(basename $(basename $(readlinefile)))
sqlite3dir=$(basename $(basename $(sqlite3file)))
python3dir=$(basename $(python3file))
zlibdir=$(basename $(basename $(zlibfile)))

cleanexpat=$(wildcard $(expatdir)/.)
cleanffi=$(wildcard $(ffidir)/.)
cleanmpdec=$(wildcard $(mpdecdir)/.)
cleanncurses=$(wildcard $(ncursesdir)/.)
cleanopenssl=$(wildcard $(openssldir)/.)
cleanreadline=$(wildcard $(readlinedir)/.)
cleansqlite3=$(wildcard $(sqlite3dir)/.)
cleanpython3=$(wildcard $(python3dir)/.)
cleanzlib=$(wildcard $(zlibdir)/.)

help:
        echo "makefile for Python $(python3) and dependancies."
        echo "Will build artifacts and install them to PREFIX=$(PREFIX)"
        echo "If PREFIX is set as an environment variable this makefile will use it."
        echo
        echo "Any source code needed will be retrieved."
        echo "Any target will automatically build and install dependancies."
        echo "Note: OpenSSL is compiled without compression support for additional security"
        echo
        printf "%-8s %-7s %s\n" "TARGET" "VERSION" "DEPENDENCIES"
        printf "%-8s %-7s %s\n" "expat" "$(expat)" "none"
        printf "%-8s %-7s %s\n" "ffi" "$(ffi)" "none"
        printf "%-8s %-7s %s\n" "mpdec" "$(mpdec)" "none"
        printf "%-8s %-7s %s\n" "ncurses" "$(ncurses)" "none"
        printf "%-8s %-7s %s\n" "openssl" "$(openssl)" "none"
        printf "%-8s %-7s %s\n" "python3" "$(python3)" "expat ffi mpdec openssl sqlite3"
        printf "%-8s %-7s %s\n" "readline" "$(readline)" "ncurses"
        printf "%-8s %-7s %s\n" "sqlite3" "$(sqlite3)" "readline zlib"
        printf "%-8s %-7s %s\n" "zlib" "$(zlib)" "none"
        echo
        echo "Special build targets:"
        echo "all   - Synonym for python3"
        echo "help  - Displays this message"
        echo "clean - Removes temp directories"
        echo
        echo "Set VERBOSE to any non-null value to see debug messages"

$(VERBOSE).SILENT:

.PHONY: \
        help \
        expat \
        ffi \
        mpdec \
        ncurses \
        openssl \
        python3 \
        readline \
        sqlite3 \
        zlib \
        $(cleanexpat) \
        $(cleanffi) \
        $(cleanmpdec) \
        $(cleanncurses) \
        $(cleanopenssl) \
        $(cleanpython3) \
        $(cleanreadline) \
        $(cleansqlite3) \
        $(cleanzlib)

all: python3

expat:
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: expatfile=$(expatfile)"
        echo "INFO: expatsource=$(expatsource)"
        test -f "$(expatfile)" \
                && echo "INFO: Found $(expatfile)" \
                || (echo "INFO: Retrieving $(expatsource)"; curl -Lso "$(expatfile)" "$(expatsource)")
        tar -xf "$(expatfile)"
        cd "$(expatdir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --enable-shared \
                --enable-static \
                --without-docbook && \
        make && \
        make install
        echo "expat $(expat)" >> $(PREFIX)/version

ffi:
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: ffifile=$(ffifile)"
        echo "INFO: ffisource=$(ffisource)"
        test -f "$(ffifile)" \
                && echo "INFO: Found $(ffifile)" \
                || (echo "INFO: Retrieving $(ffisource)"; curl -Lso "$(ffifile)" "$(ffisource)")
        tar -xf "$(ffifile)"
        cd "$(ffidir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --enable-shared \
                --enable-static && \
        make && \
        make install
        echo "ffi $(ffi)" >> $(PREFIX)/version

mpdec:
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: mpdecfile=$(mpdecfile)"
        echo "INFO: mpdecsource=$(mpdecsource)"
        test -f "$(mpdecfile)" \
                && echo "INFO: Found $(mpdecfile)" \
                || (echo "INFO: Retrieving $(mpdecsource)"; curl -Lso "$(mpdecfile)" "$(mpdecsource)")
        tar -xf "$(mpdecfile)"
        cd "$(mpdecdir)" && \
        ./configure \
                --prefix="$(PREFIX)" && \
        make && \
        make install
        echo "mpdec $(mpdec)" >> $(PREFIX)/version

ncurses:
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: ncursesfile=$(ncursesfile)"
        echo "INFO: ncursessource=$(ncursessource)"
        test -f "$(ncursesfile)" \
                && echo "INFO: Found $(ncursesfile)" \
                || (echo "INFO: Retrieving $(ncursessource)"; curl -Lso "$(ncursesfile)" "$(ncursessource)")
        tar -xf "$(ncursesfile)"
        cd "$(ncursesdir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --with-cxx-shared \
                --with-profile \
                --with-pthread \
                --with-shared \
                --with-termlib \
                --enable-broken_linker \
                --enable-ext-colors \
                --enable-ext-mouse \
                --enable-ext-putwin \
                --enable-hard-tabs \
                --enable-no-padding \
                --enable-opaque-curses \
                --enable-opaque-form \
                --enable-opaque-menu \
                --enable-opaque-panel \
                --enable-pthreads-eintr \
                --enable-reentrant \
                --enable-rpath \
                --enable-signed-char \
                --enable-sigwinch \
                --enable-symlinks \
                --enable-tcap-names \
                --enable-termcap \
                --enable-weak-symbols \
                --enable-widec \
                --enable-xmc-glitch && \
        make && \
        make install
        echo "ncurses $(ncurses)" >> $(PREFIX)/version

openssl:
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: opensslfile=$(opensslfile)"
        echo "INFO: opensslsource=$(opensslsource)"
        test -f "$(opensslfile)" \
                && echo "INFO: Found $(opensslfile)" \
                || (echo "INFO: Retrieving $(opensslsource)"; curl -Lso "$(opensslfile)" "$(opensslsource)")
        tar -xf "$(opensslfile)"
        cd "$(openssldir)" && \
        ./config \
                --prefix="$(PREFIX)" \
                --openssldir="$(PREFIX)" \
                threads \
                shared \
                no-comp \
                "$(CFLAGS)" && \
        make depend && \
        make && \
        make install
        echo "OpenSSL $(openssl)" >> $(PREFIX)/version

python3: expat ffi mpdec openssl sqlite3
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: python3file=$(python3file)"
        echo "INFO: python3source=$(python3source)"
        test -f "$(python3file)" \
                && echo "INFO: Found $(python3file)" \
                || (echo "INFO: Retreiving $(python3source)"; curl -Lso "$(python3file)" "$(python3source)")
        tar -xf "$(python3file)"
        cd "$(python3dir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --enable-shared \
                --enable-optimizations \
                --enable-profiling \
                --enable-loadable-sqlite-extensions \
                --enable-ipv6 \
                --with-system-expat=$(PREFIX) \
                --with-system-ffi=$(PREFIX) \
                --with-system-libmpdec=$(PREFIX) \
                --with-openssl=$(PREFIX) \
                --with-ensurepip=upgrade \
                "CFLAGS=$(CFLAGS) $(PYTHON_LIBS)" \
                "CPPFLAGS=$(CPPFLAGS) $(PYTHON_LIBS)" \
                "LDFLAGS=$(LDFLAGS) $(PYTHON_LIBS)" && \
        make \
                "CFLAGS=$(CFLAGS) $(PYTHON_LIBS)" \
                "CPPFLAGS=$(CPPFLAGS) $(PYTHON_LIBS)" \
                "LDFLAGS=$(LDFLAGS) $(PYTHON_LIBS)" && \
        make install
        echo -n "INFO: Testing if Python runs: "
        $(PREFIX)/bin/python3 --version
        [[ -x find_deps.sh ]] && echo "Testing binary dependancies:" && ./find_deps.sh
        echo "python3 $(python3)" >> $(PREFIX)/version

readline: ncurses
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: readlinefile=$(readlinefile)"
        echo "INFO: readlinesource=$(readlinesource)"
        test -f "$(readlinefile)" \
                && echo "INFO: Found $(readlinefile)" \
                || (echo "INFO: Retrieving $(readlinesource)"; curl -Lso "$(readlinefile)" "$(readlinesource)")
        tar -xf "$(readlinefile)"
        cd "$(readlinedir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --with-curses="$(PREFIX)" \
                --enable-shared \
                --enable-static && \
        make && \
        make install
        echo "readline $(readline)" >> $(PREFIX)/version

sqlite3: readline zlib
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: sqlite3file=$(sqlite3file)"
        echo "INFO: sqlite3source=$(sqlite3source)"
        test -f "$(sqlite3file)" \
                && echo "INFO: Found $(sqlite3file)" \
                || (echo "INFO: Retreiving $(sqlite3source)"; curl -Lso "$(sqlite3file)" "$(sqlite3source)")
        tar -xf "$(sqlite3file)"
        cd "$(sqlite3dir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --enable-shared \
                --enable-static \
                --enable-readline \
                --enable-threadsafe \
                --enable-dynamic-extensions && \
        make && \
        make install
        echo "sqlite3 $(sqlite3)" >> $(PREFIX)/version

zlib:
        echo "INFO: CFLAGS=$(CFLAGS)"
        echo "INFO: CPPFLAGS=$(CPPFLAGS)"
        echo "INFO: PREFIX=$(PREFIX)"
        echo "INFO: zlibfile=$(zlibfile)"
        echo "INFO: zlibsource=$(zlibsource)"
        test -f "$(zlibfile)" \
                && echo "INFO: Found $(zlibfile)" \
                || (echo "INFO: Retrieving $(zlibsource)"; curl -Lso "$(zlibfile)" "$(zlibsource)")
        tar -xf "$(zlibfile)"
        cd "$(zlibdir)" && \
        ./configure \
                --prefix="$(PREFIX)" \
                --64 \
                --shared \
                --libdir="$(PREFIX)/lib" \
                --sharedlibdir="$(PREFIX)/lib" && \
        make && \
        make install
        echo "zlib $(zlib)" >> $(PREFIX)/version


clean: | \
        $(cleanexpat) \
        $(cleanffi) \
        $(cleanmpdec) \
        $(cleanncurses) \
        $(cleanopenssl) \
        $(cleanpython3) \
        $(cleanreadline) \
        $(cleansqlite3) \
        $(cleanzlib)
        @echo "INFO: Done cleaning"

$(cleanexpat):
        @echo "INFO: Removing $(expatdir)"
        rm -rf "$(expatdir)"

$(cleanffi):
        @echo "INFO: Removing $(ffidir)"
        rm -rf "$(ffidir)"

$(cleanmpdec):
        @echo "INFO: Removing $(mpdecdir)"
        rm -rf "$(mpdecdir)"

$(cleanncurses):
        @echo "INFO: Removing $(ncursesdir)"
        rm -rf "$(ncursesdir)"

$(cleanopenssl):
        @echo "INFO: Removing $(openssldir)"
        rm -rf "$(openssldir)"

$(cleanpython3):
        @echo "INFO: Removing $(python3dir)"
        rm -rf "$(python3dir)"

$(cleanreadline):
        @echo "INFO: Removing $(readlinedir)"
        rm -rf "$(readlinedir)"

$(cleansqlite3):
        @echo "INFO: Removing $(sqlite3dir)"
        rm -rf "$(sqlite3dir)"

$(cleanzlib):
        @echo "INFO: Removing $(zlibdir)"
        rm -rf "$(zlibdir)"
Pastebin has better syntax highlighting support for Make, you can find that >>here<<.



find_deps.sh

The script find_deps.sh is a short Bash script I threw together to make sure that everything is linked correctly. It searches through $PREFIX for all ELF binaries and uses env -i ldd to get a list of all libraries the file would pull in a vacuum, and lists any files that can't find one or more libraries using the baked-in rpath.

#!/usr/bin/env bash

[[ -z "$PREFIX" ]] && export PREFIX=/opt/utils/python3

pushd "$(dirname $0)" &>/dev/null
MYDIR=$(pwd)
popd &>/dev/null

declare -a MISSING

function lddd {
  env -i ldd "$1" 2>/dev/null
}

echo "Enumerating imported libraries:"
echo " Count - Library"
while read file; do
  if [[ $(file $file) =~ ELF ]]; then
    lddd "${file}"
    if [[ ! $file =~ ${MYDIR} && $(lddd "${file}") =~ "not found" ]]; then
      MISSING+=($file)
    fi
  fi
done < <(find ${PREFIX} -type f -name '*.so*' | sort) \
| awk '$3 ~ "^/"{print $3}' \
| sort \
| uniq -c \
| sort -k 2

echo "Missing libs:"
for file in "${MISSING[@]}"; do
  echo "${file}"
  lddd "${file}"
done
Reply
#2
How long does it last to build Python?
Reply
#3
(May-27-2019, 03:10 PM)heiner55 Wrote: How long does it last to build Python?

It varies by system. For me, it takes a bit less than an hour and a half to build everything.
Reply
#4
Thanks you.
Then I will try it also.
Reply
#5
Still compiling. My PC is too slow.
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020