This commit fixes the following error specific to AUFS:
/dockcross/entrypoint.sh: line 47: /usr/bin/gosu: Text file busy
/dockcross/entrypoint.sh: line 47: /usr/bin/gosu: Success
Thanks @sobolevn
See https://github.com/moby/moby/issues/13594#issuecomment-262441366
Backported from thewtex/docker-centos-build@26f7fd3
CentOS 5 is no longer supported and requests to obtain information from
the mirrors fails. However, the packages are still available on the
"Vault" mirrors.
Comment out the `mirrorlist` entries and set `enabled=0` in
CentOS-Base.repo, add entries for 5.11 in CentOS-Vault.repo and mark
them `enabled=1`.
This commit will fix installation of packages like nose that would otherwise
fail to install with error like the following:
Installing collected packages: nose
Exception:
Traceback (most recent call last):
[...]
Permission denied: '/opt/_internal/cpython-2.7.11-ucs2/man'
This commit will avoid the following error from being reported
on MacOSX:
```
docker run --rm dockcross/linux-armv7 > ./dockcross-linux-armv7
chmod +x ./dockcross-linux-armv7
./dockcross-linux-armv7 bash -c '$CC test/C/hello.c -o hello_arm'
grep: /proc/version: No such file or directory
grep: /proc/version: No such file or directory
```
Reported-by: Chris Warth <cswarth@gmail.com>
This commit follows up on 2e71db2 (dockcross: Ignore deletion error
when running in unprivileged LXC container) by hiding the message
only when (a) the command output matches the expected error message
and (b) is executed on CircleCI.
Doing so will avoid adding "noise" to log of service like CircleCI
that are effectively running docker in unprivileged LXC container.
See #50
As recommended in by CircleCI, explicitly specify the "latest"
tag. This will be important when we will start versioning our images.
Copied from CircleCI documentation [1]:
"Make sure you tag the image you use in the FROM command in your
Dockerfile, even if it is the default “latest” tag. Otherwise, Docker
will pull down all tags for the image you specify. This will create
significant performance problems because the save/load strategy described
above only caches the image layers (and thus tags) that you specify in
the docker save command, so other tags will be re-pulled on every build
if a tag is not specified in the FROM command."
[1] https://circleci.com/docs/docker/#caching-docker-layers
To workaround, CircleCI limitation [1]:
"Please note that since we do not support parallel deployment, specifying
‘parallel:true’ in the deployment phase will cause an error."
This commit moves the deployment step as the last action of the test
step. Doing so will ensure, every container will take care of pushing
images it was responsible to build.
[1] https://circleci.com/docs/parallel-manual-setup/#simple-example
* style: Rename TASK to STEP
* do not pull base image after it was copied from node0 and loaded.
It caused the current Dockerfile to be out-of-sync with the current image.
* do not pull images after loading the downloaded and up-to-date
base.tar. It was overwriting the base.tar downloaded and loaded.
* save debian:jessie into base.tar
* use "time" command to help identify long running operation
* rename marker file from "BASE_READY" to "BASE_AVAILABLE"
* build base image only if needed.
This commit updates the "browser-asmjs" images to avoid systematic
rebuild of emscripten sdk. Instead, it introduces dependency on the
image maintained by @trzeci
Note also:
* emscripten sdk updated from 1.36.7 to 1.36.14
* symlinking of "/bin/bash" as "/bin/sh" is reverted. This ensures other
images including file like "common.debian" and relying on standard "sh"
behavior will work as expected.
* workaround associated with CMakeForceCompiler has been removed from
Dockerfile. Emscripten as been fixed.
See https://github.com/kripken/emscripten/pull/4477
* we will revisit once official images are available.
See https://github.com/kripken/emscripten/issues/4682
To accommodate the requirements associated with x86 and x64 images, the
command building OpenSSL and CMake became overly complex and hard to
maintain.
This commit has multiple purposes:
(1) simplify common.docker
(2) fix the building of 64-bit shared libraries against the static openssl
libraries by passing the -fPIC flag.
(3) ensure [many]linux-x86 and [many]linux-x64 images have an up-to-date
OpenSSL install. Openssl static libraries are installed in /usr
(4) simplify and speedup CMake build avoiding the second build with
explicit -DCMAKE_USE_OPENSSL:BOOL=ON. Indeed, configuring CMake on Linux
already looks for OpenSSL.
(5) speedup download of CMake source directly downloading the archive
corresponding to the revision.
(6) test CMake by:
- running CMake.FileDownload test
- trying to download a file served over https
This commit fixes the following error reported when building the base
image.
sh: 1: [: =: unexpected operator
sh: 1: [: =: unexpected operator
It turns out that (1) DEFAULT_DOCKCROSS_IMAGE is not set when building the
base image and (2) latest openssl will not be available in linux-x64 and
linux-x86 images.
While the use of sudo (made possible by 53cf084) allows to install
additional packages, the warning copied below was still reported.
To avoid this warning and streamline the installation of new packages,
this commit (1) introduces the concept of "pre_exec" entrypoint hook
and (2) adds such a hook to change the ownership of python "bin" and
"site-packages" directories for the manylinux images.
Warning reported are similar to this one:
```
The directory '/home/jcfr/.cache/pip/http' or its parent directory is
not owned by the current user and the cache has been disabled. Please
check the permissions and owner of that directory. If executing pip
with sudo, you may want sudo's -H flag.
```
Note that the sudo "-H" flag suggested in the warning is not available
in centos.
This commit build each images with the following arguments:
* IMAGE: Name of the image (e.g dockcross/base, dockcross/manylinux-x64, ...)
* VCS_REF: dockcross/dockcross commit from which this image is built
* VCS_URL: this repository obtained reading remote.origin.url
* BUILD_DATE: Date and time when the build was initiated
Then, within the Dockerfile, the metadata are associated with the image
using the "LABEL" instruction.
See https://docs.docker.com/engine/reference/builder/#/label
The corresponding labels can be found here:
http://label-schema.org/rc1/#build-time-labels
See #28
This commit removes the complex pipeline leveraging output from
/dev/urandom and simply uses $RANDOM variable. For the purpose
of generating an available container name, it will works well.
Suggested-by: Matt McCormick <matt.mccormick@kitware.com>
As explained in [1], when using dockcross from python subprocess, the
interactive mode is disabled and output of "cat /dev/urandom" is block
buffered instead of being line buffered.
Workaround to this problem is to simply read a fixed amount of characters
from urandom.
The following two snippets illustrates the problem and the
the implemented solution:
Works
```python
import subprocess as sp
sp.check_call("var=$(head -c 500 /dev/urandom | base64 | tr -dc 'a-zA-Z0-9' | fold -w 7 | head -n 1); echo $var", shell=True)
```
=> output random string
Fail:
```python
import subprocess as sp
sp.check_call("var=$(cat /dev/urandom | base64 | tr -dc 'a-zA-Z0-9' | fold -w 7 | head -n 1); echo $var", shell=True)
```
=> Hang
[1] http://stackoverflow.com/questions/16805827/unable-to-read-stdout-from-a-running-process#16806506
where RAND is a 7 characters long randomly generated alphanumeric string.
Note that container name can NOT start with "_",
only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed
See #50
This commit fixes a regression [1] introduced in 606d0b3 (Makefile: Make sure
BIN directory exist when running any test)
Reported below the dry-run summary before and after, we can confirm the
bin directory is now created when building the `base.test` target.
Before:
$ make base.test --dry-run
docker build -t dockcross/base .
docker run --rm dockcross/base > bin/dockcross-base && chmod +x bin/dockcross-base
After:
$ make base.test --dry-run
docker build -t dockcross/base .
mkdir -p bin
docker run --rm dockcross/base > bin/dockcross-base && chmod +x bin/dockcross-base
[1] Error fixed by this commit:
Successfully built e6388fe49824
docker run --rm dockcross/base > bin/dockcross-base && chmod +x bin/dockcross-base
/bin/sh: 1: cannot create bin/dockcross-base: Directory nonexistent
make: *** [base.test] Error 2
make base.test returned exit code 2
This commit also ensures that:
* all image targets are declared as PHONY
* "test" and not "tests" is declared as PHONY
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared.
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
make browser-asmjs.test --dry-run > ../dockcross-make-browser-asmjs-test-baseline
make linux-ppc64le.test --dry-run > ../dockcross-make-linux-ppc64le-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
make browser-asmjs.test --dry-run > ../dockcross-make-browser-asmjs-test-current
make linux-ppc64le.test --dry-run > ../dockcross-make-linux-ppc64le-test-current
for target in make make-test make-browser-asmjs-test make-linux-ppc64le-test; do
diff --ignore-trailing-space ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff --ignore-trailing-space ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared.
The only difference was due to the extra space in the command line
associated with windows-x86
```
< bin/dockcross-windows-x86 python test/run.py --exe-suffix ".exe"
---
> bin/dockcross-windows-x86 python test/run.py --exe-suffix ".exe"
```
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff --ignore-trailing-space ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
Adding a new images with the usual build rule:
$(DOCKER) build -t $(ORG)/$@ $@
is now just a matter of adding an item to the "STANDARD_IMAGES" list.
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared.
Note that since this patch changes the order in which the images are
listed (manylinux is now last), the baseline and current files are
different. After inspection, we can confirm the difference is explained
by the order of the commands.
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
Since image targets are all "phony", they are always consider out-of-date
and will always be rebuilt. This means there is no need to add an explicit
dependency on non-generated files like "Dockerfile", "settings.mk" and
"test/run.py".
The only required dependencies on Dockerfile are the one for the "base"
and "manylinux" images.
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
Since image targets are all "phony", they are always consider out-of-date
and will always be rebuilt. This means there is no need to add an explicit
dependency on the "Toolchain.cmake" file.
Note also that the dependency on "Toolchain.cmake" was not consistent
for all images. For example, "linux-x64" was missing the dependency
whereas it was specified for "linux-x86".
This commit ensure all image targets are consistent removing the
dependency on "Toolchain.cmake".
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
See https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html#File-Name-Functions
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:
Before:
```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```
After:
```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
for target in make make-test; do
diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
[[ $? == 1 ]] && \
echo "" && \
echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
This will allow build script like "build-wheels.sh" to run command like
the following (see [1]):
sudo yum install -y atlas-devel
It will also allow to pip install using sudo and avoid error reported in [3].
That said, instead of running pip using sudo, the recommended approach
is to specify the "--user" flag [2]
[1] https://github.com/pypa/python-manylinux-demo/blob/893d92517e9e289b9d2ee063f19c3216a39534ab/travis/build-wheels.sh
[2] http://stackoverflow.com/questions/7143077/how-can-i-install-packages-in-my-home-folder-with-pip#7143496
[3] Error reported when running pip without "sudo pip install <packagename" or "pip install --user <packagename>"
```
Exception:
Traceback (most recent call last):
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
prefix=options.prefix_path,
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/req/req_set.py", line 736, in install
requirement.uninstall(auto_confirm=True)
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/req/req_install.py", line 742, in uninstall
paths_to_remove.remove(auto_confirm)
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/shutil.py", line 303, in move
os.unlink(src)
OSError: [Errno 13] Permission denied: '/opt/_internal/cpython-2.7.11-ucs2/bin/easy_install'
```
By default the toolchain file is always passed as an option to cmake/ccmake
It can be explicitly disabled setting the environment variable
DOCKCROSS_PASS_CMAKE_TOOLCHAIN_FILE to 0
This commits add test targets for all images, this will facilitate local
testing.
In addition to the usual targets:
make android-arm
make browser-asmjs
[...]
the following targets are now available:
make tests
make android-arm.test
make browser-asmjs.test
[...]
Note that the target "all" has been renamed into "images"
