Commit Graph

1056 Commits

Author SHA1 Message Date
Matt McCormick
a7c894ab50
doc: Use MicroBadger badges 2016-11-01 10:00:15 -04:00
Matt McCormick
fdb3a52a01
doc: variable -> variables 2016-11-01 09:51:20 -04:00
Matt McCormick
d7ac739c2e Merge pull request #48 from jcfr/keep-track-buildtime-metadata
dockerfile: Keep track of buildtime metadata
2016-11-01 07:28:35 -04:00
Matt McCormick
95f2462567 Merge pull request #53 from thewtex/scikit-build-0.4.0
manylinux-x{64,86}: Bump scikit-build to 0.4.0
2016-10-31 21:21:32 -04:00
Matt McCormick
623148e6e5
manylinux-x{64,86}: Bump scikit-build to 0.4.0 2016-10-31 18:40:36 -04:00
Matt McCormick
cb95cf3c94 Merge pull request #52 from jcfr/dockcross-ignore-rm-error-when-running-in-unprivileged-lxc-container-no-tty-support
dockcross: Prevent dockcross from blocking when used with Python subprocess
2016-10-31 00:12:45 -04:00
Jean-Christophe Fillion-Robin
cfcc7d6700
dockerfile: Keep track of buildtime metadata
This commit build each images with the following arguments:

* IMAGE: Name of the image (e.g dockcross/base, dockcross/manylinux-x64, ...)
* VCS_REF: dockcross/dockcross commit from which this image is built
* VCS_URL: this repository obtained reading remote.origin.url
* BUILD_DATE: Date and time when the build was initiated

Then, within the Dockerfile, the metadata are associated with the image
using the "LABEL" instruction.
See https://docs.docker.com/engine/reference/builder/#/label

The corresponding labels can be found here:
http://label-schema.org/rc1/#build-time-labels

See #28
2016-10-30 23:57:04 -04:00
Jean-Christophe Fillion-Robin
f31e582f80
dockcross: Simplify container name generation using $RANDOM
This commit removes the complex pipeline leveraging output from
/dev/urandom and simply uses $RANDOM variable. For the purpose
of generating an available container name, it will works well.

Suggested-by: Matt McCormick <matt.mccormick@kitware.com>
2016-10-30 23:46:01 -04:00
Jean-Christophe Fillion-Robin
bd234af0a9
dockcross: Prevent dockcross from blocking when used with Python subprocess
As explained in [1], when using dockcross from python subprocess, the
interactive mode is disabled and output of "cat /dev/urandom" is block
buffered instead of being line buffered.

Workaround to this problem is to simply read a fixed amount of characters
from urandom.

The following two snippets illustrates the problem and the
the implemented  solution:

Works

```python
import subprocess as sp
sp.check_call("var=$(head -c 500 /dev/urandom | base64 | tr -dc 'a-zA-Z0-9' | fold -w 7 | head -n 1); echo $var", shell=True)
```

=> output random string


Fail:

```python
import subprocess as sp
sp.check_call("var=$(cat /dev/urandom | base64 | tr -dc 'a-zA-Z0-9' | fold -w 7 | head -n 1); echo $var", shell=True)
```

=> Hang

[1] http://stackoverflow.com/questions/16805827/unable-to-read-stdout-from-a-running-process#16806506
2016-10-30 03:27:10 -04:00
Matt McCormick
e378a9b336 Merge pull request #51 from jcfr/dockcross-ignore-rm-error-when-running-in-unprivileged-lxc-container
dockcross: Ignore deletion error when running in unprivileged LXC container
2016-10-27 13:34:45 -04:00
Jean-Christophe Fillion-Robin
3288c877e5
dockcross: Fix "tr: Illegal byte sequence" when generating container name on MacOSx
See #50
2016-10-27 13:20:31 -04:00
Jean-Christophe Fillion-Robin
c5edcb3a74
dockcross: Ensure exit code associated with run command is always returned 2016-10-27 11:37:07 -04:00
Jean-Christophe Fillion-Robin
904cb09858
dockcross: Change name of container from "dockcross" to "dockcross_RAND"
where RAND is a 7 characters long randomly generated alphanumeric string.

Note that container name can NOT start with "_",
only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed

See #50
2016-10-27 11:37:03 -04:00
Jean-Christophe Fillion-Robin
7a8f4fab0e
dockcross: Do not output any text after container is removed
See #50
2016-10-27 11:19:37 -04:00
Matt McCormick
195827d40d Merge pull request #49 from thewtex/mxe-update
windows-x{64,86}: Bump MXE to 2016-10-25
2016-10-27 11:12:17 -04:00
Jean-Christophe Fillion-Robin
2e71db2234
dockcross: Ignore deletion error when running in unprivileged LXC container
This commit workarounds the problem described in [1] and [2] by ignoring error
happening when deleting container in unprivileged LXC container.

Fixes #50

[1] https://circleci.com/docs/docker-btrfs-error/
[2] https://discuss.circleci.com/t/docker-error-removing-intermediate-container/70
2016-10-26 23:24:25 -04:00
Matt McCormick
80993ad985 windows-x{64,86}: Bump MXE to 2016-10-25 2016-10-26 17:12:50 -04:00
Matt McCormick
32704fb4e4 Merge pull request #47 from jcfr/fix-circle-ci
Fix circle ci
2016-09-26 13:57:13 -04:00
Jean-Christophe Fillion-Robin
97385c1356
Makefile: Make sure base.test also create bin directory
This commit fixes a regression [1] introduced in 606d0b3 (Makefile: Make sure
BIN directory exist when running any test)

Reported below the dry-run summary before and after, we can confirm the
bin directory is now created when building the `base.test` target.

Before:

$ make base.test --dry-run
docker build -t dockcross/base .
docker run --rm dockcross/base > bin/dockcross-base && chmod +x bin/dockcross-base

After:

$ make base.test --dry-run
docker build -t dockcross/base .
mkdir -p bin
docker run --rm dockcross/base > bin/dockcross-base && chmod +x bin/dockcross-base


[1] Error fixed by this commit:

Successfully built e6388fe49824
docker run --rm dockcross/base > bin/dockcross-base && chmod +x bin/dockcross-base
/bin/sh: 1: cannot create bin/dockcross-base: Directory nonexistent
make: *** [base.test] Error 2

make base.test returned exit code 2
2016-09-26 11:38:43 -04:00
Jean-Christophe Fillion-Robin
cf3cec8819
circle: Fix error in dependencies/override - Add missing 'done' 2016-09-26 10:14:27 -04:00
Matt McCormick
89b49dea22 Merge pull request #45 from jcfr/tweak-makefile
Tweak makefile
2016-09-26 10:01:31 -04:00
Jean-Christophe Fillion-Robin
606d0b3ed9
Makefile: Make sure BIN directory exist when running any test
This commit allows to run any test directly after checking out
the source code.
2016-09-26 00:21:16 -04:00
Jean-Christophe Fillion-Robin
79938b37e5
circle.yml: Simplify reusing 'display_images' target 2016-09-25 23:55:17 -04:00
Jean-Christophe Fillion-Robin
8a4e0845da
Makefile: All images are tested. Remove ALL_IMAGES and DEFAULT_IMAGES vars.
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after has been compared.

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff --ignore-trailing-space ../dockcross-$target-baseline ../dockcross-$target-current> /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "-------------------------------" && \
    echo "Here is output of:" && \
    echo "  diff --ignore-trailing-space ../dockcross-$target-baseline ../dockcross-$target-current" && \
    echo "" && \
    diff --ignore-trailing-space ../dockcross-$target-baseline ../dockcross-$target-current
done
```

Output:

```
-------------------------------
Here is output of:
  diff --ignore-trailing-space ../dockcross-make-baseline ../dockcross-make-current

8a9
> docker build -t dockcross/linux-ppc64le linux-ppc64le
10a12,14
> cp -r test browser-asmjs/
> docker build -t dockcross/browser-asmjs browser-asmjs
> rm -rf browser-asmjs/test

-------------------------------
Here is output of:
  diff --ignore-trailing-space ../dockcross-make-test-baseline ../dockcross-make-test-current

24a25,27
> docker build -t dockcross/linux-ppc64le linux-ppc64le
> docker run --rm dockcross/linux-ppc64le > bin/dockcross-linux-ppc64le && chmod +x bin/dockcross-linux-ppc64le
> bin/dockcross-linux-ppc64le python test/run.py --languages C
30a34,38
> cp -r test browser-asmjs/
> docker build -t dockcross/browser-asmjs browser-asmjs
> rm -rf browser-asmjs/test
> docker run --rm dockcross/browser-asmjs > bin/dockcross-browser-asmjs && chmod +x bin/dockcross-browser-asmjs
> bin/dockcross-browser-asmjs python test/run.py --exe-suffix ".js"

```
2016-09-25 23:28:23 -04:00
Jean-Christophe Fillion-Robin
4caa5a367f
README: Add "Download all images" and "+Install all dockcross scripts" section 2016-09-25 19:47:23 -04:00
Jean-Christophe Fillion-Robin
aa4735ddf2
Makefile: Add "display_default_images" and "display_all_images" targets 2016-09-25 19:46:49 -04:00
Jean-Christophe Fillion-Robin
4a0c6442c2
Makefile: Introduce NON_STANDARD_IMAGES, ALL_IMAGES and DEFAULT_IMAGES vars
This commit also ensures that:
* all image targets are declared as PHONY
* "test" and not "tests" is declared as PHONY


To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared.


Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
make browser-asmjs.test --dry-run > ../dockcross-make-browser-asmjs-test-baseline
make linux-ppc64le.test --dry-run > ../dockcross-make-linux-ppc64le-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current
make browser-asmjs.test --dry-run > ../dockcross-make-browser-asmjs-test-current
make linux-ppc64le.test --dry-run > ../dockcross-make-linux-ppc64le-test-current

for target in make make-test make-browser-asmjs-test make-linux-ppc64le-test; do
  diff --ignore-trailing-space ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 19:04:07 -04:00
Jean-Christophe Fillion-Robin
b033376284
Makefile: Add comments and move implicit rules at the end of the file
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff --ignore-trailing-space ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 18:38:26 -04:00
Jean-Christophe Fillion-Robin
0710c46745
Makefile: Treat windows-(x86|x64) as regular image introducing _ARGS var
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared.

The only difference was due to the extra space in the command line
associated with windows-x86

```
< bin/dockcross-windows-x86 python test/run.py  --exe-suffix ".exe"
---
> bin/dockcross-windows-x86 python test/run.py --exe-suffix ".exe"
```


Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff --ignore-trailing-space ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 18:20:02 -04:00
Jean-Christophe Fillion-Robin
f7f6989e46
Makefile: Add implicit rule for STANDARD_TESTS
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 18:03:28 -04:00
Jean-Christophe Fillion-Robin
53b158314a
Makefile: Add implicit rule for STANDARD_IMAGES
Adding a new images with the usual build rule:

  $(DOCKER) build -t $(ORG)/$@ $@

is now just a matter of adding an item to the "STANDARD_IMAGES" list.


To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared.

Note that since this patch changes the order in which the images are
listed (manylinux is now last), the baseline and current files are
different. After inspection, we can confirm the difference is explained
by the order of the commands.

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 17:44:08 -04:00
Jean-Christophe Fillion-Robin
2a12475d36
Makefile: Remove dependencies on a non-generated files
Since image targets are all "phony", they are always consider out-of-date
and will always be rebuilt. This means there is no need to add an explicit
dependency on non-generated files like  "Dockerfile", "settings.mk" and
"test/run.py".

The only required dependencies on Dockerfile are the one for the "base"
and "manylinux" images.

To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 17:38:20 -04:00
Jean-Christophe Fillion-Robin
caba0b0cef
Makefile: Remove "Toolchain.cmake" dependencies
Since image targets are all "phony", they are always consider out-of-date
and will always be rebuilt. This means there is no need to add an explicit
dependency on the "Toolchain.cmake" file.

Note also that the dependency on "Toolchain.cmake" was not consistent
for all images. For example, "linux-x64" was missing the dependency
whereas it was specified for "linux-x86".

This commit ensure all image targets are consistent removing the
dependency on "Toolchain.cmake".

To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 17:38:12 -04:00
Jean-Christophe Fillion-Robin
e428bb3c03
Makefile: Simplify test rule using "addsuffix" filename function
See https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html#File-Name-Functions

To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 17:38:05 -04:00
Jean-Christophe Fillion-Robin
92b57f511d
Makefile: Introduce IMAGES variable
To confirm that the Makefiles works as expected after this patch, the
list of executed commands before and after is compared:

Before:

```
make --dry-run > ../dockcross-make-baseline
make test --dry-run > ../dockcross-make-test-baseline
```

After:

```
make --dry-run > ../dockcross-make-current;
make test --dry-run > ../dockcross-make-test-current

for target in make make-test; do
  diff ../dockcross-$target-current ../dockcross-$target-baseline > /dev/null 2>&1
  [[ $? == 1 ]] && \
    echo "" && \
    echo "Error: Problem with '${target}' target: Dry-run output before and after this commit do not match."
done
```
2016-09-25 17:37:08 -04:00
Matt McCormick
dda710167a Merge pull request #44 from thewtex/manylinux-x86
Add manylinux-x86
2016-09-24 23:36:09 -04:00
Matt McCormick
1354fe2586 manylinux-x86: Initial addition 2016-09-24 23:14:33 -04:00
Matt McCormick
fd2b8efe51 linux-x86: Use linux32 as entrypoint for uname -m output 2016-09-24 22:37:00 -04:00
Matt McCormick
01a3348f15 Merge pull request #43 from jcfr/support-ctrl-c
base/dockcross: Run docker using -t to support CTRL-C
2016-09-23 11:36:56 -04:00
Jean-Christophe Fillion-Robin
af75453ed0 base/dockcross: Run docker with "-ti" only if a tty or pts is attached
This commit reverts f07ca00 (base: Do not add -i -t flags to run command
by default) and ensure tty is available before specifying the flags.
2016-09-23 11:33:29 -04:00
Jean-Christophe Fillion-Robin
aa2c24cb3b base/dockcross: Run docker using -t to support CTRL-C 2016-09-22 23:34:08 -04:00
Matt McCormick
1d11afd37d Merge pull request #42 from jcfr/manylinux-allow-pip-install-squashed
manylinux: Grant current user password less sudo access.
2016-09-22 22:45:17 -04:00
Jean-Christophe Fillion-Robin
995c9091e5 manylinux: Grant current user password less sudo access.
This will allow build script like "build-wheels.sh" to run command like
the following (see [1]):

  sudo yum install -y atlas-devel

It will also allow to pip install using sudo and avoid error reported in [3].

That said, instead of running pip using sudo, the recommended approach
is to specify the "--user" flag [2]

[1] 893d92517e/travis/build-wheels.sh

[2] http://stackoverflow.com/questions/7143077/how-can-i-install-packages-in-my-home-folder-with-pip#7143496

[3] Error reported when running pip without "sudo pip install <packagename" or "pip install --user <packagename>"
```
Exception:
Traceback (most recent call last):
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/opt/_internal/cpython-2.7.11-ucs2/lib/python2.7/shutil.py", line 303, in move
    os.unlink(src)
OSError: [Errno 13] Permission denied: '/opt/_internal/cpython-2.7.11-ucs2/bin/easy_install'
```
2016-09-22 19:21:24 -04:00
Matt McCormick
131e6408ee Merge pull request #38 from jcfr/36-support-disabling-toolchain-cmake-option
36 support disabling toolchain cmake option
2016-09-22 11:23:09 -04:00
Matt McCormick
3c1a2758b0 Merge pull request #40 from thewtex/39-manylinux-cmake-ssl
manylinux-x64,base: Download cmake binary for SSL support
2016-09-22 11:22:24 -04:00
Jean-Christophe Fillion-Robin
4876f14356 base/cmake: Do not pass toolchain option when using incompatible cmake option
Also revert the changes to ccmake.sh, the restriction doesn't apply to it.

Suggested-by: Matt McCormick <matt.mccormick@kitware.com>
2016-09-21 21:11:33 -04:00
Matt McCormick
72ffa6dd1d Merge pull request #35 from jcfr/34-add-support-for-special-update-commands
34 add support for special update commands
2016-09-21 20:34:01 -04:00
Jean-Christophe Fillion-Robin
c05d52310a dockcross: Ensure running "update commands" exits script with correct status 2016-09-21 20:25:11 -04:00
Matt McCormick
b1037c9f5f manylinux-x64,base: Download cmake binary for SSL support
Download the CMake binary from CMake.org for SSL support. #39
2016-09-21 20:25:02 -04:00
Jean-Christophe Fillion-Robin
ff2512b785 manylinux-x64: Set DOCKCROSS_PASS_CMAKE_TOOLCHAIN_FILE to 0
Fixes #36
2016-09-21 19:09:41 -04:00