Page MenuHomeAleph Objects Inc

Mirror Upstream Cura Sources
Closed, ResolvedPublic

Description

Description of issue:

When building Cura a lot of source is pulled from upstream repos, such as Github. We need to mirror all of that to our own servers, and change the build to pull from our repos. So when a build is done, it shouldn't hit any other outside repos. The mirrors need to be set up on code.alephobjects.com and the build CMakeLists.txt needs to be updated to new URLs.

We also need to mirror source that doesn't have a git repo by copying the source tarball to our servers. In some cases (such as six), we are pulling a tarball, but a git repo is available, so we should switch to that. In other cases, there is a Mercurial (hg) or Subversion (svn) upstream repo. And in other cases, there is only a tarball available. (Perhaps we should create our own git repo if there is no upstream tracking in git/svn/hg and just untar and tag into that. It keeps all the source on code.ao without having to use download.ao too).

Git Mirrors:
https://github.com/ActiveState/appdirs
https://code.alephobjects.com/source/appdirs/

https://github.com/pypa/setuptools
https://code.alephobjects.com/source/setuptools/

https://github.com/nose-devs/nose
https://code.alephobjects.com/source/nose/

https://github.com/numpy/numpy
https://code.alephobjects.com/source/numpy/

https://github.com/pypa/packaging
https://code.alephobjects.com/source/packaging/

https://github.com/scipy/scipy
https://code.alephobjects.com/source/scipy/

https://github.com/xianyi/OpenBLAS
https://code.alephobjects.com/source/OpenBLAS/

https://github.com/anthony-tuininga/cx_Freeze
https://code.alephobjects.com/source/cx_Freeze/

https://github.com/google/protobuf
https://code.alephobjects.com/source/protobuf/

https://github.com/jstasiak/python-zeroconf
https://code.alephobjects.com/source/python-zeroconf/

https://github.com/ericfaurot/py-app
https://code.alephobjects.com/source/py-app/
This was mirrored, but py2app is needed, not this. Repo de-activated.

Tarball Mirrors
https://sourceforge.net/projects/pyqt/files/PyQt5/PyQt-5.8.1/PyQt5_gpl-5.8.1.tar.gz/download
Upstream for this only publishes tarballs, as far as I can tell. They publish release and development tarballs.
A git archive was made of all the PyQt5 versions released so far. There is a git tag for each version. This repo will need to be updated manually when new tarballs are released.
https://code.alephobjects.com/source/PyQt5_gpl/
git tag 5.8.1
git hash 815193dc7c0219de561e7acb332540e784341c59

http://downloads.sourceforge.net/project/pyqt/sip/sip-${SIP_VERSION_STR}/sip-${SIP_VERSION_STR}.zip
All builds use version 4.19.3.
This is in a Mercurial (hg) repo:
https://www.riverbankcomputing.com/hg/sip
https://code.alephobjects.com/source/sip/
hg tag 4.19.3
hg hash 14685a6e736e
https://download.lulzbot.com/Software/Cura2/build-resources/sip-4.19.3.zip

https://pypi.python.org/packages/d9/ff/48a8aede5e99db5983039fd0e6c54bb4c0806ef696430d2684348e78249f/py2app-0.12.tar.gz
Mercurial (hg):
https://bitbucket.org/ronaldoussoren/py2app
https://code.alephobjects.com/source/py2app/
hg tag py2app-0.12
hg hash 6e21535e4832
https://download.lulzbot.com/Software/Cura2/build-resources/py2app-0.12.tar.gz

https://pypi.python.org/packages/b0/4f/93d8b4beac7b686c3f1a8f682062a9e65c87e5e45a99863ea7e4ba43813f/pyobjc-core-4.0.tar.gz
This is in a Mercurial (hg) repo:
https://bitbucket.org/ronaldoussoren/pyobjc
https://code.alephobjects.com/source/pyobjc/
https://download.lulzbot.com/Software/Cura2/build-resources/pyobjc-core-4.0.tar.gz

https://pypi.python.org/packages/a7/8d/fc2b9d569ba23775724849eeed39a53665b3b1058eac00c3d3ac3a54e3cc/pyobjc-framework-Cocoa-4.0.tar.gz
This is in a Mercurial (hg) repo:
https://bitbucket.org/ronaldoussoren/pyobjc
https://code.alephobjects.com/source/pyobjc/
https://download.lulzbot.com/Software/Cura2/build-resources/pyobjc-framework-Cocoa-4.0.tar.gz

https://www.python.org/ftp/python/${PYTHON_VERSION_STRING}/Python-${PYTHON_VERSION_STRING}.tgz
Currently version 3.5.3 is used for GNU/Linux and OSX. Version 3.5.2 for Windows.
https://github.com/python/cpython.git
https://code.alephobjects.com/source/cpython/
Matches (untested):
git tag v3.5.2
git hash 6c1aa6ba30e38fb9708526e27f9158162c37d909
git tag v3.5.3
git hash 51ba5b7d0c194b0c2b1e5d647e70e3538b8dde3e
Copied to lulzbot:
https://download.lulzbot.com/Software/Cura2/build-resources/Python-3.5.2.tgz
https://download.lulzbot.com/Software/Cura2/build-resources/Python-3.5.3.tgz

https://pypi.python.org/packages/b3/b2/238e2590826bfdd113244a40d9d3eb26918bd798fc187e2360a8367068db/six-1.10.0.tar.gz
https://github.com/benjaminp/six
https://code.alephobjects.com/source/six/
Matches (untested):
git tag 1.10.0
git hash c3ec058bf8c4d6329224eac53366c8e9ce511417

https://downloads.sourceforge.net/project/pyparsing/pyparsing/pyparsing-2.2.0/pyparsing-2.2.0.tar.gz
https://github.com/pyparsing/pyparsing
https://code.alephobjects.com/source/pyparsing/
Matches (untested):
git tag pyparsing_2.2.0
git hash d7b0aa17d3a300cb64b5796b681eb63e856247a8
This fails because the files in the git repo are in a src/ subdirectory.
Tarball copied to download.lulzbot.com:
https://download.lulzbot.com/Software/Cura2/build-resources/pyparsing-2.2.0.tar.gz

https://pypi.python.org/packages/18/fa/dd13d4910aea339c0bb87d2b3838d8fd923c11869b1f6e741dbd0ff3bc00/netifaces-0.10.4.tar.gz
https://github.com/al45tair/netifaces
https://code.alephobjects.com/source/netifaces/
Matches (untested):
git tag release_0_10_4
git hash d3531a51f336e19f0dd2c54102ee79078fd16623

https://pypi.python.org/packages/b7/67/7e2a817f9e9c773ee3995c1e15204f5d01c8da71882016cac10342ef031b/Cython-0.25.2.tar.gz
https://github.com/cython/cython
https://code.alephobjects.com/source/cython/
Matches (untested):
git tag 0.25.2
git hash c9bcf1bed3acf367d6deb0c273cf22db0f18dab2

https://github.com/materialsvirtuallab/pyhull/archive/v1.5.0.tar.gz
https://github.com/materialsvirtuallab/pyhull
https://code.alephobjects.com/source/pyhull/
Matches (untested):
git tag v1.5.0
git hash e1ef80c0170eeba67f9befdae8e5c627ac656596

https://github.com/pyserial/pyserial/releases/download/v3.2.1/pyserial-3.2.1.tar.gz
https://github.com/pyserial/pyserial
https://code.alephobjects.com/source/pyserial/
Matches (untested):
git tag v3.2.1
git hash ffb44d6d7394e2b73bfb1bb91dd69c377c0f8aa0

Subversion (svn) Mirrors
None. I think the only Subversion mirror was actually empty.

Tarballs to mirror:
https://pypi.python.org/packages/17/75/3698d7992a828ad6d7be99c0a888b75ed173a9280e53dbae67326029b60e/typing-3.6.1.tar.gz
Is this even used? After python 3.5, it is included in python itself.
https://pypi.org/project/typing/
"NOTE: in Python 3.5 and later, the typing module lives in the stdlib, and installing this package has NO EFFECT."
The source to this is in the cpython git archive.
https://download.lulzbot.com/Software/Cura2/build-resources/typing-3.6.1.tar.gz

https://downloads.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9-7-bin.zip
This Subversion (svn) repo is empty:
svn://svn.code.sf.net/p/gnuwin32/code/trunk gnuwin32-code
https://svn.code.sf.net/p/gnuwin32/code/trunk gnuwin32-code
Not syncing. This Subversion repo doesn't have any commits.
https://code.alephobjects.com/source/gnuwin32-code/
The matching tarball:
https://downloads.sourceforge.net/project/gnuwin32/patch/2.5.9-7/patch-2.5.9-7-src.zip
Copied to:
https://download.lulzbot.com/Software/Cura2/build-resources/patch-2.5.9-7-src.zip
https://download.lulzbot.com/Software/Cura2/build-resources/patch-2.5.9-7-bin.zip

http://download.lulzbot.com/Software/Cura2/build-resources/qt-everywhere-opensource-src-5.9.1.tar.gz
This is a huge repo, with many 42 submodule repos. Submodules are mirrored. QT's submodule script automagickally uses our URL.
https://code.qt.io/qt/qt5.git
https://code.alephobjects.com/source/qt5/
git tag v5.9.1
git hash d07943b32239ed52807b8f115cdc4e40e75a952a
Still use tarball for now. Downloading all git repos is approximately 5 gigs.
https://download.lulzbot.com/Software/Cura2/build-resources/qt-everywhere-opensource-src-5.9.1.tar.gz

Git QT5 Submodules to Mirror:
These are all of qt5's submodules:

https://code.qt.io/qt/qt5.git
https://code.alephobjects.com/source/qt5/

https://code.qt.io/qt/qt3d.git
https://code.alephobjects.com/source/qt3d/

https://code.qt.io/qt/qtactiveqt.git
https://code.alephobjects.com/source/qtactiveqt/

https://code.qt.io/qt/qtandroidextras.git
https://code.alephobjects.com/source/qtandroidextras/

https://code.qt.io/qt/qtbase.git
https://code.alephobjects.com/source/qtbase/

https://code.qt.io/qt/qtcanvas3d.git
https://code.alephobjects.com/source/qtcanvas3d/

https://code.qt.io/qt/qtcharts.git
https://code.alephobjects.com/source/qtcharts/

https://code.qt.io/qt/qtconnectivity.git
https://code.alephobjects.com/source/qtconnectivity/

https://code.qt.io/qt/qtdatavis3d.git
https://code.alephobjects.com/source/qtdatavis3d/

https://code.qt.io/qt/qtdeclarative.git
https://code.alephobjects.com/source/qtdeclarative/

https://code.qt.io/qt/qtdoc.git
https://code.alephobjects.com/source/qtdoc/

https://code.qt.io/qt/qtgamepad.git
https://code.alephobjects.com/source/qtgamepad/

https://code.qt.io/qt/qtgraphicaleffects.git
https://code.alephobjects.com/source/qtgraphicaleffects/

https://code.qt.io/qt/qtimageformats.git
https://code.alephobjects.com/source/qtimageformats/

https://code.qt.io/qt/qtlocation.git
https://code.alephobjects.com/source/qtlocation/

https://code.qt.io/qt/qtmacextras.git
https://code.alephobjects.com/source/qtmacextras/

https://code.qt.io/qt/qtmultimedia.git
https://code.alephobjects.com/source/qtmultimedia/

https://code.qt.io/qt/qtnetworkauth.git
https://code.alephobjects.com/source/qtnetworkauth/

https://code.qt.io/qt/qtpurchasing.git
https://code.alephobjects.com/source/qtpurchasing/

https://code.qt.io/qt/qtqa.git
https://code.alephobjects.com/source/qtqa/

https://code.qt.io/qt/qtquickcontrols.git
https://code.alephobjects.com/source/qtquickcontrols/

https://code.qt.io/qt/qtquickcontrols2.git
https://code.alephobjects.com/source/qtquickcontrols2/

https://code.qt.io/qt/qtremoteobjects.git
https://code.alephobjects.com/source/qtremoteobjects/

https://code.qt.io/qt/qtrepotools.git
https://code.alephobjects.com/source/qtrepotools/

https://code.qt.io/qt/qtscript.git
https://code.alephobjects.com/source/qtscript/

https://code.qt.io/qt/qtscxml.git
https://code.alephobjects.com/source/qtscxml/

https://code.qt.io/qt/qtsensors.git
https://code.alephobjects.com/source/qtsensors/

https://code.qt.io/qt/qtserialbus.git
https://code.alephobjects.com/source/qtserialbus/

https://code.qt.io/qt/qtserialport.git
https://code.alephobjects.com/source/qtserialport/

https://code.qt.io/qt/qtspeech.git
https://code.alephobjects.com/source/qtspeech/

https://code.qt.io/qt/qtsvg.git
https://code.alephobjects.com/source/qtsvg/

https://code.qt.io/qt/qttools.git
https://code.alephobjects.com/source/qttools/

https://code.qt.io/qt/qttranslations.git
https://code.alephobjects.com/source/qttranslations/

https://code.qt.io/qt/qtvirtualkeyboard.git
https://code.alephobjects.com/source/qtvirtualkeyboard/

https://code.qt.io/qt/qtwayland.git
https://code.alephobjects.com/source/qtwayland/

https://code.qt.io/qt/qtwebchannel.git
https://code.alephobjects.com/source/qtwebchannel/

https://code.qt.io/qt/qtwebengine.git
https://code.alephobjects.com/source/qtwebengine/

https://code.qt.io/qt/qtwebglplugin.git
https://code.alephobjects.com/source/qtwebglplugin/

https://code.qt.io/qt/qtwebsockets.git
https://code.alephobjects.com/source/qtwebsockets/

https://code.qt.io/qt/qtwebview.git
https://code.alephobjects.com/source/qtwebview/

https://code.qt.io/qt/qtwinextras.git
https://code.alephobjects.com/source/qtwinextras/

https://code.qt.io/qt/qtx11extras.git
https://code.alephobjects.com/source/qtx11extras/

https://code.qt.io/qt/qtxmlpatterns.git
https://code.alephobjects.com/source/qtxmlpatterns/

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jebba claimed this task.Jan 4 2019, 2:30 PM
jebba triaged this task as Normal priority.
jebba updated the task description. (Show Details)

In Phabricator language, a mirror is called an "Observing Repository". Docs here:

https://secure.phabricator.com/book/phabricator/article/diffusion_existing/
https://secure.phabricator.com/book/phabricator/article/diffusion_uris/

The docs don't say it, but you also have to edit all the URIs that get created automatically when creating a repo to be Read only. For example see cython:

https://code.alephobjects.com/source/cython/manage/uris/

jebba updated the task description. (Show Details)Jan 4 2019, 2:36 PM
jebba updated the task description. (Show Details)Jan 4 2019, 2:39 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:06 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:09 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:11 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:13 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:16 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:18 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:21 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:23 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:25 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:29 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:31 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:33 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 3:38 PM
jebba updated the task description. (Show Details)Jan 4 2019, 3:59 PM
jebba updated the task description. (Show Details)Jan 4 2019, 4:13 PM
jebba added a comment.Jan 4 2019, 4:26 PM

Attached is patchlet that uses code.alephobjects.com mirrors for repos that were pulling from git. Note, this patch only does repos that were using git before. Source that was pulled from a tarball is still using upstream, not our repos. This patch won't work until all the mirrors finish syncing (maybe a couple hours from now).

jebba updated the task description. (Show Details)Jan 4 2019, 4:48 PM
jebba updated the task description. (Show Details)Jan 4 2019, 4:53 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 4:56 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 4:58 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:03 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:09 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:12 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 5:19 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:27 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:40 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:42 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 5:46 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:48 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:54 PM
jebba updated the task description. (Show Details)Jan 4 2019, 5:57 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 6:01 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 6:09 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 6:15 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:23 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:30 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:35 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:38 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:41 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:47 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 6:51 PM
jebba updated the task description. (Show Details)Jan 4 2019, 6:56 PM
jebba updated the task description. (Show Details)Jan 4 2019, 7:00 PM
jebba updated the task description. (Show Details)Jan 4 2019, 7:03 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 4 2019, 7:06 PM
jebba updated the task description. (Show Details)Jan 4 2019, 7:13 PM
jebba updated the task description. (Show Details)Jan 4 2019, 8:28 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 5 2019, 2:53 PM
jebba updated the task description. (Show Details)Jan 5 2019, 3:00 PM
jebba updated the task description. (Show Details)Jan 5 2019, 3:13 PM
jebba updated the task description. (Show Details)Jan 5 2019, 3:25 PM
jebba updated the task description. (Show Details)Jan 5 2019, 3:52 PM
jebba updated the task description. (Show Details)Jan 5 2019, 4:11 PM
jebba added a comment.Jan 5 2019, 9:58 PM

@alexei I did two builds on stretch using the git-mirrors.patch and it works fine. This patch just re-points the git repos at our servers, not all files. I think it can be applied to master OK.

jebba added a comment.Jan 6 2019, 3:32 PM

Attached is a draft patch that built under Debian Stretch. All git mirrors point to code.ao mirrors. Some tarballs now point to code.ao git mirrors with matching git hashes.

jebba updated the task description. (Show Details)Jan 6 2019, 4:43 PM
jebba updated the task description. (Show Details)
jebba updated the task description. (Show Details)Jan 6 2019, 4:47 PM
jebba added a comment.Jan 6 2019, 4:55 PM

For files that wouldn't "conveniently" use git, I have copied the files to our server:

https://download.lulzbot.com/Software/Cura2/build-resources/

jebba added a comment.EditedJan 6 2019, 5:16 PM

@alexei

CMakeLists-repos.patch

This patch replaces all non-Aleph Objects downloads with URLs that point at our servers. Wherever possible, I used a git repo. In cases where that wasn't smooth to set up, I uploaded tarballs here:
https://download.lulzbot.com/Software/Cura2/build-resources/

Testing on Debian Strech now.

jebba updated the task description. (Show Details)Jan 6 2019, 5:26 PM
jebba added a comment.Jan 6 2019, 5:53 PM

CMakeLists-repos.patch builds fine under Debian Stretch. Should work under Mac/Windows too.

alexei claimed this task.

@jebba . I will add this patch to master.

alexei closed this task as Resolved.Jan 7 2019, 7:05 PM

Merged to master.

alexei changed the edit policy from "Custom Policy" to "Cura LulzBot Edition (Project)".Jan 9 2019, 2:23 PM