[mosh-devel] Mosh OS X package build on Travis

Wed Nov 2 13:33:56 EDT 2016

Hi there.

Pardon the de-lurk, but I have a few thoughts on the topic that might help.
I use mosh on os x and linux. I also happen to use homebrew.

As you've already pointed out, a proper package manager makes a lot of this
much easier. There are a couple ways to get closer to this on a mac.

First, a stupid linux trick. In the past, on debian based distros when I
have needed to replicate an environment that was lacking in automation for
builds, I've fallen back to
dpkg --get-selections > packages.txt
scp packages.txt host:/tmp
dpkg --set-selections < /tmp/packages.txt
apt-get dselect-upgrade

Ok, so at that point I can be confident that system B is similar enough to
system A to start working on whatever the issue is. The only problems that
have cropped up are where deps are so old they've rsynced off the public
mirrors, or in unversioned crap in /usr/local. You can do it with rpm too,
it's just spelled differently.

A way to replicate that technique on a mac could be as simple as:
Track the os version 10.x
Track the xcode version.
Track the homebrew version.

Homebrew has some options here.
brew list > packages.txt
scp packages.txt hostb:packages.txt
brew install $(cat packages.txt)

Where this kind of falls on its face, is that homebrew more or less just
chases the tip of a branch (stable) and updates are rolling.

But, it's all in git. If you were to go to /usr/local/Homebrew (the new
home), you could figure out what sha you were using for the current build
by running
✔ /usr/local/Homebrew [stable L|✔]
13:24 $ git rev-parse HEAD
8a7317aa8ff5f12067eb65e529a13490bc69deda  (example sha to save somewhere)

They also have tags. And other branches. Point being, you could pin to a
version/branch or whatever. It's just an exercise in juggling the metadata
when building and capturing it somewhere. At that point you're much much
closer to being able to stand up a brand new fresh os x box and install
known versions of packages and known configurations of homebrew without
inventing nearly as much of this solo. And lets face it... Idiosyncrasies
with third party libs are a drag. At least engage a large community to
share/shake out wrinkles. At a high level it's just 4 sets of facts.

Re-lurking.

On Wed, Nov 2, 2016 at 1:45 AM, john hood <cgull at glup.org> wrote:

> On 10/31/16 10:46 AM, john hood wrote:
> > On 10/31/16 12:41 AM, Jim Cheetham wrote:
> >> Quoting john hood (2016-10-31 17:12:06)
> >>> Alas, we will not get any kind of repeatable builds out of this, Travis
> >>> constantly updates their build images and we update to current Homebrew
> >>> for dependencies on every build.
> >>
> >> That's the worst bit. Using external services that are *unlikely* to
> attack
> >> your process is generally just fine, as long as there is a way to
> verify their
> >> output.
> >
> > The repeatability is of course no worse than what we've got now :)
> >
> >> Perhaps you could use Travis to report in the buildability of a
> revision,
> >> and the source of a 'nightly build' version, but keep a repeatable
> >> build chain for official releases?
> >
> > We already use Travis for CI builds on Linux and OS X.  I could of
> > course maintain a build VM for Mosh releases and snapshot it for each
> > release build, but having to maintain it (OS X upgrades, Xcode upgrades)
> > for Mosh's fairly infrequent releases is a significant burden for a
> > small project, and it would bring us full circle to manually-maintained
> > private builds.
> >
> > We can at least report the Travis image id (I think this is already in
> > the build log) and the Homebrew version + Git revisions, and other tool
> > versions.  Homebrew has a way to dump its configuration ('homebrew info
> > --json=v1') which may or may not be complete.
> >
> > The elephant in the room is that we're trying to improve builds for a
> > binary-only, proprietary OS with relatively ad-hoc installation,
> > configuration, packaging, etc.
>
> I've had some more thinking on this, and I'm getting over the
> you-piled-what?-onto-my-tasklist reaction.
>
> There's a number of separate pieces to this problem:
>
> * Fully automating the OS X build and installing its build deps.  This
> is a pretty basic first step to all the rest of it, and what I am trying
> to do with Travis.
>
> * Reporting: describing exactly what was built, how it was built, what
> dependencies were used, other environmental factors.
>
> * Repeatability: getting the same thing twice in a row :)
>
> * Reproducibility:  making it possible, maybe even easy, for you to get
> the same thing out of a build that I did.
>
> * Trusted build infrastructure:  in theory a reproducible build reduces
> the need for this.  In practice...yeah, we should do that.
>
> A lot of this stuff amortizes nicely if you are a distribution or
> package management system.  We're trying to do just one package, only
> for OS X (and not our many other platforms), and not even for all OS X
> users.
>
> My current OS X package manager of choice is Homebrew.  It seems they've
> done some work towards reproducible builds internally, but it's not
> clear at all how you might reproducibly install Homebrew, or do
> non-Homebrew builds reproducibly with Homebrew deps.  I just put a query
> out on that.
>
> It would be nice if Homebrew or Macports or some other package system
> could build standalone apps, and had reproducible builds.
>
> Another alternative might be to abandon package managers entirely for
> release builds, and build our dependencies ourselves.  Xcode might have
> some useful capabilities in this area, I haven't looked.  This may
> require separate, clean OS X instances for package builds.
>
> This isn't going to happen instantly.  One approach to the trust issue
> here might be to just cop out-- stop doing OS X package builds and tell
> people to build their own, until we get this stuff into better shape.
>
> Have I missed anything obvious?  (Yes.)  Have any of you got wisdom on
> dealing with this stuff in an OS X environment that you can share?
> (Please!)
>
> regards,
>
>   --jh
>
>
>
> _______________________________________________
> mosh-devel mailing list
> mosh-devel at mit.edu
> http://mailman.mit.edu/mailman/listinfo/mosh-devel
>
>

-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/mosh-devel/attachments/20161102/767ecd82/attachment.html