Create redistributable package for Julia

[Andrey1994]

Not sure that its possible to bundle native libraries with Julia packages right now, but need to double check and investigate different options

[matthijscox]

As mentioned, we can examine BinaryBuilder.jl . I'll see if I can do that myself. But perhaps you prefer to control the build process yourself?

[Andrey1994]

Here its not a few cpp files which you can easily compile wo modern C++ build systems.

I use Cmake to build it, and config file for Cmake is pretty big(https://github.com/brainflow-dev/brainflow/blob/master/CMakeLists.txt) I would like to avoid porting it if its possible.

Best case scenario and how it works for all other languages - compile C++ code using C++ specific build system(Cmake in our case) and add compiled libraries to bindings. If it's possible for Julia - its the best option, if not - need to check alternatives.

E.g how it looks in python - you build c++ code first and when you build python package libraries are included into whl via package data https://github.com/brainflow-dev/brainflow/blob/master/python-package/setup.py#L25 So this way python users dont need to worry about compilation at all and can download package with precompiled libs included

[matthijscox]

OK, so for Python the PYPI installs the binaries automatically in /lib. That's nice indeed, I haven't yet seen that for Julia.

For future reference:
I'll start by investigating Yggdrasil which is the backbone of BinaryBuilder.jl. It links to PRs of example packages implementing the Artifact system, like Cairo.jl 293

And this blog post is mentioned.

As mentioned in the blog "the artifacts subsystem is much more general and is widely applicable to all Julia packages." This suggests there must be a way to deploy a manual artifact via the package manager. The question is how difficult it is for a newbie like me.

Also the docs have a section on Artifacts:
"These containers, (called "Artifacts") can be created locally, hosted anywhere, and automatically downloaded and unpacked upon installation of your Julia package."

Come on, it must be possible!

[Andrey1994]

" so for Python the PYPI installs the binaries automatically in /lib"
Not 100% correct, PYPI doesnt install it to /lib. Its distributed as a ZIP archive in fact and you can add any files to this archive, I am adding them manually to /lib subfolder inside this arch. And in Python binding I load these libs from archive by full path

This https://julialang.github.io/Pkg.jl/dev/artifacts/ looks promising, I will read about it too

[matthijscox]

yeah, I see. The Julia _jll packages are also distributed as tarballs, so it just unpacks the folder structure you defined yourself in the tar ball.

The Julia packages typically split the package in 2; one the regular Julia code, the other the _jll wrapper and libraries. But it's not necessary.

Apparently all we need is an Artifacts.toml file which we fill with the relevant information:

[[brainflow]]
arch = "x86_64
git-tree-sha1 = "c7c61446219c91ab68a5841186d59b2dba00ab2c"
libc = "glibc"
os = "linux"

    [[brainflow.download]]
    sha256 = "... dunno yet what this, maybe you do..."
    url = "https://github.com/brainflow-dev/brainflow/.... wherever you have your linux tar ball..."

Then we should be able to add your package, use it and the library should be installed. We can try with:

import Pkg
Pkg.add(url="https://github.com/brainflow-dev/brainflow.git", subdir="julia-package/brainflow")
using brainflow

Note Julia package are typically uppercase camelcase, so I expected BrainFlow

[matthijscox]

Ah wait, I'm checking out the release page. You don't have cross-compiled binaries available right? So we still need the client to compile somehow. Or we do have to set up something like BinaryBuilder.jl for your package.

[Andrey1994]

There are libs for all OSes bundled inside whl\jar\nuget files. We can add compiled C++ libs as well to release page, or upload it somewhere else if needed.

For example currently CI uploads all artifacts to AWS S3 maybe we can use it instead github releases

[matthijscox]

Ok, then I guess the easiest way would be to just have the /lib folder in a .tar.gz or something. Do you have that available behind a url somewhere?

I can also try to see what happens if I use the .whl url.

[Andrey1994]

I can patch this https://github.com/brainflow-dev/brainflow/blob/master/.github/workflows/run_unix.yml#L213 for all OSes and configure AWS to provide download link...

I dont think that targz is a good option since for Julia it should and can work for all OSes, I doubt that it will open targz file on windows automatically. Can it download multiple files or only one?

[matthijscox]

Just tried the docs example. I got an error.. but in the end it did work...
And I can unpack the .tar.gz on windows
Anyway Julia uses a Tar.jl package, so it's all included in the language itself.

[Andrey1994]

Good to know, I think then we can create zip\targz with libs and upload it to release page

[matthijscox]

TODO: check if tar.gz is really necessary

TODO: check create_artifact(), archive_artifact(), bind_artifact!(), as used here for example: https://github.com/r3tex/ObjectDetector.jl/blob/master/dev/artifacts/generate_artifacts.jl

Also https://github.com/CiaranOMara/ArtifactHelpers.jl