Yeah, so there we want to present a good project of us, where we try to kind of revamp
the way we do a cross-compilation using basal and LLVM.
So you all know a cross-compilation is quite hard topic, it's quite arcane, mysterious, you
have to master a lot of pieces to get it right, and it gets harder and harder the more platform
you have, you add.
So you know, you know, existing solutions are mostly sisterhood based, which you know are
great, one stop shop, but also sometimes kind of a black box, you just trust what's
in there.
If you want to configure things differently, you can have projects like that, they are really
great of the shelf solution, will cross-compile for any targets and do it quite well,
using these routes, using pre-bilt.
So all of these tools exist, which was a lot, you know, way beyond the scope of this project
that is not just cross-compiling user programs, but entire Linux built, or as we so a couple
presentations before stuff that also do from source, entirely from source, but very specific
to one specific target.
Then you have stuff like XXCC, which actually this project is heavy based on, as in,
we don't use XCC, but we use a lot of the pre-sept in that, they do everything from source,
and that's what we wanted to achieve with this project.
Then you have stuff like Nix, which I'm not going to talk so much about, but it's cross-compilation
is very well supported, but it is, makes a lot of assumption about where it's going
to run, so it's also a great solution, just not what we're going to talk about.
So what we want to do here is just cross-compile absolutely everything from source, because
if you can cross-compile a very simple source, it means that using clung, which is a cross-compiler,
we can target any of the platforms that LLVM supports, and essentially, allows us to
cross-compile to any possible target within the realm of LLVM supports.
So what we need for that, LLVM pre-built whole chain, source code for LVM, and all of
the other one times, like G-lipsy, LLVM, which is a muscle, any kind of other one times,
or esoteric, one times that you can think of, we need the source, and of course, we need
basal, because that's what the talk is about, basal for dos, who not familiar, it's a
build system, open source, build by Google, and one of the philosophy is that it drives you
to declare absolutely everything as an input.
So absolutely everything is an input, on variable, on variable, on input, tools, on input,
every single header, and source, file is an input, and it allows us to do a much more
correct build.
So we use our own pre-built LLVM tool chain, with a slight twist, is that we use the very
very cool tool from the LLVM project, the David Hunt, the other day, called the LLVM
driver, that basically allows you to use lots of the LLVM tools, but as a single binary,
pretty much like the Busy Box works, and this is interesting for us, because it allows
us to ship a very, very minimal, a very, very small tool chains, with all of the
clang, LLD, what else, object, object, object, object properties, and yeah, sort of
megabytes, and that's pretty much all we need, plus, during time sources.
So once we have all of that, and all honorable mention for David Hunt, for upstreaming a lot of
things to LLVM, to make that possible, and to support a lot of tools that were not
supported before.
So with all of that, what we have is this kind of complex graph, not the complex one,
when you know it, but basically the tool chain that we build is a multi-stage one, because
the LLVM cannot depend on the LLVM, so we have to build the LLVM, or the parts of the
LLVM, a very specific way, LLVM winds, same LLVM plus plus, same, all different stages,
all different kind of dependencies, and at the end, we have a final, a complete tool chain
that has all of the runtime dependencies that the normal program requires, and all of that
built from source.
The cool part is that, because everything is, is from source, and everything has been,
like, download it, everything is aromatic, we can actually pass this flag, which is kind of
unusual for cross-compilation, but we, we find it was pretty cool to add and to just prove
that there really is no C3D.
What happens next is 100% aromatic link, we are absolutely everything has been cross-compalled
from source of all of the runtime, including the Syrtis,
and that Indian allows us to build a user program
fully air-metically compiled and linked
for any target that we can support.
So any host, any target, as we can build,
as long as the target is supported by LLVM,
and we have the build sources for that,
then we can target absolutely everything.
So far, we support these whole matrix and it's growing,
and it's getting easier and easier for us
because the big work was at the beginning
while you're in everything.
Now, adding targets is mostly writing build files,
and then adding that to the graph,
and we can support more targets.
Yeah, the problem is we need to write the build files
that's kind of the hard part there
between David and I, we implemented a lot of that.
So it's a lot of editing on how all of those runtime
is built, how to do it properly.
In the special case of the GDBC, we go, we are far,
every single version up to a cutting point
that is the oldest LTS of RHDL,
which we start with was a good minimal version to support.
So we have to adapt the build files for the versions.
The problem also with the GDBC is a kind of a specific case
because it is very, very hard to compile.
It's quite hard to understand.
Yeah, I need to go fast, sorry.
It's very hard, lots of McFiles cross-compiler.
So we decided, OK, we're going to generate a stub library
with all of the version symbols and then links against it.
So we generate, basically, like a stub assembly file,
with all of the versions per version of the GDBC.
We compile it and we link against it.
All of the symbols are null, but that's fully all right
for all the satisfied linker.
We actually took this from ZIG, if you want to check out,
it's the very cool tool called ABL, ABL, ABL,
I list all, which we have in our best disk project
for the GDBC.
Quick demo before I give the mic to David.
Oh, is it working?
OK.
So here, we're going to cross-compile for the next front Mac
of an SSL.
And so it builds absolutely everything, as we mentioned,
all of the runtime is all of the compiler-arty.
It's a craft custom resource-dier.
It crafts, basically, everything to hand over
to the clung driver, we do not pass any kind of adi-chonol
flags to it.
As you can see, it's dynamically linked links are cheesy for.
And if we run it, line is just basically
Docker to be able to run from Mac and it works.
All right, so let's talk about some of the other components
that we built.
We added support for sanitizers.
They're built from source, that's part of the build graph.
So that's kind of cool.
If you pass, like dash dash config equals UBCAN,
your binary gets built with UBCAN across all the platforms
we support.
We can link it statically dynamically, whatever.
But cool, other cool thing is, because the tool
changed part of the build, we can rebuild LVM with UBCAN
and then build your binary with that.
So if folks are working on debugging LVM or something,
that's like a cool, fun, handy thing.
So we support Windows.
So Windows, we built from source.
So we use the MNGW project, which is great.
But we wrote build rules to build a CRT's.
We build main works, which is like this extension library
and a couple other like C libraries, which
generate import lives from all the.dev files.
And then there's a bunch of like MRI recipes
for assembling these things into the way
that the normal MNGW assist route layout looks like.
So we replicate all of that and do it on the fly and prepare
all that for the linker to do the final link.
So demo.
Is that everyone?
You have it up?
Yeah.
So here, we're going to change the platform slide to target
Windows AMB64, but we also support ARM64.
And we're going to go ahead and build that from source as well.
So right now, it's creating all the import libraries
compiling with libc++, targeting Windows.
And then it'll finish this sec.
And we have a, oh, there we go.
We have our Windows file.
We're going to run it on the line.
And it's a little world.
That's kind of neat.
That better.
Cool.
So why do we do all this other than the fact
that we're nerds and we love doing crazy stuff like this?
So this lets you do remote builds with zero setup,
because everything is defined from source
as part of your build graph.
All of your build actions are able to be
scheduled for remotely across a ton of cores.
So we'll show a demo of that in a bit.
This also means you get a working cross compilation
for C and a cross linker for other languages.
So you can plug this in as a linker for rust,
for go, et cetera, for go requires a patch
to go tool chain, what we submitted.
But for rust, it kind of just works out in the box.
Big shout out to our friends at build buddy.
Build buddy is a remote execution provider for basal builds.
We kind of abuse them a ton to run a bunch of these things,
because as you can imagine, we are building LVM from source
a lot.
And it's not that fast to build, except for us.
It's very fast to build.
I don't know if you can see this, but it's just showing
a profile of the graph.
There's like 700, 800 cores working on this build.
So if you just kind of sharded out, it's OK.
Let's show them the sample of that.
What is the example?
What are we showing, Ashley?
Same with the remote.
Oh, yes, this is remote build of a thing we should
produce, see with open SSL.
So it's going to load, it's going to build, it's done building.
That's kind of nice.
And then build buddy has a nice UI for like visualizing the graph.
That's the same thing.
Cool.
The final thing is bootstrapping.
So once you have this like hermetic cross tool chain,
you can build LVM itself, which
is what we do as part of our release process.
But we also integrated it into the build graph.
So if you pass dash-configure was bootstrapped,
then with a single build command, we're
going to build you a fresh compiler, and then use that compiler
to compile your user code.
And it's basically the graph de-plicated,
because that's a bootstrapping, because right?
Yep, it's a single build.
Cool, let's show a demo of that as well.
So here we pass the extra bootstrapped, bootstrapped flag.
And we're going to build LVM from source.
And I'm going to build Open SSL using the new LVM from source.
This is all running remotely, because otherwise,
it's going to take like 20 minutes or whatever, right?
But if you throw it off, of course, at it,
it's actually kind of pretty fast.
Can see the 7,000 actions, which I haven't
7,000 files to combine.
So that LVM's done, now we're building up in SSL.
If you ever are developing LVM, again,
you can make a change to LVM in your local checkout,
and then iterate on the entire thing
and then to end the single build command.
Cool.
OK, I don't know if we need that.
Cool.
So what's next?
So right now, we build LVM.
We build with LTO and everything, but we
would love to instrument it and do FDU.
So the best way to instrument a compiler is to compile some stuff.
So our build graph will look like this.
We're going to build LVM from source.
We're going to use that new LVM to build an optimized
and instrument to LVM.
We're going to use that to compile LVM
to get your profiling data.
And then we're going to combine all of that
into like a final optimized compiler.
It's going to take a while, but it's OK.
What I also want to do, we are not support
for more targets, more run times.
If you have other ones, send us patches, come talk to us.
We kind of want to be like complete and serve
a variety of use cases.
We already have some folks who submitted some stuff
for Windows.
We're looking at trying to get Cosmo Lipsy working.
We're going to be pretty neat to have a single compiler binary
that works across various platforms,
because especially with like remote execution,
you can share your action inputs and get cache hits,
no matter which platform the actual compiler ran on
if it's the same exact binary.
So this is a little bit of a science project for now,
but making some progress on it.
And the final thing that we think will be really neat
is to be able to eject from basil and kind of use
basil to produce a normal cross compiler,
or a normal sys route, and then use that to power other projects.
So we're not quite sure how this would look yet.
It might be like this, where you actually create the cross
end, and then use that, or it might be like a make cc approach,
which will have like a shell script that uses
basil under the hood to create it,
and then run the filter to the build.
We're going to play with it, see what feels good, see what works well.
But yeah, definitely trying to bring the power
basil to non-basely uses as well if possible.
Cool.
As all we have, thanks for listening.
Here's our repo, everything's up there.
And yeah, any questions?
Yeah.
Yeah, thanks for the architects, it's very exciting.
I actually did something similar, so this time here.
But mine's the highest of work, which is, and therefore,
I cannot set it in English, but how I understood what basically
you may see when it's a delicious restaurant.
The texture, basil, cc like a pork can be in place
with the grass, so you will just make a gc, and then
also become a kind of cool material, like the deal,
by a server cc like that.
So they're really good to be speaking visculose over
which is cc, apart from the part where you are actually
building people, but I don't understand what cc is.
Yeah, the question is, how we handle glip c
and water is going to be made to work, or gcc?
So I think we like simplify it a little bit.
We don't build glip c itself from source.
We kind of build the headers in an interface library,
and then when we dynamically link,
we end up using the system glip c at one time.
If that makes sense.
But I don't know if you want to say more about that.
Yeah, and for all gcc, as you mentioned,
do we so far?
I don't think I have the energy to try to specify gcc.
We have plenty to actually specify lb,
is it a stdc plus plus, and the bunch of other
new projects.
But gcc itself is just out of RAM, really.
Let's start.
OK.
Yeah.
Let's imagine you have a way to run a possible by binary.
And then could you rearrange the gls and all things,
and specializing for the workload?
Yeah, the question is, can you run the cross-compiled
binary to gather pg of statistics and recovery compile?
Yeah.
So basal already has support for running a compilation
with like a pre-uptained workflow.
And you can kind of do it out of the box.
It would be two separate build commands.
Like you would build a gathered profile
than build again.
We only get to cheat because our workload is compilation.
So we can make it a very fancy build graph.
But that's also kind of us, like, showing off.
But I do think, like, they go the same approach.
You could, if you encoded your like profile gathering
thing as a build action, like it could also be wired
together in similar way.
Got it?
Do you plan to have many, many packages to do?
Do you make your show relevant to them?
Let's say that I won't use them for a complex project,
but do you want to account that kind of basically
can work in the different way?
Yeah, the question is, do we intend to become a package manager?
The answer is no, because Basel has something called
Badalcentra Registry, which is repository of build rules
for a bunch of packages.
Not everything, but there's a fair amount growing like day by day.
So the idea is, this is a tool chain that you bring into your
project with like three lines of config.
And then you can bring in all of those packages, as well,
through the Basel module system, and then basically compose those.
So as long as they have build rules that build the thing
and don't, some things only compile in certain platforms,
but as long as they haven't really done that,
we'll provide all the run times and all the standard library
and everything so you can kind of combine them.
And this does work off the shelf for most things we try to.
Yeah, good.
How could them work for things that I won't want to build?
If I have to depend on some other, just to let me know
if we need to depend on the S-O library, like Pribil, the S-O library.
So actually, Basel already provides such a feature called CC Import.
And for us, it's going to be armetic, because you refer
us to the file as far as Basel knows, it will know about that file.
It will come into the links and box and then Nicky.
So it just works.
And we actually do that a lot.
The file should exist on the platform, or yes.
If not, I mean, I do that on another project.
We just take them out of the wherever they are,
and we just CC Import in Basel.
Yeah, Basel also has really good support for bringing in remote files.
You're picking your trust files by URL and hash.
So you don't have to go and download things yourself.
You can just say, hey, I depend on this file.
That's hosted wherever.
And then, you know, write a rule, bring that into your build,
the SCC Import.
So that way, like, if I build a project this way,
and then you go, close my repo.
You don't have to go to download stuff.
You can just run the build.
Come upon Assassin, after?
Thank you.