Around a glass, please, for Paul Meier, who's going to talk about, go in the next ecosystem
and some interesting work he did in the area.
Welcome to my talk. I'm Paul. My guitar panel is Cut X-O-Kin.
I'm also a master-dom, I'm a security software engineer.
Leak spectators, maintainer, and I'm also part of the Leak Specter good.
Take it to the scope team, so if you're wondering why you didn't receive the latest GoVinability update,
I would tell you that it is currently in staging next and you have to wait while applying overlay.
So I want to quickly give another view over the GoV dependency management and general outside of mix.
GoVin manages dependencies as modules, which is the collection of packages that is released version and distributed together.
So this is usually a repository scope.
We have to go that not file that declares minimum required versions of module dependencies.
Let's also go that some file that contains cryptographic hashes for these dependencies,
but to go that some file is not an exact log file, and also the hashes in there are incompatible with nix,
so we cannot use import from javiation here.
And then you go use this minimum version selection to, which is a deterministic algorithm to select the actual versions that I used in the build list.
And only at the one version of a dependency is selected in the end.
And use a quick look at the GoV Mod file.
Just to get an impression, the required statements declare or direct, and at the bottom indirect.
So transfer to dependencies.
So now come to the nix part, how do we build or package go stuff nix.
In nix packages we have a single build up of this called the GoMod view.
And for most, this might be straightforward.
We have a source that we fetch from GitHub, and we have a package name and version.
And what I really want to point out here is the vendor hashes at the bottom, which pins all dependencies.
So we will take a closer look at this, and also at the delay, the relations structure here.
So we have one derivation of, which is by GoMod view, the thing we want to build.
Then we have the source, which is a fixed output derivation that pulls our source from GitHub.
And then we have a third derivation here called GoMod Views.
And this depends on the source.
And use the GoV Mod file and the Go command to download the dependencies of the module we want to build.
Interfixed output derivation.
And then this is pinned by the vendor hashes.
So if you now look at two different GoPackatures pattern pin act.
Both will have GoMod Views, derivation they depend on.
And yeah both will have some Go dependencies in there.
And as we see here, now these two packages depend on the same GoMod Views.
They have the same dependencies.
But in next, we do not have any introspection in each of this.
So we will download the same dependency again.
And yeah, we do not know that these two fixed output derivations actually have the same content.
So what does this mean for the current status and next packages from GoMod View?
We have to download things multiple times.
We are storing these dependencies over and over again.
Very important.
We don't have any form of caching.
So we have to rebuild the dependencies over and over again.
Both between two different packages.
But also in case we change something about our actual package.
We want to build.
There's no caching for the build of the dependencies.
So we also have to rebuild our dependencies in this case.
Then we do not have any introspection into the dependencies in nix.
It is really hard to update dependencies of the package.
For this we would actually need to create a patch for the GoMod file.
And we would have to do this for each GoMod package individually.
And following we do not have an easy way to patch vulnerabilities.
So first I want to look at the vulnerability aspect.
How bad is this?
I've created a small project.
GoMod check nix packages.
GoMod check is the official vulnerability scanning tool of GoMod.
And it has a separate vulnerability database containing CVE.
But also go vulnerabilities that are not assigned to CVE.
GoMod check is quite cool because it checks if the vulnerable code is actually reachable.
So you get less false positive in case there would just import a vulnerable package.
But the reason isn't a core path that would make your code reach the vulnerability.
And then go Vynscheck nix packages runs GoMod check on the source of all GoPecaches and nix packages that are built with GoMod.
So what are the results here?
And black we see the packages that are discovered by GoMod check nix packages.
This cover is currently based on simple attributes of the output.
Then in blue we have the packages that are actually scanning successfully.
This is just like a really simple project.
For example, for packages that you see go, we do not provide the cd dependencies.
So the vulnerability check I will just fail because the dependencies aren't there.
So we are missing out on some of those.
And then in red we have the number of vulnerable packages, which is quite stable around 30% of the scan packages.
And then in yellow we have the number of vulnerabilities in total.
We have like around three vulnerabilities per vulnerable package.
Just remember we don't have like an easy way to fix vulnerabilities, especially in that at this scale.
So now I want to to make a jump to the caching problem again.
And just yeah, how the cache usually works with Go is we have like an environment variable GoCash,
which points to a five system path and there's our cache located and go will read and write to this path and manage the cache itself.
And now there's a really cool feature in go124 and also in earlier versions as experimental feature that lets us provide binary code GoCash Proc.
And yeah, this gives us a lot more flexibility.
Go will just invoke the binary during the bit presses and talk a JSON based protocol.
He has done that in standard out with two basic operations, get to request a cache entry via an external idea.
And then in case we got the cache miss, go will build the package and then call put.
So the go cache Proc can provide the cache result somewhere.
And at this point I want to introduce goboots.next, which is next generation build up for go.
I use this go cache Proc.
And just so you get an idea, goboots.next tries to provide the package sets of go dependencies.
Where each go module has its own derivation.
You will try to module the go dependency graph index.
And first I want to show how go.next uses go cache Proc.
So we have written go build nix cache binary that implements go cache Proc.
And the way this works is we have an environment variable where we read a number of paths from.
And then go the cache over will save the cache entries found at these paths.
And the paths will point to the next door.
And in case we have a cache miss, the cache just writes the build result into a defined output.
And here we can add the stop paths we set up hook.
So we put the go module dependencies into our build input.
The set up hook will add the stop path to the environment and then the go cache the cache can serve easily.
Now looking at this from a derivation level.
So if we want to build GitHub.com, full slash a, we have the source in green.
Then we have the source of our dependencies, which is required for the build process.
We match that with the source of full slash a by linking the dependencies and so the vendor directory.
And we also provide the cache.
So the build output of when we build the dependencies to the cache.
And now the idea is that we will not have to rebuild the dependencies because they are already cached.
The cache will serve the build cache for the dependencies and we will only need to rebuild the packages that belong to full slash a.
And for these packages, the output of the build process is written to out.
And then the final build result will contain the cache entries for full slash a.
And looking at this at the package set perspective.
And these packages will construct the package set.
And mainly consisting of the build results.
The cache entries that the cache are stored in the output.
Also, quite important is that we have to have the source of our transitive dependencies available for build.
And if we now want to build a go binary, we would do that outside of the package set and just add all the built inputs.
As usual, so the go modules as built and put.
And we would have all the dependencies already available and also built.
And yeah, every dependencies only available in one single version in this package set.
So we are doing like some other writing of the of the version.
Where where package depends on.
So what we get here is we get incremented for fetching.
We can fetch every dependency on it's own.
We get rid of this vendor hash.
Also, we get rid of the of the bad development workflow that comes with that.
We get incremented builds.
Tapping for our for a build process because the dependencies are already cursed.
We get great composability because we're using set up hooks.
We get the possibility to patch vulnerabilities because we only have dependency once in our set and we can then bump that.
And it will it will propagate to to all the dependence.
So that's really cool.
This project targets next packages.
The status is highly experimental.
It's not usable yet.
I did some research and I figured we will have to have like 12,000 go dependencies.
So next packages we've got packages to build all the other packages that are currently.
The next packages.
And yeah, but now we are like about 30 packages that we are building.
So there's let's do a lot of work.
And we still are figuring out like how can this this work at all.
Yeah, and it's a lot of work.
I'm looking for people to collaborate on this.
I hope for the the builder, the new builder, but also for the vulnerability research thingy.
Please talk to me.
You can reach me on metrics or here and person.
And then I want to give shout out to all go related next projects.
And I want to point out especially the one at the bottom which is quite new.
And which uses the same go cash product mechanism
to serve the go cash from outside the sandbox.
So this introduces some impurity.
But it relies on go to ensure the correctness of the build.
So yeah, that's something that might be usable today.
Yeah, thanks a lot.
Thank you very much for the great talk.
I have a lot of time available for questions.
Are there any questions?
Can I see your hand back there?
Are you going to use the go box?
Do you want to use the go box?
So the question is, if this still uses the go.mod file.
And the answer is no, it does not.
You have to add the build dependencies as built inputs in the next.
In your next derivation.
And we will also rewrite the go.mod file.
So because if you as I said, there's only like one version available.
And this means that when there's like a diff in the version.
You have in your go.mod file and the next packages.
There could be like 10 to 50 dependencies that have changed.
So we rewriting the go.mod file completely at this point.
There is another question here.
I think a form of the question.
Is there any kind of equivalent, next equivalent,
and walk file that you are generating?
And next.
Because you have to manually add all the dependencies to your build file.
Or do you create a walk file?
No. So the idea is that you do not decrease.
If you create some kind of lock file for this.
And the answer is no.
That first you have your dependencies available in the next packages.
So you just have to add them as built input.
And then they are pinned as we fetch the source.
And at this point they are pinned.
So that's quite a lot of work initially.
Yes, this is a lot of work.
No.
I don't have better random.
But you can maybe automate this run.
You can kind of think.
Have a tool that takes the walk file.
That gives you the next array of.
That doesn't see that you have been created.
Yes, so the ideas that this could be.
There's like a lot of room for automation.
And sure, there is.
I don't think we will have to.
Yeah, package all these things by hand.
Especially as there are a lot of libraries that are really easy.
And do not have like any, any built input at all.
Or we can use the input stick to go out of Mod file.
Yeah, but we're not there right now.
Quick in between question is Josh.
Can you hear who's going to be the speaker for next talk?
Okay.
Question is about the action of the talking.
No.
No more questions.
Well, then thank you very much.
Thank you for finishing on time.