And he's going to talk about rewriting PYC files for fun and profit.
Well, reproducibility anyway, sir.
Thank you very much.
Give him a warm welcome.
Thank you.
Continuing, kind of, in the subject of the previous talk,
I'm assistant developer, and I work in Red Hat on various open search things,
in particular, and the federal inux distribution.
I'm not a Python developer per se,
but I do a lot of stuff related to Python.
And one of the initiatives we are doing in federal right now
is making package builds reproducibility.
And I will be talking first a little bit about package build,
reproducibility, and then how this ties into Python,
and how we are taking Python files and rewriting them.
So, build reproducibility is this idea
that somebody builds a program, a package, whatever.
Give it to another person and this other person can repeat the build process
or have somebody repeat the build process
and get the result that is bit for a bit identical.
This bit for a bit part is important because if it's exactly the same
that is very easy to compare.
If it is like 99% the same, then it's much harder to compare things.
So, we would want to be exactly the same.
And this is an ongoing effort in particular driven by Debian for more than 10 years.
There is a reproducibility of builds organization,
and also arch Linux developers do a lot of work on reproducibility and so on.
And there is like two reasons why we want to do this.
Like this initial entry reason is that this increases security
because if somebody tries to give you a binary that does not correspond to the sources
that it should correspond to or they mess something up on the way
and maybe they got hacked, then you can repeat the build and figure this out.
But there is also like a more interesting, more immediate angle at least for me
is that if builds are reproducible and we try to make them reproducible
we figure out lots of little bugs and by doing this,
we just increase the quality of the software.
In particular, there is also this issue that if the build is reproducible
and you introduce a change and you rebuild, then the difference in the output is what you
have caused by you. And if they build this unstable, then every time you build
in principle you get some variation and then you can be confused
what is actually caused by your changes and what is random.
And so to make package reproducible, in general,
I mean with this anyway, we build packages in a container with no network access
in a hermitic environment and the dependencies of the package build are injected
from the outside and in case of package builds they come as package contents.
And so the build environment is exactly reproducible if we make the effort
and then to have the package build reproducible, we actually need to have a deterministic build process
and all the tools that produce outputs must do it in a way where it is
deterministic and independent of the environment, like most of the environment.
In particular, there is this issue that we have to pretend that the time is some fixed value
because if we repeat the build a lighter time and we are actually used the clock,
then of course everything will change and all the time.
So for example, all the time stamps on files that produce must be clamped to some value.
And there is this source data epoch environment variable that is used to communicate
this to the build process.
And it turns out that if you do this at scale,
like further has 25,000 packages, then there is lots of issues and you find little bugs in tools.
And sometimes it is just too complicated to think.
I mean sometimes it is nice and we fix the tools themselves and then this work is shared
between distribution and everything is nice.
But sometimes we just find issues that need to be fixed, kind of like in a positive phase.
And because some issues are annoying to fix in a different way,
I think this will become clear in a moment.
So I mentioned that Debion has been working on build reproducibility for many years
and they have stripped on the dimism. I try to be positive.
So in fact, I have added the dimism and it is a little tool that runs in the build process.
And then at the end just like a walk over the files and apply some cleanups.
And currently it is fixing of modification times and ownership issues in various kind of archives.
In particular, static libraries are also archives in the AR archive format.
That is not used for anything else since 1990s.
But I don't want to talk about this.
We also fix times times in Java. files because Java.com likes to put times times everywhere.
And we also fix Python PYC files.
Because in general, it turns out that if you do this,
the version of scale Python writes those files irreproducibly.
So this was kind of like the intro and now let me get to the actual topic.
So what are PYC files? I know that people here are Python users,
but just like a short reminder.
We have a Python source file and when we import the file,
Python will try to optimize this by pressing the source file.
And then saving the results of the parsing into a binary file that can then on the second run
and subsequent runs below that faster than the reading of the source file and parsing of the source file.
And the way this works is that the attempt to do this is made on every import of a Python source file.
If Python looks, if it finds the cache file, then it will use the cache file.
If the cache file is not there, it will attempt to write it.
And this writing may fail, for example, because we don't have permission to write in the right place.
So the right place is next to the source file.
And this, for instance, distributions, which distribute Python source files, packages.
This creates a special situation where most of the time when you run Python code as a user,
you cannot write byte code files at all because the Python files
they're stored somewhere in the equation that's only writeable by user.
So if we don't provide Python files, I'm sorry, the PIC byte code files ahead of time,
then a normal user cannot write them.
And on the hand, if we don't provide them and the root, the root user would run the Python code,
they would end up writing the files because they have the permission to write the files.
And then when we upgrade the package, we would leave those files behind, and they would be out of date,
and over time this would be messy.
So we want to have them in the package.
And we want to do the way there for those.
This is that they are built as a part of the package build in the environment where the builds happen.
And they already part of the package payload so that they get nicely deployed.
And yes, so that's a nice way to do it.
Other distributions do it slightly differently, but that's the way we do it.
And so let's talk about the content of those files.
So actually I want to go kind of no level, also like in the previous talk.
So how does this file look?
So it is very, very simple.
There is a header.
The first two bytes are constituted a version number that it specifies the version of Python,
that wrote a given byte code file.
Then there is a magic number to kind of let them be distinguished from other files.
And then the rest of the header.
The whole header is 16 bytes on lightest Python versions.
And then there is a series of objects.
And it's just one object and the next object.
And that's essentially the whole format.
And how do the objects look?
So they're all kind of in the same format where we have a single byte identifier of the type.
And it's an aski byte.
So aski letter.
For example, an integer is an i.
And then four bytes of the payload in little and young order.
And you'll notice that, like since our machines are little and young,
and I'll have, well, possibly four byte integers depending on this situation.
Python can take this data, put it directly in memory.
And it doesn't need to do any parsing.
It already has the data in the format that corresponds to the actual layout of this data in a Python object in memory of time.
So this is very, very quick.
And so we have integers.
We have floats that are egg bytes.
So edg and egg bytes.
We have floating point numbers.
So typing identifier, egg bytes of the real part and egg bytes of the imaginary part.
Well, Python integers are famously of arbitrary range.
So this is a bit tricky.
We have L which I think stands for pi long.
Then we have the number of digits as for bytes.
And then we have a digit, which is a digit in base to the 15th.
And then we have the second digit and so on.
I did the number, specify the number of digits.
And then I mean, this is red direct into memory and you get a Python float.
Sorry, a Python integer.
And we have also strings.
So those are like, they vary by how much unique code is in them.
They're like, ask a unique code.
There's lots of Python history with strings.
I don't even know the details.
But the format is always the same.
You have a size and a bunch of characters or bytes in case of unique code.
And we also have a nice optimization here.
We can have a short ask is string where the size is just a single byte.
So the string cannot be more than 256 characters.
And then we have the characters.
And it's all the simple types.
We also have more complicated type.
Oh, no, I forgot about special Python stuff.
So none falls through ellipses and stop exception.
Each one of those ends up being just a single byte in the byte code bytes stream.
So nice and efficient, complex objects.
So Python loves lists.
A list is signified by the opening bracket, the size of the list.
And then we have objects.
And the objects.
So here the upper-case stuff is kind of fixed size.
And the lower-case stuff is variable size.
So each object is written as in the way I described.
So each of those objects in the list starts with a type specifyer.
Then the object payload and the type specifyer in the next object payload.
And so on.
We have two poles.
Just the same format, just a different character.
We also have short two poles, which are, well, different type.
But the size is one byte.
So again, we save three bytes on the whole object.
But there's lots of very small strings and very small two poles in Python byte code.
So this actually makes sense.
We have sets.
We have dictionaries that for some reason have a slightly different format.
So yes, there's a terminator instead of a size.
In Python 314, we got a new type.
There is a slice object signified by a column.
And then there are very complex objects.
So this thing called a code object.
And it's like a whole list of things that are of fields that have fixed size.
So for example, the number of arguments or the number of positional arguments or
the number of keyword only arguments and flags.
There are all fixed size integers.
And then there are other fields which are objects of arbitrary type.
So for example, code would be stored as string of bytes.
Well, written in one of the formats described previously.
And this is, well, kind of messy.
And I like that because this is the only complex type that there is.
And the whole Python file is actually the header and a single code object.
And the whole contents are stored as objects inside of the code object that
will require simply contain the whole module.
And there is one more complication.
Oh, yes, there.
They expand.
So we can have references.
And the idea is that if we have, if you have an object stream with stuff that
repeats instead of rewriting again and again the same contents, we put
their reference.
So the way this works is actually quite simple.
On certain objects in the byte stream, we set, we flip a single bit.
This is the first bit in the object, which is otherwise unused.
And this sets a flag.
And the flags are numbered.
Here object number 2 has flag numbers 0 and object 4 has flag number 1.
And then we can write it on in the stream, refer to them by number.
Nice and easy.
So I have an example here.
This is output from my program.
I made it front a bit small.
I'm sorry about that.
So the whole thing is a code object.
It's name is module.
This string itself module is flag number 204.
And then the name repeats.
So instead of repeating this string, it's actually a reference to flag number 204.
So this is like the name and the qualified name or something like that.
And the whole object itself is flag this object number 0.
In case we would refer to it later.
And then it has the arc counts.
And the constants, which is a typo with certain fields.
And then we have a code, which itself is a reference to something.
And so on.
So that's how the file looks.
And what the references give us a bit of a headache.
Because objects need to be flag to be referenced.
But you can flag an object and then not reference it.
For example, in the previous slide, the code itself, the top level object was reference with flag number 0.
But we cannot refer to it ever later because it's the whole file.
So this flag is obviously unused.
And you may or may not replace an object by reference when writing it.
And this creates a possibility of obvious.
Well, there's multiple equivalents realization.
So this creates a possibility of irreproducibility.
If you are not careful.
And like there's one obvious representation.
So this is the representation that replaces everything that is possible to be replaced by a reference
with a reference and then doesn't set any unused flags.
And so essentially we take the Python byte code files that we see.
They are written by Python and rewrite them in this way.
This is our cleanup phase.
And here I have a, so I'm showing the cleanup by taking the diff
between the output from the original byte code file and the clean up byte code file.
So for example, we had a flag number 0 here and we got three of it.
Of course, we don't need it.
And in general, we see that the flag numbers go down quite a bit because there's lots of unused flags written.
And here I have an example.
Here I had a line table that was an empty list.
And in the written version line table is replaced by a reference to list number 19,
which is actually here, which is also an empty list.
So we, well, yes.
So this is the written file that specifies it like this.
It's me and more version.
So that's, that's more than I said.
So this is like a question.
I don't know, like maybe do people who work on Python.
So Python itself could do this and write files reproducibly.
It would be, I think, nice for C Python in general.
But also, like, maybe we could, so when files are written like that, they shrink a bit.
In my tests, they shrink like 1% usually less than 1%.
And some files shrink a few percent.
So maybe this could be like a little optimization possibility.
But also I wasn't sure how much it is possible to replace objects by references to objects.
So, like, if I change that this is the string to an ask is string would Python care.
Or a string to a short ask is string.
There's also like, in terms strings and non-in terms strings would Python care.
Like, if we change the types and then put more references in place.
Longing, I mean, four Python integers to normal into short integers.
Can we change, like, if the exception table is a list and I replace it by a tuple would Python care.
And, oh, the tool itself is called Adent New Museum as I mentioned.
And it has this minus piece which that prints bytecode files in a fancy way.
And I think that this output is nicer than in any of the existing tools.
But it doesn't have a byte, it doesn't based on this internal representation that has in memory.
Like, if the object, it is about to write, it's also used internally linked reference somewhere.
Then it would, I think, put the flag on it, even though it's not actually referencing the byte stream.
So, doing this cleanly requires first writing everything and then going back over this treatment setting flags or unsetting them, whatever way.
That is, you probably cannot, you need to do some extra work to do this cleanly, right?
And the way the whole issue was discovered was that federal builds architecture independent packages on multiple architectures and then in certain situations compares that they are the same.
And people noticed that occasionally we actually get a different result on different architectures and
I think it's, I mean, I did not look at details in how C Python implement this, so I don't know what the exact reason.
Thank you.
So, one thing I could imagine that is causing this non-deterministic way of writing PYC files is the hash randomization that we use in Python.
Because whenever you process a dictionary, then, you know, they, entries may end up in different hash positions, so maybe not causing it.
I've heard of this problem before, other distributions have it as well.
And especially in the context of software security, like the supply chain security, this is actually a big issue.
So, it may be good to open an issue on the Python C Python tracker.
Okay, I mean, I, I mean, so the Python developers in Fedora, who are in touch with upstream or a part of upstream,
and I know about this issue, so I think that the issue is kind of known, but maybe not.
It is not, maybe there is an issue already, I don't know.
I've seen some discussion on discourse about this, so maybe it's already.
To answer the hash map part, the actual contents are in general identical, right?
I have, I haven't actually seen a case, okay, I have seen a case, but I think it's a completely separate issue.
But in general, the, like the objects themselves are the same, if you ignore the flags and the reference replacements.
So, we always get the same stream, so I think that if hash randomization changed this, then we would get different order, but we get the same order.
Well, at the end of the day, it's the same object, but the order may be different, and the order of processing may be different.
While writing the PYC file, and that may be the cause for these kind of strange things happening there with references not being used and stuff.
Anyway, other questions.
Hi, thank you for the presentation.
I was wondering, how do you ensure that the, that the before and after byte code is equivalent, so that it actually doesn't actually change your changes code?
So, I mean, relatively simple algorithm, right? I look at the difference between what is written, and I assume that.
So, the first version changed only the flags, so I would, I would see that it's exactly the same except some single bits and the sizes and change and so on.
And then when I started changing the references, I mean, it seems that it's the same Python behaves the same.
Of course, it's not exactly the same, but unless I messed up the algorithm to the replacement, it should, and being the fact that this is a references instead of the original object.
I mean, if this if this replaced correctly, and this doesn't make a difference, then it should behave the same.
Would it be a stupid same if you read before an answer?
Yes, yes.
I mean, we have, we have, like, for example, we did a massive build recently, so we ran the
full tests on thousands of, then, nine thousand of Python packages, and they all passed.
I mean, nobody reported an issue with that, so I think it will be defective, you know, some
difference in behavior.
Yeah.
My question was, first of all, why not go to upstream Python if this is a Python issue?
And then, another thing, I mean, if it's like a runtime issue, because it's a cache file,
why don't like ignore it and run it anyway?
I mean, it's, it's, it's because the cache of the other device, it's different than your
current device.
I mean, I don't get why you can get different results.
I say, like, it's not a built-in issue, first of all, because Python is interpreted,
but I mean, it's a caching issue, issue, and it doesn't have an information about the device,
an information about the timestamp, how can it be different between different devices?
I don't know if I am saying that correctly.
So, okay, you're right, of course, this is a runtime issue, as far as Python is concerned.
We run this bytecode creation in a package build.
So, we make it a built-in issue by just running it in a specific place, and so this version
was the air-producibility.
It works completely fine.
So, this is not a problem that anything else is wrong.
It's just that we would like to be able to repeat the build process and get the exact same
bit for bit result, right?
So, this only matters for our specific situation where we want to have repeatable builds,
and to go to the very beginning, I mean, if this is a Python issue, it's not a bug, certainly.
I should have said that.
It's like a tiny difference in behavior that we care about for our very specific reasons,
and it could be a more feature-request.