I hope it's on, yeah. So, hello, I'm Fantyshek, I'm product owner for the couple of projects
we'll be mentioning later on and I'll join by Christian from the TMT team and we'll be describing
or discussing how we are trying to polish the way from the user down, down, down to the
area to the real people. So, flow of code, that's the first topic then we would describe
the packaging and testing experience projects, we work on, then the term of shift left
ink and how we can, you can test, share the test and how you can collaborate. So, that's the
plan. This is the space we are working on or around, we are used to fit around a central
stream, but it works for other distributions in a similar way. So, we've really, after the development
happens on GitHub, GitHub, GitHub or any other instances and there are plenty of ways how people
test, there are GitHub actions, GitHub applications, GitHub runners and a lot of lot of stuff.
On the Federas field, there is Gitforge, there are actually currently, there is a move from
page, however it's pronounced, and to forge, you can, but not need to open a public rest to get
get-out code there and there are generic checks and also you can define your own,
there is also a possibility to run a test inside the RPM field, there are also post-built tests
and there are also compost tests and to make it more messy on the center stream, it's different as
well. So, we have to GitHub and here everything is supposed to happen before the magic
rest is merged. You can also have like a separatist and RPM build tests and yet there are
compost tests, but run completely differently than the previous one. And also when we are moving
down the stream, we are no longer speaking about the projects, but packages, we don't longer speak
about developers, but about maintainers, a lot of not-of-names. So, we have a group of related projects
that recently got closer to get under one umbrella and also I'm currently like overviewing
these projects and we call them packaging and testing experience, so the projects are related to
like packaging, testing and also ideally this like work for the program. So, this is the mess,
and this is the world how we are trying to treat this. So, yeah, we are all over the place
in a some way integrated between various fields, so we'll try to decouple that a bit. So,
one thing we are trying to do is to unify this. So, as I mentioned, there are a lot of
differences between all these steps. We are trying to hide this from the user, ideally.
Second thing, what we are trying to do is to make it more like a user-centered, so it's like a
workflow from the user perspective, and also ideally change the manner of tasks as much as possible.
So, it's like this change, change, and also ideally to not as exact as a 10 random projects,
but ideally one unified interface, etc. Plenty of works still to do, but we are trying.
Okay, so first, now I'll circle through the projects that are related to that just so, you know,
first one is copper, it's an arp-embelled system and project management, and it allows anyone to
provide an easy way, arp-em, yam repository, so if you know, PPA repository is from the
world, that's the similar thing. Unlike if the cogey or blue, this is like, can be used for anyone
to provide a repository. This one is log detective, log detective is a project is actually like a couple
of pieces together, and the basic goal is like an arp-em block analysis, so you have an arp-em,
your arp-embelled failed, and you have a huge log, and you want to get something meaning from
top of it. Yeah, there are a lot of experience people, but there are also a lot of people that
see this as like, oh, where is the thing that I should look, and also what should I do? So,
there is a way how to put a log to that service, and it will run it through an AR analysis,
and hopefully it will try to help you with that. Second piece is on the previous slide.
The second piece is that we are actually collecting the like human defined data, so we are also
collecting that you can provide your URL that you know, with the log you know,
click on the snippets, so mark the relevant bits, and then describe them. So, we are
trying to collect this database, so and providing the model out of it, so we can have a
better input for this. And the last bit is actually the turning this as a service for you,
so for sentencing merge requests. If there is an RPM built failure, you get this command,
that tries to like describe the problem for you, and the team is currently working on
getting these two federals, so close to upstream, so there's actually the place where more
programs are happening. Hopefully, you can get far, far to the upstream.
Third point is packet, which is an integrator or gitforge integration that tries to glue various pieces
together. It can act as a CI, either for upstream and in a month or so for federals,
git packages, but also it acts as a release automation, so if there is an upstream release,
you can get automatic releases to federals and sentencing, and the federals also I mentioned these
like gluing together and making a change in the activity, so if there is a disk, it would be
as merged, you have automatic codej build, then automatic board here, I've determined the
codej build succeed. So, as a CI, yeah, we are combining other two, so we have a GitHub application
and GitLab integration where I have books, and with that you can have RPM builds, either for a
copper or coggy, and then you can have testing farm tests, more about that later, either directly,
or using that VM with the artifacts, you have installed, there is also pilot for the image builder
integration that we talk about that in us, also, and also static analysis is available and we are
working on the logitative path, and I'll pass my work to Chris Scher. Okay, so another topic of here
is the TMT, it's a way to just manage all of your tests and your test environments, and the goal of
it is to make all of the tests that run in the CI also be able to run locally, so you can define
like more of the tests that you want to pass, and also where do you want to run them? I can
know, destroy the container, what package to install and so on, and the main one of the other
main goals is to share and reuse all of these kinds of test definitions, or share it like from
upstream to downstream, or like all across like other packages to run as well, and the last part
over here is testing farm, basically just run TMT tests inside the cloud, well, there's also like
reading the TMT and like providing any kind of harder that we need, like GPU or whatever,
and it also gives you like a more readable way of displaying the tests and how they felt.
Okay, back. Yeah, and I will take a look at the term of shifting left.
So what mean by that, that this is the flow of goal going down the stream, and we are trying to move
as much testing and variation back to the left. So it's happening right, and the development
is there, and it's not only like a move as a MV command, but it's ideally like putting it to the
left, to upstream, but also verify on each step, because there can like people, there can be
contributors and contributions happening all over the place, like on various of these steps,
so we can't rely only on the upstream, and also like the environment can change slightly between these,
but also like time happens, so yeah, slide different, also before something gets from upstream to
better, for example. So what I mean by that, on the left, that the like usual workflow, that there is a
pool request or a magic request happening in upstream, this gets merged, hopefully released,
and in some time it gets to fedora, and in some time also it gets to cento stream, and if the
user or the maintenance of the packages in a fedora and cento stream realize any issue,
they need to go back to the upstream, file an issue or back to send a pool request,
then there is a new PR that needs to be opened, and the circle goes back, then it gets to finally
to upstream release, whether it's cento stream, it takes a lot of time. So what we are trying to
promote is that let's get the feed back directly to the PR, so we are just making this a
much, much, much quicker. So yes, so what we are there are two parts of that, one is the like
verification, so learning, for example, build directly for upstream changes, and providing the
environment for that, but also ideally, as you mentioned, by TNT, you can also provide the tests,
and give them an upstream, and ideally with testing found, we are also providing the testing infrastructure,
so it's not like, hey, upstream, give this testing, but also like trying to help them.
Yeah, all that seems very nice, and we thought that like everyone will be happy about that,
like we'll be like upstream and downstream, we'll be closer together, and yeah, but we had a lot of
complaints for both of the sites, because yeah, not everyone is happy about that. So yeah,
there is like a people from upstream complaining about that it's getting more work, they don't understand
the distributions, and what don't want any file in my gate, from the other perspective, from the
maintainers, we are getting like, oh, we are also maintaining those, and don't want to care about the
view upstream, we are those that understand the RPM stuff, and too many upstreams to handle, and
we actually don't trust it in some time, then. So let's take a look at them, one, one.
So, more work for us, it's actually not so true, because the issue, if you see the arrows,
will happen, or you will get it nevertheless, but with the shift shifting left perspective,
you see just right when the development is happening, so when there is a new change,
you see that there is a problem with it, you can fix it, no context switching, you just have it
directly there. Also, we can help with the infrastructure, so it's not so tough for you. Second,
complain is that people don't understand it, and that's correct, and ideally there should be a
relation between upstream developers and downstream maintainers, trying to help each other in some
collaboration, and also with a few projects, we realize that there is a way how to, like,
include downstream maintainers to certain situation when they help us need it, so for example,
automatically if there is an RPM build that failed, we can just create the automatically,
comment that he, he, these people, that are from up downstream, please take a look at that,
so they are notified only when that something irrelevant is happening.
Yeah, I don't, there are projects that don't like any digital specific file, anything,
maybe some tests, so we have, we provide a couple of things that can help here,
we are kind of flexible and we're trying to be kind of flexible, so we can reference, for example,
we have also very, how to, like, load a lot of stuff, it's downstream, for example, spec files,
but there are absolutely, don't like that, streams, yeah, understandable, so we have also poor
workflows, for example, for the release automation, we also have a very, how to, how to avoid this,
and we just pull the, pull the stuff from upstream, trying to glue it, but yeah, we don't have that
history, from the other point of view, if you don't trust the upstream and you are maintainer,
you are just providing a list of patches and put trying to maintain it and you are just long-term
getting a lot of work for yourself and yeah, so ideally, you are just losing time and trying to
maintain something on top of upstream and yeah, a lot of work for you, same, if you don't
to do the real stuff, you're actually doing the same thing, you are just doing it elsewhere, so
ideally, you are helping to fix these problems right on upstream part, and if there are too many
upstreams, I've mentioned it already, the ideal case is if the upstream developers and downstream
are somehow in contact or trying to help each other communicating, I think there is a benefit
for both sides, so and long-term maybe the upstream developer will start understanding the basics,
so can actually start commenting, to downstream stuff, so hopefully long-term it's definitely for
it. So let's look at this kind of shifting left in practice with
how we do some test sharing, so there are two ways of sharing the test, one of them is like sharing
between upstream and downstream, so over here you just care about like do the installation, don't
care about like how the stuff you get to there, back it or whatever other CI will take care of that,
and it's kind of nice because like it can bypass some of the limitations that other
disk has, for example, in Fedora you cannot run tests that require internet over here you can,
and it keeps things a bit cleaner because you can also use some files from from the upstream
get repositories and their sources and so on. Another way that the test are shared is like across
different packages, the simplest thing is just run the same tests for all the packages,
RPM inspect, installability, and so on. Another one that some people have used is like some kind
of reverse dependency tests, let's say like one project has a built failure, one project has like
some tests defined, and then another project depends on the other one, then you can just run
backwards like the test from other projects and see if any of your changes like affect the other ones.
We can take an example with a concrete project like psychic build core, it's like one of
one project that has adopted this kind of workflow, and all of the definitions of the tests are
inside the upstream repo, and you can just see like just like some simple tests that
Python important see how it goes. They also implemented some tests that are run only in the upstream,
so for example like their Python, and this is useful for checking that it works with different
the GCC that is currently in draw height and will not appear and will catch some errors much quicker.
And with testing farm and packet all of those tests are run inside the GitHub actions,
and you get like some nice results, yes, smoke test, there is a version sure.
And they also have like all of the tests from the upstream does all of the Python so on.
On the upstream side same story, you also have like fed RCI or packet or whatever
run equal of this test, and then you can check the same kind of interface, you get testing farm
showing all of the downstream tests that were defined in upstream, around over here with a bit of
different context. Anyway, so get like all of the tests that are running for all the packages,
this is the instability one, and so on. Now, how can you get involved? Like we really,
we really like for other disorders to also like get this kind of like workflow of like shifting
left and like contributing more towards upstream so that we as fed RCI can also benefit and
maybe also you. So what you can do is like just like try to contribute more to upstream,
and maybe also like have this kind of TMT test. So in principle, this should work for all
distributions. If it doesn't work, please come to us and like let's try to make it work.
The files are like relatively simple. We'll probably have like some better documentation later on
of like the minimum setup, but it's basically just defining what package you want to install
and then the test over here, like at the Hello World and so on. And you can also run them locally.
So just do a command like TMT run and it will do the same thing on your locomotion.
And try to like find out like what is the basic kind of functionality the package does.
So maybe just a Hello version over there or whatever. And that would help like
unify all of the tests between distributions as well and share them all across us.
And kind of so like think of like what kind of like limitations you have in the CI of like
GitHub not having many runners or so on. TMT and test in front we can have like multiple machines
that you can run tests and like you can have a server and a client and you can do some
color operations as well.
Otherwise you can get the full is like contribute to like some of the shared tests. Like in
Federation CI we have this kind of shared RPM inspect test, installability and so on.
Sometimes they fail yes and we could get like some more people like looking into them and like see
how you can contribute. For us you can find us like other Federation CI matrix from and that's
about it. Yeah also in the package can also find all of us.
So there are any questions. So I can get a nice question before but as we have a lot of
the tests in both places they're getting understanding the outputs as a user and it's a making
a developer user is about how many things can you take in your team.
Yeah so the question was that if you came out of testing and maybe increasing that how to
understand the output yeah that's a tough one definitely like one is the representation yeah the
testing from visualization needs some love and we are discussing that so like the formal and
the second thing is that like it would be nice to get it to the right place so maybe like
in a good time you get this but yeah I can definitely definitely might help as a little
log detective there are some initiatives like doing a very similar thing from that and
Chris you might have more ideas. From the TMT side I think there is one way you can kind of improve
this is to manage your tests. So not having for example love for the pilots don't run all of
the pilots at one like just run individual individual tests and like have more concrete what has
felt then you can try running locally and then with only that test and see exactly what is
the minimum environment to reproduce also there's another thing could do is like TMT has this kind
of login operation so you did the TMT run and then login and whenever it felt it gives you a shell
script a shell environment and then can try debugging inside there. Yeah maybe to go to it
a TMT has this like higher concept it can't it's a different thing then for example
fighters and these frameworks it's like a more high level where you can document also the group
of test disciplines but the second thing TMT also allows to report the test somewhere the
issue is that we have so many reporting places and not every man uses that so that's maybe
what I would like to look at to maybe get rid of all the duplication also on the reporting side
because there are so many places where you can put this if anyone has a good idea better idea please
very sure we'll be happy to improve you are a specific one last question or any complaints
rather not thank you