For whatever reason, this is not full of screen, but you know, all right, hey, welcome
everybody to our little presentation about how to break things.
My name is Marco Salbe, I'm fairly technical.
Let's just keep all this, we don't have too much time.
Let's go short agenda, why break things and how to.
Why it's fun and it helps you reproduce bugs.
It helps you learn new software, you know, like you get a new database, you need to learn,
you can start breaking it and see what it shows in the looks and that gives you a very
good experience, right?
And then I think what everybody is interested here, it helps you test software.
And you may tell me, reproduce bugs, but if the disk is full, it's not a bug, right?
Thing is, what is nice to have is this, right?
Proper handling of errors and elegant retry or elegant abortion that gives us information
and that we know it's going to be handled in a consistent way in production.
So you know, this run out of space on this and it says I'm going to retry for minutes.
So it knows that space can come back.
So how you get to have this is, you need to have a test where the disk space is fully
consumed.
So what we're going to be looking at is precisely this.
This was a very long presentation, it was planted for 90 minutes, so I have to cut it down
quite substantially.
Or for that, I can give you all the slides later.
And we're going to go through the slides, but I'm going to be doing mostly demo.
So we're going to switch to terminal in a minute.
So we're going to see TCQ disk, toxic proxy, carried the FS and C groups, PR limit,
tasks set, C surface, and hopefully we'll get time to a trace.
So let's get started.
TCQ disk is the transport control Q discipline.
And it helps you inject latency, corruption, reorder the packets in your TCP stream.
So essentially, it's a really, really useful tool.
It's the Swiss Army Knife of networking is poking.
So, correct packets, again duplicate packets, limit transfer rate, etc.
Let's go very quickly and make that demo.
So I have a few, let's clear this out.
Just one second, I was keeping those alive.
So I have some Docker machines here, Docker, C, T, Mark, one, oops.
I'm already there.
I was already there.
Oh, that was.
Bash, Jesus, I'm sorry.
Docker, C, K, D, J, T, Mark, who was number two, Bash.
So I'm just going to do very simple, let me go copy, paste my stuff.
So let's start a listener here, good Lord, why are you not copying things?
God's of demos are not with me today.
Jesus, what's wrong with you?
Say that again?
Oh, Jesus, yeah, I got nervous, I'm sorry.
I'm new to this.
So I have a listener and then let's set up a stream that is going to push by it's
in there and you can see it's running hundreds of bits.
And what we're going to do is find out the interface for those Docker's and okay, that's
not going to be the whole thing perfect and then the bridge AD is going to give us something
going to work as I expected, Jesus, what a day.
Okay, I promise I practice, okay, that's good and then bridge, okay, that's good.
And now I can do, get the IP address, okay, and now I can actually use that bridge AD with
this EQ disk and I will add a Qdcpling to that device and I'm going to say delay, right?
I'm going to add some delay between 500 microseconds to 10 milliseconds to 25% of the packets.
And yep, there you go, and you know, my rate drop is substantially right and it's one
megabyte and we can show, oops, oops, this one, this one, this one, this one, it's here.
Okay, okay, so again, essentially, root is the root Q, netM is network emulator which
is the Qdcpling you are injecting and then the net accumulator delay is the function and
then the parameters and there are, you know, you can, again, instead of adding delay,
you can add a retrise or a read, sorry, reorder or you could add bandwidth limit.
So the different commands for the same netM, which is again the type of networker adding.
And you can see here, it is showing me that there is a Qdcpling netM on the root and it's
between 400 milliseconds, 400 microseconds and 10 milliseconds.
And then we can remove it, and there we go, and it goes back to 100 megabytes.
Okay, so again, this is one way to do it.
You could also do it at the host level, let me go out of here and then I need the V4, okay.
And these guys are using host network.
So, will you sit?
Okay, have it.
Oh, no, I did that one for taxi proxy, I'm sorry, I thought I'd have it for this eq disk.
Okay, then let's go to the next tool, which is, again, taxi proxy.
And, okay, many, many, okay, taxi proxy.
This was developed by the guys at Shopify.
And again, it also allows you to do networking errors, latency, limit bandwidth, trigger
timeouts, and it has a rest API, so you can actually reach it through curve or whatever
HTTP library you have.
And again, very simple, you create a configuration with a given name, oops, I'm sorry.
And then, what port the proxy will listen, and what port will your application connect
to go through the proxy?
And then you simply start taxi proxy like that, yes, I already have it running here, let's
see, let's do it local, the host here.
So, I had to run my container using net host, and I'm exposing this port, and I'm running
just the Shopify, oops, so do I make this bigger, come on.
Using the Shopify proxy proxy container.
And then, I will set up my container here, and set up my listener, what's one with you,
and oops, I already use, okay, why, oh come on, oh, I was also testing with net server,
but it was just we complicated, okay, so we have a listener, and on my other Docker,
I will do the pushing, and I'm not sure if I already have my proxy running, no, okay,
I have my proxy running, so I have to go there and start it.
Come on, and then I will have to, this will actually create the configuration, instead
of having in a JSON file, I will just have a command through the proxy proxy CLI, and let's
see how that goes, okay, and you can see on the left that it created a new proxy, and now
it's listening on the 666, on 999, I'm sorry, so I can do this, and you can see this guy
is on 666, and this guy is on 999, and it's pushing by through the proxy, so now we can
go ahead and add what it's called, toxicity, and a toxic, and so again, using the proxy
proxy CLI, I do toxic add, and I want to add latency, and I want to add 1000 microseconds
of latency, and I'm going to give that an aim, and it's going to work on the NC2 stream,
which if you notice is the one I use it here, so it corresponds with that name, and
oops, not there, here, and you can see that the rate has plummeted, and we can go ahead and
then remove the toxicity, the toxic, oops, again, but I'm sorry, and you can see that the rate
is at full speed again, okay, next, cherry DFS, it's hard to pronounce, has very fancy
and poetic license, and it can inject latency errors, and it can affect the specific IO
C schools, so there is a large list of C schools, I'll show you in a second, and this works
on top of PUSFS, so you will need to have PUSFS installed, and let's go, let's go, let's go,
I did more profuse, I created my oops, sorry, I created my application directory, and I
carried the back end directory, and then I run carried D, and I said on application that
you mount the carried D back end, and then I can do CD's last application, and I will run
C-Smanch, I guess, no, let me just copy from here, okay, I already run the C-Smanch
prepay, okay, so I'm just going to execute C-Smanch, and I'm inside the application that
I did, so C-Smanch is going to run on the fake file system we have, and there you have,
it's showing us some rate, and then I can go here, and I can say, I will do net estate,
and I'm going to check, it's running, and it should be running on 1990, so, and it's
there, it's established, so we have a working file system, and it has a set of examples for
recipes, very simple, it has like, these are examples, but very easy to extend, so I'm just
going to do a recipe, perhaps, delay, and we should see, why is it not working, hmm, that should
be working, I'm going to write this, why is that not working, well, that's quite unexpected,
I don't know why it's, if you seem to delay it, that's very weird, and actually it should
bring me back, you know, when I do recipes delay, it should bring me, it should give me my
prompt back, but this is quite embarrassing, okay, I'll skip this one, I'll come back to it if
time allows, oh, this is really cumbersome, oh, oh, oh, there's V, oh, I'm not quite sure, yeah,
I'm not quite patient that either, I'm right there, yeah, okay, let me, okay, I know, perhaps
the recipe is clear, no, something, so it looks like a bug in charity that really, it should work
just like that, I'm super sorry, promise I tested all these, have no clue what's not working,
let me just try one more, what if I do a fool, let's try this, all the recipes, who, no, that should
make it fail instantly, okay, again it does look like a bug, I'm not sure why, but I'll look
it up later, let's move on because we don't have much time, okay, C groups, you know, is what
you do containers with, and you can also limit a number of resources using C groups, and what
we're going to do is we have our Cs, Fs, C group, and a container, okay, oh, and here I created a slow
disk directory, you just create, you know, like let's look at a slow, and once you create the
directory, it will create a number of virtual files that are actually an interface to the C group
configuration, so now what we can do is we run that Csbench again, six-manage, okay, okay,
it's just not, and okay, and now I can, um, D-grab, the shigs, shigsbench, and I can do Echo 5, 6, 7,
6, 6, into our C group blocks, so I'm adding Csbench to this C group, and now I'm going to flottle,
and I can do that, you know, what is it here, so I will
find the ID of the device is 259C, and then you do, and this is not a slow disk, it's just a slow,
so I'm just going to send a fifth, I'm going to limit to 15 megabytes on the right BPS device,
so, and once you do that, oh god, why is not more working?
I, you hear my word, I tested this a hundred times, I got bored,
people, oh in a way, Csbench, 5, 6, 8, 5, 6, 8, 5, 3, okay, the gods of demo are not with me today,
okay, it's not going as I expected, I give you my word, no, no, I know, I have absolutely no clue why
this will not work, no, the device is fine, I just see it right there, and no, that was,
wow, okay, I don't know what to say, truly, truly embarrassed, you say that,
let me try with my original C group and say if that works, okay, no, not here, here,
if I do, so I added the PID and now I add the latency, wow, I can fully tell you it's how it works,
it's how it should work, okay, and I had a pure limit, let's see if this one works,
we don't have time, we have one, one minute, okay, I'm so sorry, I truly don't know why it didn't work,
like, but I can tell you they work, I tested a hundred times, and I guess I left something broken
in my previous testing, I'll be glad to sit down outside and show you how they actually do work,
I'm truly sorry for the miss help, and I will, okay, I will also answer any questions you have,
but I'm truly sorry that it didn't went as expected, I can share the slides, I don't have a,
you can upload on the website and then you can download on the website, okay, I will make sure they're there,
and I will also add the notes for all the examples, I am truly, truly apologetic, but