tattoosandtechhouse

[tattoosandtechhouse]

Efficiently copying files to multiple destinations

If you need to copy large files between two machines, combining nc (netcat) with pigz (parallel gzip) is an easy and fast option. But what if you need to copy the same set of files to multiple destinations? This is a common need here at Tumblr, for example when we need to spin up several additional replicas of a MySQL instance at once in a fast, automation friendly way.

You could copy from a single source to two destinations serially, but that takes twice as long. Or you could copy from a single source to two destinations in parallel, but that doesn’t end up being faster, due to contention for the network uplink.

Happily, there’s a better way, utilizing some standard UNIX tools. By adding tee and a FIFO to the mix, you can create a fast copy chain: each node in the chain saves the data locally while simultaneously sending it to the next server in the chain.

First, set up the last destination box. This one just listens with nc (we’re using port 1234 for example), and pipe to pigz to decompress, and tar to extract:

nc -l 1234 | pigz -d | tar xvf -

Next, working backwards along the chain, set up the other destination boxes. We do the same listen/decompress/extract combo; but before the decompress/extract stages, we use the tee command to also redirect the output to a FIFO.  A separate shell pipeline reads from the FIFO and sends the data – which is already archived and compressed – to the next destination box in the chain:

mkfifo myfifo nc hostname_of_next_box 1234 <myfifo & nc -l 1234 | tee myfifo | pigz -d | tar xvf -

Finally, on the source box, kick off the copy chain by sending the files to the first destination in the chain:

tar cv some_files | pigz | nc hostname_of_first_box 1234

In my testing, each box added to the chain only incurred a 3 to 10% performance penalty, which is a huge improvement over copying serially or in parallel from a single source.

We’ve wrapped this technique in some Ruby code that automates the process and verifies every step. Besides using it to spin up database replicas, we also use it as part of our process to split shards that have grown too large – a topic we’ll delve into further in a future post.

[tattoosandtechhouse]

I’ve done this countless times and I’m pretty sure I know the best way to do it. In fact, I just set up a demo for you while I was saying that. That’s how well I know this.

please don’t argue with me on this one (via devops)

[tattoosandtechhouse]

Me when any of the databases go down.

Oncall handling the outage - everyone else stands, looking at their monitors

by David

[tattoosandtechhouse]

"This dogs a fucking necromancer" is the best sentence ever.

[tattoosandtechhouse]

#cthulhu is keeping an eye on these #lemoncheesecakebars with #blueberries. #bmcthulhulegs #blackmilk #blackmilkclothing @blackmilkclothing

[tattoosandtechhouse]

Describing our network infrastructure

by aj, tero, empperi and others

[tattoosandtechhouse]

These #bmtartanredspartanslegs tho... #blackmilk #blackmilkclothing #nylon #plaid #leggings #thightattoo #lovethyneighbor #sailorjerry #traditional #tattoo #luckydragon #seattle #girlswithtattoos #instagood @blackmilkclothing

[tattoosandtechhouse]

Today was a good mail day. #r2t2 #teeturtle #starwars #c3po #r2d2 #deathstar #et #tshirt #cardsagainsthumanity #fuck #stickers #tendaysorwhateverofkwanzaa

[tattoosandtechhouse]

Systemd

How it’s presented to the community:

How Lennart says it works:

How fans see it:

How it feels:

by john

[tattoosandtechhouse]

Wisdom comes with time. Plus a little touch up on some leaves. #owl #hourglass #traditional #american #tattoo #owltattoo #traditionaltattoo #girlswithtattoos #tattooedandemployed #tattooedwomen #freshink #luckydragon #seattle #misfits #evilive #danzig #instagood

[tattoosandtechhouse]

My work laptop. Dell XPS 13 Developer Edition, Linux Mint, Cinnamon DE, Seamod Conky that I did some tweaks on.

[tattoosandtechhouse]

Well thanks #xda for the #birthday message! #android #birthdaygirl #yay!

[tattoosandtechhouse]

Morning walks on the dark side. #murray #americanfoxhound #lightsaber #darkside #starwars

[tattoosandtechhouse]

Thinking it's time for some back work... #girlswithtattoos #tattooedwomen #phoenix #backpiece #booty #blackmilk #bmqueenofheartslegs #blackmilkclothing #nylon #leggings #luckydragon #tattoo #inked #tattooedmom #seattle #instagood #instaink @luckydragontattoo @blackmilkclothing

[tattoosandtechhouse]

[tattoosandtechhouse]

Linux Mint Debian Edition, MATE desktop, conky by me. Whoever made that background, you are my hero.

[tattoosandtechhouse]

Fall sun on the hen and chicks! #snapseed on the minimal lighting adjustment, otherwise #nofilter =) #succulents #henandchicks #fall #autumn #pretty #plants #pacificnorthwest #pnw #igers_seattle #galaxyS5 #instagood #potd