#The tiny video at the bottom says volumes about their dynamics so I recommend

[manjuhitorie]

HOWLS Digest - Hitorie Skream! Interview

Interview by Hata Riee! Translated by Manju!

With evolution upon them, Hitorie show their fangs with confidence.

Summary: HOWLS

[First section: ygarshy fanboys over HOWLS. Discussions about the impressions and mindsets which went into the album. “Many of the songs on HOWLS are the result of my battles with painful, excruciating, frustrated feelings.” “Though in reality it was the most fun I've ever had, in regards to the music. We tried out things, we challenged new things... These bandmates right here, their ideas and curiosity were like saviors to me.”] [Second section: ygarshy fanboys over Ao and how liberating for his heart it is. Ao’s “We got shit-faced” backstory. Disclosure that Sappukei, Garandou Mae Zero Machi, Coyote and Ghost all were the result of unrestrained integrated jam sessions. Diligence in musically thrashing, all hail guitar cuddling! Members are glad that leader is getting nice and lax.] [Third section: The reasons those jams were able to cause such a bang. What values they had to throw away and which they had to adopt, such as simple melodies or not. Synopsis on the journeys they took to hone their new conjurations, to solve their frustrations and stagnations they felt with the band. wowaka being stupid and selfish by not sharing his problems per usual. STUPID. Band confesses to being socially awkward but still trying to understand each other subliminally.... Plus Idol Junkfeed lore/jokes, also SLEEPWALK]. [Fourth section: Overall the writer equates HOWLS lyrics to a discussion, with simple human language that is easy to convey. The power of words is more than just as a tool for expression, but music can help unlock those shackles of perception so don’t worry leader.... Balance, desires, future pavement laid - The “I think we’re so revved up that we could even explode right in this moment!” quote. It, it matches up with their new year’s resolution which was about living in the moment!]

Your new album blew my mind. Each song alone even has the power to strike into the minds of old and new listeners alike.

     wowaka: Happy to hear it.          Shinoda: It feels like we’ve created something really amazing.      wowaka: Even after many years of finishing countless pieces and thinking “This is good I did good.”... This time I’m especially smiley and giddy about this piece.      Yumao: We covered a way more broad scope of genre.      ygarshy: It feels like the people’s album.      wowaka: The people’s (laughing)?!      ygarshy: Up until now, as an honest listener I couldn't consider our pieces an object of interest, but this time I could listen forever and ever I like it so much.

In our 2016 interview for the release of IKI, ygarshy you had also exclaimed “I’ve been listening to it forever”. 

    Everyone: Ahahaha!     wowaka: Effort makes confidence.     ygarshy: Though the undertones are different. With IKI I had the perspective of a musician looking back upon their work with pride, but this time I’m uninterested in that sense. I’m akin to an uninvolved consumer, honestly enjoying my favorite album.     Shinoda: We ourselves barely understand this phenomenon.

Then let’s use this interview as a chance to divulge the mystery.

    Shinoda: Let’s do it.

In between your most recent full album IKI, there was the release of the mini album ai/SOlate (2017) and the single Polaris (2018), but when did you first start considering the possibilities of an album?

   wowaka: We had been on tour for ai/SOlate, The UNKNOWN-TOUR 2018 “Loveless", spanning over about four months... so it was directly after that. Our single Polaris was used as an ending theme for the anime Boruto (NARUTO NEXT GENERATIONS), which helped ease us into our next full album.

So you had been working on this album during the later months of last year.

    wowaka: Yes. Between about July and October.     Yumao: Though SLEEPWALK was an older song.     wowaka: Ah, that’s true.     Yumao: Windmill or Sappuukei even also, we had about one chorus in tow of each even before we began the hard-core production of HOWLS.

With IKI, wowaka said he went into a “I can’t stop writing songs” beast mode, but how did the pace pan out this time?

    wowaka: I struggled a lot. Writing the songs of IKI had been prompting revelations within me, for the first time I was finding the meaning behind being human and the fact that I’m alive. It was like the first time I properly stood on two legs. Positive vibes were popping songs out of me. But in contrast, many of the songs on HOWLS are the result of my battles with painful, excruciating, frustrated feelings.

I do see the subject of solitude shows up frequently. 

    wowaka: Yes. Though in reality it was the most fun I've ever had, in regards to the music. We tried out things, we challenged new things... These bandmates right here, their ideas and curiosity were like saviors to me.

Were you able to pinpoint the album’s direction during the prototype stages?

　wowaka: “What am I to do with these feelings?” was my only sense of direction (laughing).    Yumao: Though as we proceeded further and further, we did realize “We can add a bunch of different types of songs into this next one”.

   ygarshy: The demo for Ao was made fairly early in the scheme of HOWLS production, and it was so good I wanted to cry. You can interpret it as a so-called breakup song, but it’s the first time we’ve ever done such. So no matter the circumstances, I want to show this song to the world, I want to show our band’s becomings to the world. That was my impression. It was then when I let go of any ego I had as a musician. And I'm now able to see the full picture, to figure out what elements may match up with what song… Which not only brought the album into broader territories, but also freed me from so many restrictions.

So Ao has had those lyrics ever since the demo version.

    wowaka: Well it’s my first heartbreak song ever.     ygarshy: It’s such a good song that, I had contemplated long and hard as to how best turn these emotions into something universally understandable.     wowaka: My extremely personal experiences and universal concepts alike, they're both contacting by a hair’s breadth. As a result of me always spilling out my personal experiences in goops, I somehow truly am hoping that the band, our audience, and others alike will be able to take a plunge into the core of the person and the world within.

Was Ao a pivotal piece for any one else in the band also?

     wowaka: I’m pretty positive that when I was recording my guitar, Ao was the most emotional. I felt "Ahh, this is what I’ve always wanted to do”. This is probably the extremity I’ve always been aiming for. The singing and the melody are in the dead center. Yet, at the antipode is…. something else also. Effort went into incorporating an almost elaborate aesthetic. If I could only say one thing about this song, I want to tell you about how the recording all began with us saying something like “What if we turned the volume up really crazy loud?”. I have an amp which normally never sees any action it’s so gargantuan, but I turned the knobs up all the way from the get-go.     Shinoda: We got shit-faced.

It’s a ballad and yet you chose to start with the instruments loud, that idea is interesting.

    wowaka: Because it's a ballad, that's exactly the reason why I wanted to have the intensity to equate it. Then through the process of trial and error, were we able to create a sound unlike ever before. I feel like we stumbled into a strange trance almost.

   Shinoda: Up until now, I’ve always been the one who yearned to do the extreme. He (wowaka) has always been the one who acted as my safety net, but this time, his screws came loose too.     wowaka: No matter as to how elaborately we made it, no matter as to how we may have thrashed open our gates, the message still managed to hold “We are who we are.”    ygarshy: For me, I felt that as long as we have this song, everything we turn out okay. I was no longer questioning myself over trying to be this or that. During recording I pondered the question “What is my own ultimate talent, that only I have as a bassist?” and, the answer is that my bandmates and their skill. It makes no difference what I do: the bandmates around me are skilled so everything will turn out fine. That's why I'm no longer putting much time and worry into my bass playing. The song Sappuukei uses my first take, simply as was. And with Windmill I decided not to practice at all.    wowaka: Is that so (laughing).     ygarshy: I realized it works out well if I don’t think much and just return fire back directly at the moment. I think that’s all thanks to the existence of the song Ao.

I see. The mood which develops over the span of this album, it may be just as ygarshy says, it may be because you as Hitorie you discovered new methods and means.

    Shinoda: Well basically, we had pitted ideas against each other and cherry-picked from there. Like the guitar in the background of Coyote and Ghost’s chorus is something which would’ve definitely never ever been OK-ed before. I was permitted to have a go at throwing myself into spontaneousness.     Yumao: Hence, we’ve gained a lot more freedom bit by bit. 

So I’m guessing you set your guitars loud for Coyote and Ghost also (laughing).

    Shinoda: Yeah yeah (laughing).     Yumao: It didn’t even have a prototype either, we just huddled together to say “Ready… Go!” and played from there. I’m positive that Sappuukei and Garandou Mae Zero Machi both took the same route also. I remember discussing “We need some fast songs huh”.. So that was when Shinoda went and cuddled up with his guitar at home.     Shinoda: Though working from home comes with too many problems. After I come up with one phrase, the temperature of it won't match with the one before it, then I’ll jump into my own advancement, and before I know it it'll turn into an unsalvageable mess. The phrase I play behind the choruses all the time are already unique melodies in themselves after all. I’m amazed Leader is able to make vocals to fit on top of them.      wowaka: Cause it’s not vocals behind the guitar, it’s the guitar that goes behind the vocals.      Everyone: Ahahahaha!     Shinoda: With this song especially, I didn’t know what the vocals were until the designated vocal day. So when I saw the lyrics, the mass of words on the page was astounding. “Is this guy serious!?” I thought.

Yumao claiming an 8-beat time on this sort of aggressive song was yet another surprise for me.

     Yumao: If it were 3 or 4 years ago, I’m sure I would’ve played in the typical rock song 4/4 time. That idea also occurred to me but, my core values seemed to have changed.

Though you said you were ailed by how playing simple beats for the sake of the vocals unsettles your reputation as a drummer? During our interview for Polaris.

    Yumao: That was true. But with this album I think I was able to find an answer or something.      ygarshy: As we were writing, we often had moments where we’d agree that a simple phrase is best. Even if we have to tear away ourselves and our ideas, if it’s to conjure the melody conspicuous, in the end it still becomes one of Hitorie’s songs.

I’m starting to perceive how this album was able to evolve into such a masterpiece. It’s evident that each of your priorities and wants for the band have changed over the last 2-3 years.

      wowaka: I just remembered now but, after our designated ai/SOlate tour, after trekking over oceans and roads, we had found a concrete resolution in regards to our progress as a band. Which ironically made us feel frustration over ourselves as music creators. The songs we make made us think “Ah, we’re kinda always trying to do the same thing huh...” We felt we had to change that, or else we’re headed down a slippy slope. 

Ahh, I see how it is.

    wowaka: Then something struck me like lightning, so I took a sudden solo trip all the way to India. “What will happen to my mind if I throw myself into a place completely unknown to me, where even my language is rendered ineffective?” I thought... So specifically what did happen to that mind, you ask? Well, nothing really actually. They say that your perspective on the world will change right. Nothing that powerful came to me. Yet that's very reason I realized “Ahh, I’m simply the human called me.” That sentiment is seeping into the album this time I think. The album is somewhat nomadic, it feels a bit pessimistic, it’s dry and arid… and overall just so fuzzy.

Back then, did you discuss that stale stagnated feeling with any of the other members?

    wowaka: I didn’t. It was my own selfish inner implosion, like gaaah. So after the tour it was almost as if I went into shock therapy: going to India, going out drinking with people I don’t usually hang with, etc. Thinking back, I may have been in a bit of a daring phase.

Could that solve Shinoda’s initial puzzle of "We ourselves barely understand this phenomenon”? You each subconsciously clicked with a need to revitalize the band, and your hunt for more funk became your spunk.

    wowaka: It’s true that after spending 5 whole years taking various shots together, our collective conscience was “What are we gonna do next?”    Shinoda: It’s not as if we sat down and discussed it though, we each felt it in our soul.    wowaka: We all suck at sitting down and talking to each other though. So even choosing a theme and sticking to it itself is impossible for us, instead we’ve taken to each doing whatever we like, and that’s the kind of band we are.    Yumao: The bits and pieces we notice about each other, and the bits we don't, are half-and-half both in this album… That's why song-wise, there’s opening up and shutting up, both merging together as one. When we were writing we ourselves even saw where it was headed. And so the hope that “Maybe listeners will change their view on Hitorie after they hear this.” was hammered in.

—Onward. Shinoda participated in the production for Idol Junkfeed right?

   Shinoda: When production was in a slump, I chimed in “I’ll make a song”. I was told “Just make it a high-spirited one” and so I went.    Yumao: It has a Shinoda taste to it, but it also sounds as if it's parodying Hitorie.     wowaka: It’s like a song written by a Hitorie fan (laughing).     Everyone: Ahahahaha!    wowaka: I don’t know if I said that right but….    Shinoda: Naw, you’re right. 'Cause I did lean towards that. I definitely did want to make it a Hitorie-sounding song.    Yumao: These two handling the director job together was a refreshing sight to behold. The leadership was half-and-half but, the brains were slightly more on Shinoda’s side with it. Him going “This is good, but this is good too, I can’t choose what to do!” was hilarious.    Shinoda: I did something I’m not used to after all (laughing). Yet, in the end it still became Hitorie.     wowaka: If we put our strengths together, then it becomes our's. That fact was proven yet again by this song. Even if I’m not in the middle pulling the strings, this song gave me the confidence that “Ahh this band is capable of so so much”.

SLEEPWALK has a similar vibe to current trendy foreign pop music, that sound is by virtue of ai/SOlate isn’t it.

    wowaka: It’s a direct descendant of ai/SOlate. It’s a song that exposes how much of a music fiend and too avid of a listener I am. Before we started album productions I had trained by making about a singular chorus. At that time I would never believe I would be showing that chorus to the band, rather off completing it as a one-off. In the sea of all the various styles in this album, I came to realize “If I write a whole song solo then it will truly seal the deal, that this has become Hitorie”. In the end I really did write it all by myself.        Yumao: I had nothing to do with it.     Shinoda: I didn’t do anything officer.

As his bandmates, did you want to do it as a band?

    ygarshy: It was more along the lines of “We want to hear how it would sound, so please do it”. We typically play our instruments strictly based on wowaka’s demos, but no matter how much effort we put into being loyal to his original, there's parts which will alter. That's why we wanted him to make it exactly as he saw fit.          wowaka: Even if I produce it alone, there's also the saving grace in playing it together during concerts.

So you're planning on performing this song together in upcoming concerts?  

   wowaka: I sense that we must.

Was there anything you kept in mind as you wrote the lyrics for the album?

    wowaka: I feel it’s all a discussion. I've mentioned this in many interviews before but, confining words only to mean only their meaning - there's a me who holds utter abhorrence for the basic functionality of words. I feel like this world is wrong, handling words as nothing but tools and merely getting controlled by them. They don't really believe in the power of words. I'm still staunch on that idea but... Even so I know I have to face the reality of being understood by people. By the audience, staff, friends, lovers, parents, family alike. I had also talked about this in our past Polaris interview didn’t I, about how at the concert in Fukuoka I received a revelation from the heavens almost.

Ah, you said there was a moment when you figured out “This is exactly what I want to tell people!”

   wowaka: Retrospectively that was huge for me. That’s why I aimed to tell people things this time, I poured special consideration into how to share word through song.

The lyrics of the final track Windmill, “I’m here where I belong/We are here where we belong”: it seems cohesive that you want to tell people this, now that it’s the 5 year anniversary of Hitorie’s debut.

   wowaka: Yep. After all these years I have new stories to tell, and now that I’ve figured out how to write lyrics like those, I want a lot of people to see them, I want people who have never heard us to hear them. That’s how I’ve come to see it.

I see. After finishing this album, has it laid down the pavement for Hitorie’s future in any way?

   wowaka: This year will mark seven years since we originally banded up, and after spending this much time together.. at this point our moments are human, our perceptions are human, our emotions are human. Hence we have no clue what lies ahead. That's why each member will surely figure out how best to push the band, and they'll show me the way forward. I think we’re so revved up that we could even explode right in this moment.      ygarshy: Leader (wowaka) has learned how to say “It’s actually the best when you’re each going all-out on your own accords”. There were still times when he wanted us to play parallel to his image, but ultimately the times when we're playing freely, do turn out to be the best, he says. That’s ridiculously rivetingly huge. So we took that exact method with Coyote and Ghost and it turned out so good. If we continue just like this, I think the balance of our band will change exponentially. SLEEPWALK, the songs we are bursting out in, and Shinoda’s song alike, they all became one of Hitorie’s songs. So It gives me so much confidence to hear you say “It might just reach new listeners”.

~~END~~

Bonus video at the bottom!

To all viewers of Skream, we’re Hitorie! I’m the vocal/guitarist wowaka. - I’m the guitarist Shinoda. - The bassist ygarshy. - The drummer Yumao. wowaka: Us, Hitorie, in this 2019th… century…? Shinoda: Yes. wowaka: Are, this February, Shinoda: That’s it. wowaka: on the 27th, Shinoda: That’s right. wowaka: We're releasing our new album HOWLS! Shinoda: So give it a listen. (All members clap) wowaka: With 10 songs, that are all itchy…. Irritated.. Hairy? Fuzzy? Shinoda: Fuzzy. wowaka: Hmm, that’s still not right.. It’s more… animalish? Outlawish? ...Anyway, it’s fast-paced with sadness looming over, yet despite that, there’s still a bit of love and-. Shinoda: (Snickers). wowaka: My explanation is starting to make less and less sense (laughing). It’s thick, it’s good, we have confidence in how this album turned out. So we’d be happy if you could acquire it and give it a listen, please. All members: Please (bow). wowaka: Furthermore, starting on March 1st at Osaka’s Big Cat, and heading on and on to the final destination on June 1st at Shinkiba’s Studio COAST, we’ll be touring the country on Hitorie’s "Coyote Howling” tour. It’s our first tour in a damn while. So dear the people who will follow us anywhere- Or the people who may hate us- Shinoda: People who hate us? wowaka: There’s the chance they’ll come too right? Yumao: I think so. wowaka: For example when someone drags a friend along, who may not have much interest, Yumao: They think “It might be fun” and just tag along, wowaka: Yeah yeah, you might be able to have an experience like that! So by all means, come and play with us. Bye-bye~

[gobacktothe-classics-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[coloss-us-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[eve-evi-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[tvrdojezgreno-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[bruised-barbie-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[hoshimiii-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[v-ermouth-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[laesfingedeloshielos-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[un-chat-sauvage-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[iimio-high-blog1]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[golehyas]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[bloodyslytherxn-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[nankaih-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[solongdarling-blog]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

[fromthegrotto-blog1]

Surprising SEO A/B Test Results - Whiteboard Friday

Posted by willcritchlow

You can make all the tweaks and changes in the world, but how do you know they're the best choice for the site you're working on? Without data to support your hypotheses, it's hard to say. In this week's edition of Whiteboard Friday, Will Critchlow explains a bit about what A/B testing for SEO entails and describes some of the surprising results he's seen that prove you can't always trust your instinct in our industry.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hi, everyone. Welcome to another British Whiteboard Friday. My name is Will Critchlow. I'm the founder and CEO at Distilled. At Distilled, one of the things that we've been working on recently is building an SEO A/B testing platform. It's called the ODN, the Optimization Delivery Network. We're now deployed on a bunch of big sites, and we've been running these SEO A/B tests for a little while. I want to tell you about some of the surprising results that we've seen.

What is SEO A/B testing?

We're going to link to some resources that will show you more about what SEO A/B testing is. But very quickly, the general principle is that you take a site section, so a bunch of pages that have a similar structure and layout and template and so forth, and you split those pages into control and variant, so a group of A pages and a group of B pages.

Then you make the change that you're hypothesizing is going to make a difference just to one of those groups of pages, and you leave the other set unchanged. Then, using your analytics data, you build a forecast of what would have happened to the variant pages if you hadn't made any changes to them, and you compare what actually happens to the forecast. Out of that you get some statistical confidence intervals, and you get to say, yes, this is an uplift, or there was no difference, or no, this hurt the performance of your site.

This is data that we've never really had in SEO before, because this is very different to running a controlled experiment in a kind of lab environment or on a test domain. This is in the wild, on real, actual, live websites. So let's get to the material. The first surprising result I want to talk about is based off some of the most basic advice that you've ever seen.

Result #1: Targeting higher-volume keywords can actually result in traffic drops

I've stood on stage and given this advice. I have recommended this stuff to clients. Probably you have too. You know that process where you do some keyword research and you find that there's one particular way of searching for whatever it is that you offer that has more search volume than the way that you're talking about it on your website right now, so higher search volume for a particular way of phrasing?

You make the recommendation, "Let's talk about this stuff on our website the way that people are searching for it. Let's put this kind of phrasing in our title and elsewhere on our pages." I've made those recommendations. You've probably made those recommendations. They don't always work. We've seen a few times now actually of testing this kind of process and seeing what are actually dramatic drops.

We saw up to 20-plus-percent drops in organic traffic after updating meta information in titles and so forth to target the more commonly-searched-for variant. Various different reasons for this. Maybe you end up with a worse click-through rate from the search results. So maybe you rank where you used to, but get a worse click-through rate. Maybe you improve your ranking for the higher volume target term and you move up a little bit, but you move down for the other one and the new one is more competitive.

So yes, you've moved up a little bit, but you're still out of the running, and so it's a net loss. Or maybe you end up ranking for fewer variations of key phrases on these pages. However it happens, you can't be certain that just putting the higher-volume keyword phrasing on your pages is going to perform better. So that's surprising result number one. Surprising result number two is possibly not that surprising, but pretty important I think.

Result #2: 30–40% of common tech audit recommendations make no difference

So this is that we see as many as 30% or 40% of the common recommendations in a classic tech audit make no difference. You do all of this work auditing the website. You follow SEO best practices. You find a thing that, in theory, makes the website better. You go and make the change. You test it.

Nothing, flatlines. You get the same performance as the forecast, as if you had made no change. This is a big deal because it's making these kinds of recommendations that damages trust with engineers and product teams. You're constantly asking them to do stuff. They feel like it's pointless. They do all this stuff, and there's no difference. That is what burns authority with engineering teams too often.

This is one of the reasons why we built the platform is that we can then take our 20 recommendations and hypotheses, test them all, find the 5 or 6 that move the needle, only go to the engineering team to build those ones, and that builds so much trust and relationship over time, and they get to work on stuff that moves the needle on the product side.

So the big deal there is really be a bit skeptical about some of this stuff. The best practices, at the limit, probably make a difference. If everything else is equal and you make that one tiny, little tweak to the alt attribute or a particular image somewhere deep on the page, if everything else had been equal, maybe that would have made the difference.

But is it going to move you up in a competitive ranking environment? That's what we need to be skeptical about.

Result #3: Many lessons don't generalize

So surprising result number three is: How many lessons do not generalize? We've seen this broadly across different sections on the same website, even different industries. Some of this is about the competitive dynamics of the industry.

Some of it is probably just the complexity of the ranking algorithm these days. But we see this in particular with things like this. Who's seen SEO text on a category page? Those kind of you've got all of your products, and then somebody says, "You know what? We need 200 or 250 words that mention our key phrase a bunch of times down at the bottom of the page." Sometimes, helpfully, your engineers will even put this in an SEO-text div for you.

So we see this pretty often, and we've tested removing it. We said, "You know what? No users are looking at this. We know that overstuffing the keyword on the page can be a negative ranking signal. I wonder if we'll do better if we just cut that div." So we remove it, and the first time we did it, plus 6% result. This was a good thing.

The pages are better without it. They're now ranking better. We're getting better performance. So we say, "You know what? We've learnt this lesson. You should remove this really low-quality text from the bottom of your category pages." But then we tested it on another site, and we see there's a drop, a small one admittedly, but it was helping on these particular pages.

So I think what that's just telling us is we need to be testing these recommendations every time. We need to be trying to build testing into our core methodologies, and I think this trend is only going to increase and continue, because the more complex the ranking algorithms get, the more machine learning is baked into it and it's not as deterministic as it used to be, and the more competitive the markets get, so the narrower the gap between you and your competitors, the less stable all this stuff is, the smaller differences there will be, and the bigger opportunity there will be for something that works in one place to be null or negative in another.

So I hope I have inspired you to check out some SEO A/B testing. We're going to link to some of the resources that describe how you do it, how you can do it yourself, and how you can build a program around this as well as some other of our case studies and lessons that we've learnt. But I hope you enjoyed this journey on surprising results from SEO A/B tests.

Resources:

SEO Split-Testing: How to A/B Test Changes for Google

Do it Yourself SEO Split Testing Tool With Causal Impact

Case studies:

SmokyMountains.com

iCanvas

ConcertHotels.com

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!