1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to effectively run split testing of Adwords campaigns

Discussion in 'Google AdWords' started by fastimprovement, Nov 3, 2006.

  1. #1
    Ok guys, so here's a conundrum for you ...

    Whilst split testing on a particular Ad group I get the following stats from two of the creatives:

    Ad A - CTR = 6.8%
    Ad B - CTR = 1.2%

    This is measured from approximately 160 clicks.

    Which ad should be made the winner? ...

    ... and is this a trick question?

    H
    p.s. I'm actually using Taguchi methods for my split testing which means I can extract the winning parts of the puzzle from all my tests, but for this exercise let's just assume I'm using standard A/B split testing.
     
    fastimprovement, Nov 3, 2006 IP
  2. 30k Challenge

    30k Challenge Peon

    Messages:
    1,188
    Likes Received:
    54
    Best Answers:
    0
    Trophy Points:
    0
    #2
    The one that pays you the most money.
     
    30k Challenge, Nov 3, 2006 IP
  3. Terry1122

    Terry1122 Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    LOL

    I agree....The one that pays is the one that you keep!
     
    Terry1122, Nov 3, 2006 IP
  4. chrysostom

    chrysostom Guest

    Messages:
    152
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I think you also need to know how much each click pays. If click B pays 10 times as much, it would make sense to go with B.
     
    chrysostom, Nov 3, 2006 IP
  5. Terry1122

    Terry1122 Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Very True... But I was talking about Adwords
     
    Terry1122, Nov 3, 2006 IP
  6. fastimprovement

    fastimprovement Peon

    Messages:
    168
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #6
    And of course, we *are* talking about Adwords, and not a sniff of adsense anywhere ...

    and CPC is the same in each case.

    Anyway, here's some more info to go on ...

    Ad A - CONVERTS at 0.9%
    Ad B - CONVERTS at 9.4%

    ... does that change anyone's mind?

    H
    (of course, it would have been more fun if someone without such astute knowledge had answered first :) )
     
    fastimprovement, Nov 3, 2006 IP
  7. Micromag

    Micromag Well-Known Member

    Messages:
    1,757
    Likes Received:
    67
    Best Answers:
    0
    Trophy Points:
    165
    #7

    Go to ad A regardless of the conversion stats of your ads - never use pure conversion stats on your ads due you get wrong values - use your conversion stats for your keywords only.

    Let me tell why:

    If you have Optimize option on, AdWords will show Ad A for a group of keywords that has better CTR for ad A and will show ad B for the other keywords that has better CTR for ad B.

    So you have a virtual adGroup of keywords inside your adGroup: Group A and Group B - each one with his each CTR and conversion ratio that you can now see when you run reports for your ads A and B.

    Deleting one of your ads will not solve the conversion problem for these keywords but will have a high influence on CTR of these keywords as you already know .

    So think 2 times before deleting a high CTR with poor conversion ad - Ad conversion stats are still important but take care: if read your ad and figure out that it is not attracting wrong audience to your site don't delete it, delete the low CTR ad.
     
    Micromag, Nov 3, 2006 IP
  8. tarka

    tarka Peon

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Say the CPC is $0.10 after 1000 impressions:

    A 68 hits costs you $6.8 and you would get 0.612 of a sale
    B 12 hits costs you $1.2 and you would get 1.128 of a sale

    So as long as the profit on the sales outweighs the cost of the hits the surely you would keep both.

    The hits from A wouldn't be picked up by the searches on B's keywords, and vice versa. So wouldn't you just be throwing away profit?

    Edited to ask, is it the same keywords with different text showing or different keywords. I might have the wrong end of the stick about what a creative is?
     
    tarka, Nov 3, 2006 IP
  9. fastimprovement

    fastimprovement Peon

    Messages:
    168
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Ok, here's a little more information to clear a few more things up.

    A single ad group with a set of keywords (it doesn't matter how many) and 2 creatives (A and B), with the optimise option OFF (i.e. the ads are evenly distributed for each keyword, the closest you can get for the split testing measurements I'm taking).

    Micromag is right that if you have optimise option ON then Google Adwords would be giving more precedent to those ads/keywords that have a better clickthrough.

    The clicks from A and B are from the same set of keywords with Google Adwords distributing the impressions as evenly as possible. A creative is the ad text that is shown when a match occurs, and in this example we have two versions in the one ad group. So with that in mind Tarka, what if the sale value is only $1? Do we *still* keep ad A?

    H
     
    fastimprovement, Nov 3, 2006 IP
  10. tarka

    tarka Peon

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Thanks for carifying what a creative is.

    The cost per sale for A is $11.11
    The cost per sale for B is $1.06

    So you'd keep neither. :D
     
    tarka, Nov 3, 2006 IP
  11. fastimprovement

    fastimprovement Peon

    Messages:
    168
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Well here's a final twist, the return on the 'sale', the conversion, is $0 ... or rather there's no immediate sale value.

    Which Ad do we go for now?

    H
     
    fastimprovement, Nov 4, 2006 IP
  12. britishguy

    britishguy Prominent Member

    Messages:
    7,949
    Likes Received:
    892
    Best Answers:
    0
    Trophy Points:
    360
    #12
    Maybe need a few minutes to work that out I will put the kettle on for a cuppa
    Quote
    'so the tea bags have to be PG Tips' :cool:

    Then I will see if I know the Ad ;)
     
    britishguy, Nov 4, 2006 IP
  13. tarka

    tarka Peon

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    So really its cost per lead?

    It would make sense to stick with B.
     
    tarka, Nov 4, 2006 IP
  14. Grokodile

    Grokodile Peon

    Messages:
    425
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Cancel both of them... you are driving my prices up with your silly experiments!
     
    Grokodile, Nov 4, 2006 IP
  15. fastimprovement

    fastimprovement Peon

    Messages:
    168
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #15
    :)

    Don't worry, it's a thought experiment, not a real ad campaign -- although it's based on something that happened with an actual client.

    He first insisted that we delete Ad B because the CTR was too low.

    I showed him the conversions from his landing page.

    Then he insisted we delete Ad A because the conversion rate was too low!

    Now the real answer is that you cannot delete either of them because you don't have *enough* data to make a statistically significant decision.

    If you check out the actual figures, rather than the percentages, you're trying to make a decision based on only 3 actual conversions (136 clicks ad A = 1 conversion, 24 clicks ad B = 2 conversions).

    The split test ran for an additional 2 days and *then* we chose the combination which gave the highest conversions (which strangely enough was a combination of factors based on A *and* B) :)

    Anyway, hope that helps others ... the moral of the story being don't make important decisions based on incomplete data.

    H
    (now off to have another cuppa of PG-Tips!)
     
    fastimprovement, Nov 5, 2006 IP
  16. catchafire

    catchafire Guest

    Best Answers:
    0
    #16
    split tests for 2 days with 136 clicks?

    wow. not nearly long enough.
     
    catchafire, Nov 5, 2006 IP
  17. fastimprovement

    fastimprovement Peon

    Messages:
    168
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #17
    No, an *additional* two days ... 136 *wasn't* enough data, but there was plenty a few days after that!

    H
     
    fastimprovement, Nov 5, 2006 IP
  18. catchafire

    catchafire Guest

    Best Answers:
    0
    #18
    gotcha...interesting scenario and definitely intriguing to see how everyone approaches the problem differently.

    i recently had the opposite problem with a client, where they insisted on running a split test for more than a month. their ctr improved by 30% or so, but the conversion rate was just as poor. i have a finite number of ways i can tell someone that their site/offer doesn't work. after that, they'll just have to learn the hard way.

    another thought...when do you usually find yourself crossing the 'it's time for split testing' bridge?

    i always try to leave the ab testing for what i would call phase 3, for lack of a better term. meaning, after we've established the benchmarks, and after we've made basic campaign adjustments, and after we've established whether the site/offer are going to get us in the general vicinity of where we want to be. meaning, if the general goal is to generate leads at $20, and the site is converting at $200, i'm not going to worry about split testing as it's clear that we have bigger problems. i'm more worried about the keywords, offer, and site.
     
    catchafire, Nov 6, 2006 IP
  19. fastimprovement

    fastimprovement Peon

    Messages:
    168
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #19
    Clients ... who'd 'ave 'em, eh? :)

    Usually I set up a split test right at the start to test different combinations of creatives, but like you say getting the 'basics' right first ...

    H
     
    fastimprovement, Nov 6, 2006 IP
  20. MarketingJunkie

    MarketingJunkie Active Member

    Messages:
    117
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    58
    #20
    One of the questions I see most often when it comes to split testing is, "When is enough enough?"

    And while there are plenty of nifty online confidence calculators found on the 'net, one of the best formulas I've found on the topic was given by kea12345 (a member of WebmasterWorld.com) and offered this nifty calculation:

    Does this help answer your question?
     
    MarketingJunkie, May 20, 2007 IP