The difference between A/B Testing (or split testing) and Multivariate Testing

There are two main types of landing optimization testing methodologies: A/B Tests (Split tests) and multivariate tests.

Here are the basics for the two main types of landing page optimization methodologies:

A/B Testing (or split testing), is the direct testing and statistical comparison between 2 or more pages. In A/B split testing, you are not testing basic elements on a page, you are actually testing the different between the performance of separate pages. In figure 1, this is an example of a test between two different pages. After a statistically significant amount of traffic is sent sent to these 2 pages over time, you will be able to determine which page performed better, and use that page as the main page for the pages you were testing.  A/B testing can be done between 2 or more pages at a time, depending on the amount of traffic you have.

Pros for using A/B testing:

• Easy to understand and implement.
• Good for smaller trafficked sites.
• Allows testing of completely new design changes.

Cons for using A/B testing:

• Not efficient for big sites.
• Does not allow you analyze why it worked better

Multivariate testing is the testing a different elements within a single page.  For example, maybe you would like to test 4 different headers, 7 different headlines, 2 different font styles, 4 different testimonials, etc.  This testing process allows you keep the same main page, and test specific elements on your page. I will not there are many ways in which you can do multivariate testing, focused factorial, full factorial, parametric, non-parametric, the Taguchi method, etc. etc. Google Website Optimizer, for example, does basic full-factorial testing (test every single combination of possibilities equally). Figure 2 shows a multivariate testing example, with the different elements that would be tested on a specific page.

Pros for using multivariate testing:

• Great for high-traffic sites.
• You will know EXACTLY what affected your site.

Cons for using multivariate testing:

• Will not work for a low-traffic websites.
• Much more difficult to understand and implement.

Those are the basic differences between the two main testing methods. The most important thing to remember: TEST, TEST , TEST!

Generally speaking, if you have a small site, use the A/B Split Testing method. For larger sites, BOTH methods should be used overtime (as well as other methods that are available, depending on the tools you choose). Start with split testing, simply because it is easier to implement for your first tests. Bot no matter what you do test. It will pay off in huge dividends!