Blog

Digital Marketing frustration with Multivariate Testing investment

What’s Your Data Telling You?

A/B or Multivariate Testing is not the new concept in the digital world. For over a decade now; we have been talking about these concepts and trying to bring them in the mainstream but, it’s been consistently neglected by the management and front-line teams due to various factors. However, their is been a tremendous behavioral shift has occurred in the last 3-4 years; which has not only created tremendous amount of opportunities for the existing Web Analytics firms to invest in this space and broaden their portfolio but spawned; lot of early stage start-ups in this space too. Some of the bigger players in the market are Adobe Test & Target, SiteSpect, Maxymiser and Google Web Optimizer.

In last 3-4 years, our team of digital marketing experts have hosted various Testing & Targeting workshops in APAC and North American region, and assisted various Financial, Insurance, Retail and Telecommunication companies in devising an actionable and economical A/B and Multivariate testing strategy without significantly impacting their monthly marketing budget.

Everytime, we host a workshop and get an opportunity to meet new prospects, we hear the same set of questions from the Digital Marketing, Content and Management Teams regarding their recent investment in the Testing and Targeting solutions and overall, their is been a tremendous amount of frustration and curiosity among the digital marketing teams to learn the nitty-gritty of the A/B and Multivariate Testing Solutions, and extract the decent ROI from their existing platform investment.

Here are some of the most common questions or challenges faced by some of the large and mid-size online companies, who have invested in the Testing & Targeting Solutions in the last 2-3 years:

1. How to identify what to test and what not to test?
2. We often get confused between running a Multivariate Test and bringing in Usability Specialists to conduct Usability Study.
3. What if we run a test and nothing significant comes out of it?
4. What defines significant?
5. We often hear the terms Confidence Level or Confidence Interval, Lift or Influence % from our Vendor consulting team but, we are not statisticians and don’t understand these terms?
6. Do we have any industry benchmarks that we should be looking at prior to devise our testing strategy?
7. We have a pretty decent understanding about our visitors behavior, and how do they engage and what content they consume on the website so; do you think we should waste time and resources executing a test or, simply go for a targeted campaign?
8. For how long we should run a test?
9. What defines the success and what metrics we should be focusing on?
10. We have a fear that by putting the javascript code on the pages would shoot up our monthly cost, and we don’t want that.
11. We are not getting much support and creativity from our content team and often times, they are busy and don’t have time to create alternative content to run the test.
12. The development team doesn’t understand the rationale of these solutions, and often times take a careless approach, which often results in higher operational cost.
13. Who should own this solution?
14. We don’t have right skills and resources in in-house to manage this solution.
15. We don’t have process in place to roll-out the Testing and Targeting solution across the organization.
16. Should we put the code on every page of the website?

Above mentioned is just a small list of questions that we often receive from our prospects or training participants and unfortunately, all these companies are spending over $100K/Annum on their Testing & Targeting solution, and getting almost no value out of their investment.

Some of the companies we met in last 12 months are paying over $75K-80K per quarter to their vendors, and not getting any significant value out of this recurring investment.

Now the question arises, why the heck companies are investing in these solutions when, they are not evangelized in this space and secondly, what the major enterprise vendors are doing to answer the above questions.

In the next few posts, we will be making an attempt to answer the above questions, and assist you in getting an ROI from your solution investment.

We are very keen to hear your frustrations with your Testing and Targeting solution, and what efforts you are making to overcome them?

Categorised in: Analytics

This post was written by sushantajmani