Checklist
So far, you have:
- Installed our Javascript beacon and verified that we are collecting data.
- Tested that you are receiving recommendations.
- Verified that the items and fields we are returning are correct.
A/B Testing Methodology
We strongly recommend using our user hash for A/B testing. If you are using any other method for A/B testing, we request that you get in touch with the LiftIgniter team so we can review your bucketing methodology and understand exactly how it might affect the interpretability of results.
Below, we include sample instructions that can be used for either of these two use cases:
- You want to overwrite an existing recommendation widget on 50% (or an appropriate percentage) of your users with LiftIgniter recommendations, while preserving the same styling.
- You want to power a new recommendation widget with LiftIgniter recommendations on 50% (or an appropriate percentage) of your users, you want to populate that HTML template with LiftIgniter recommendations. For the other 50% of users, you want to show nothing, replicating your pre-LiftIgniter user experience.
The sample code on this page covers not just the A/B testing but also all the other code components needed to launch LiftIgniter: registering and fetching recommendations, rendering recommendations, and tracking recommendations. You can read the documentations on those components separately in order to get more detail on how they work.
1. How we split Users
The standard approach with A/B testing is to split by user. This makes the most sense, because we want to compare a world where users consistently see our recommendations, versus a world where they consistently do not see our recommendations.
LiftIgniter SDK offers a user hash that you can use to split users between our slice and the control slice. The user hash is a 32-bit integer hash of the user cookie. It is accessible as:
$p('userHash')
LiftIgniter SDK also offers its own A/B test slice that is obtained by taking the hash modulo 100. This creates a total of 100 bins numbered 0 through 99. This slice is accessible using a callback function in our SDK:
$p("abTestSlice", {callback:function(x){console.log(x);}})
We strongly recommend using our user hash for A/B testing. If you are using any other method for A/B testing, we request that you get in touch with the LiftIgniter team so we can review your bucketing methodology and understand exactly how it might affect the interpretability of results.
Forcing A/B hash
If you would like to force a specific hash value to make Q/A or recommendation testing easier, you can add URL parameter
igniter_abhash. i.e. adding&igniter_abhash=10to URL on browser will force the hash value to be 10. So if you've setup theabTestSlicelogic to show LI recommendations when value is less than 50, setting it to any value b/w 0~49 will show LI recommendation.
2. LiftIgniter vs Your Recommendation (Or Nothing)
The instructions below will work in both the cases we cover, namely:
- You want to overwrite an existing recommendation widget
- You want to power a new recommendation widget with LiftIgniter recommendations
<div id="li-recommendation-unit">
<div class='recommended_item'>
<a class='headline' href='//url-1'>Title 1</a>
</div>
<div class='recommended_item'>
<a class='headline' href='//url-2'>Title 1</a>
</div>
<div class='recommended_item'>
<a class='headline' href='//url-3'>Title 3</a>
</div>
<div class='recommended_item'>
<a class='headline' href='//url-4'>Title 4</a>
</div>
</div>
Suppose you want to run an A/B test on this area: you want to compare the recommendations you show by default against LiftIgniter’s recommendations.
Step 1. Tag the items you want to replace (e.g. by adding li-widget-item to the class):
<div id="li-recommendation-unit">
<div class='recommended_item li-widget-item'>
<a class='headline' href='//url-1'>Title 1</a>
</div>
<div class='recommended_item li-widget-item'>
<a class='headline' href='//url-2'>Title 1</a>
</div>
<div class='recommended_item li-widget-item'>
<a class='headline' href='//url-3'>Title 3</a>
</div>
<div class='recommended_item li-widget-item'>
<a class='headline' href='//url-4'>Title 4</a>
</div>
</div>
Step 2. Add a mustache template matching the tagged units to your page:
Overwriting vs Rendering new widget
For overwriting an existing widget, the template below will be a HTML copy of the existing widget in a mustache template (to make sure that newly rendered recommendation looks the same).
For rendering a new widget, you can determine the template based on the HTML design you want. You can see some general suggestions on placement in our guidelines on Click-Through Rate (CTR).
<script type="application/mustache" id="recommended-item-template">
<div class='recommended_item'>
<a class='headline' href='{{url}}'>{{title}}</a>
</div>
</script>
Step 3. Write callback function to render recommendations
// Renders the recommendations and overwrites innerHTML of area marked with 'li-widget-item'.
var rendering_callback = function(resp) {
var els = document.querySelectorAll('#li-recommendation-unit > div.li-widget-item');
var template = document.querySelector('#recommended-item-template').innerHTML;
for (var i = 0; i < els.length && i < resp.items.length; ++i) {
// Basically Mustache.render(template, resp.items[i]);
els[i].innerHTML = $p('render', template, resp.items[i]);
}
}
Step 4. Put in the registering and tracking logic for all slices
var trackAlgo = function(algorithm) {
$p('track', {
// marked for replacement during A/B test.
elements: document.querySelectorAll('#li-recommendation-unit > div.li-widget-item'),
name: 'default-widget',
source: algorithm,
_debug: true
});
}
var abTestHandler = function(slice) {
// Slice is modulo 100 of the hash.
// So you can set slice to 0 and 100 to check LI and base slice respectively
if (slice < 50) {
// Register call to overwrite widget being tested with LI recommendations,
// and track metrics appropriately.
$p('register', {
max: 4,
widget: 'default-widget',
callback: function(resp) {
// You might wish to wrap the code in this callback inside jQuery, to handle load order issues
// See code above
rendering_callback(resp);
// track overwritten areas as 'LI' slice.
trackAlgo('LI');
}
});
} else {
// You might wish to wrap the code in this callback inside jQuery, to handle load order issues
// track marked areas as 'base' or original slice.
trackAlgo('base');
}
// Executes all registered calls.
$p('fetch');
}
// Decide which slice the current user is on.
// $p('userHash') hashes user cookie id to a random 32-bit integer.
// $p('abTestSlice') returns the absolute value of this number mod 100
$p('abTestSlice', {
callback: abTestHandler
}
);
Load order issues
Keep in mind that if your recommendations that you intend to overwrite have not loaded when the callback is executed, LiftIgniter's recommendations will not be able to replace them. You can remedy the problem by wrapping the execution inside a jQuery. For more, see our documentation on load order.
Notes
Specific names in Steps 1-3 will differ based on the names you are using for your widget design. In Step 4, the name of the query selector in the
trackAlgofunction also needs to match these names. The abTestHandler code is completely generic.
Check that tracking works
You can test that tracking works by setting _debug to true and seeing that you get alerts for the visible and click events. More detailed information is in our tracking documentation.
Making changes based on your needs
You should be able to use the above code as a starting point and modify it for other use cases. In particular:
- If you want to use your own bucketing strategy, edit the first line (the if condition) of your abTestHandler accordingly.
- If you want to take other specific actions based on the slice, include them in the if block of the abTestHandler (for our slice) and in the else block (for the control slice). Examples of additional actions you might want to take are callbacks to other analytics services so you can use them to verify our numbers.
You can observe the results of the A/B test on the LiftIgniter dashboard.