Video: Recommender

Learn how to use Cogneato to improve a business metric of a social media recommender system.

Tutorial: A/B Test

The header of a measurements table describes the system being experimented on. The first column of this table says that you are comparing two versions of the system named "A" and "B". Your business metriic is "clicks".

Your measurements
version:A,B Categorical variable (string) Format: name:value_1,value_2,… clicks:successes Metric successes (integer) Format: name:successes clicks:tries Metric tries (integer) Format: name:tries A 45 100 B 35 90

The table reports the number of successes -- when a user clicked on an ad -- and tries -- the number of times a user saw an ad. These numbers -- successes and tries -- are numbers you would cull from your web server logs, enter into a table (ex., on a spreadsheet), and report to Cogneato.

Try reporting the table above to Cogneato: Click the copy icon , then paste your clipboard into the measurements paste bin below.

See this tutorial on Google Sheets.

See this tutorial on Colab.

Tutorial: A/B/n Test

Expanding beyond a simple A/B test to more versions -- also called arms -- is as simple as adding more arm names to the header.

Your measurements
version:A,B,C,D Categorical variable (string) Format: name:value_1,value_2,… time_spent:mean Metric mean (real) Format: [-]name:mean time_spent:se Metric standard error (real) Format: name:se A 123.4 9.8 B 67.8 10.3 C 145.6 8.2

Note that in this scenario the user has yet to measure the time_spent under version D.

We used the name version again in this example, but you may use any name you like (no colons allowed, though!). For example, you might use more descriptive names for your measurements, like:

ad_placement:top,bottom,side Categorical variable (string) Format: name:value_1,value_2,… time_spent:mean Metric mean (real) Format: [-]name:mean time_spent:se Metric standard error (real) Format: name:se top 122.5 9.6

or

style:bold,intriguing,persuasive Categorical variable (string) Format: name:value_1,value_2,… time_spent:mean Metric mean (real) Format: [-]name:mean time_spent:se Metric standard error (real) Format: name:se persuasive 132.6 8.8

You may also name your business metric whatever you like. In the tables above it is called time_spent. The suffix :mean indicates that the column holds the mean of time_spent. The suffix :se says the final column hold the standard error of time_spent. The suffixes :mean and :se may not be renamed.

Try reporting the table above to Cogneato: Click the copy icon , then paste your clipboard into the measurements paste bin below.

See this tutorial on Google Sheets

See this tutorial on Colab.

Tutorial: Categorical Variables

A/B/n tests are great for testing multiple versions of a design, ad, model, etc. But in some cirumstances you might find that you have "versions of verions". For example:

• Your web page design has two choices of color palette and three different layouts
• You want to compare two architectures of an ML model, but you also have two different sets of features which either architecture could use
• You have four ad designs, each of which could be placed in one of three positions, top, side, or inline

In each case you could try running two A/B/n tests, one to compare each of the version "types". For example, you could run on A/B/n test to compare the fours designs, then run another A/B/n test to compare ad positions.

But when you compare the 4 designs you would do done so with the position fixed at one of top, side, or inline. Then you would test the 3 different position with the design fixed. You would have tested 4 + 3 = 7 combinations of design and position -- but there are 12 combinations all together. If the designs are, say, "bold", "intriguing", "persuasive", "funny", then the 12 combinations are: (bold, top), (bold, side), (bold, inline), (intriguing, top), (intriguing, side), ...

Alternatively, you could run an A/B/n test to compare all twelve combinations.

Cogneato makes this kind of experimentation simpler to organize, and more efficient to run. It's more organized because you can specify the different variables -- design and position -- separately:

Your measurements
design:bold,intriguing,persuasive,funny Categorical variable (string) Format: name:value_1,value_2,… position:top,side,inline Categorical variable (string) Format: name:value_1,value_2,… revenue:mean Metric mean (real) Format: [-]name:mean revenue:se Metric standard error (real) Format: name:se bold top 0.0121 0.0135 persuasive side 0.0166 0.0098

It's more efficient because Cogneato understands that the variables are different and uses that understanding to better allocate your measurements. (Allocation recommendations appear in the design table that is created for you after you click Analyze.)

Try reporting the table above to Cogneato: Click the copy icon , then paste your clipboard into the measurements paste bin below.

See this tutorial on Google Sheets

See this tutorial on Colab.

Tutorial: Ordinal Variables

As we saw in the categorical tutorial, using multiple variables -- ex., one for ad design, and one for ad placement -- can make an experimental optimization more organized and efficient.

Imagine if you were testing ads that appeared in a list along the side of a web page. Instead of taking the values "top", "side", and "middle", the variable position would take the values "first", "second", and "third" or, more simply 1, 2, and 3.

In this case, it would be reasonable to expect that the revenue produced in position 2 is somewhere between that produced in position 1 and that of position 3. You can tell Cogneato about this expectation by encoding position as an ordinal variable, like this:

Your measurements
position:{1..3} Ordinal variable (integer) Format: name:{lower_bound..upper_bound} revenue:mean Metric mean (real) Format: [-]name:mean revenue:se Metric standard error (real) Format: name:se 1.0 0.021 0.018 3.0 0.018 0.022

An ordinal variable is any variable that takes numerical (integer) values, like 1, 2, 3, ... You indicate that a variable is ordinal by the suffix of the header column, :{lb..ub}, where lb is the lower bound, and ub is the upper bound. position:{1..3} says that position may take a value 1, 2, or 3. For comparision,

• position:1,2,3 would encode position as a categorical variable that could be 1, 2, or 3. In this case Cogneato would not assume that the business metric at positions 1 and 2 are any more similar than the metrics at positions 1 and 3.
• position:{0..5} would say that position could be one of 0, 1, 2, 3, 4, 5
• level:{-1..1} would say that some other variable, level, could take values -1, 0, or 1

Cogneato uses the expectation that the performance metrics of nearby numbers should be more similar to make better arm suggestions. (Arm suggestions appear in the design table which will be displayed after you click Analyze.)

Try reporting the table above to Cogneato: Click the copy icon , then paste your clipboard into the measurements paste bin below.

See this tutorial on Google Sheets

See this tutorial on Colab.

Tutorial: Continuous Variables

The ordinal tutorial showed us that experimental optimization becomes more efficient when we appropriately indicate that a variable is ordinal. Cogneato expects metric values of nearby variable values to be more similar than metric values of far away variable values. For example: The revenue for ad positions 1 and 2 is likely more similar than the revenue for positions 1 and 3.

Not all numerical variables are integers, however. For example, you might want to optimize the weights of a model or a threshold in a decision rule. Weights and thresholds take non-integral values. Weights might lie in the range [0,1] or [-1,1], for example. A threshold on a unit normal signal might lie in the range [0,3] or [-3,3].

We call variables that take non-integral (real) values continuous variables. Similar reasoning applies to continuous variables as to ordinal ones: Nearby variable values lead to more similar metric outcomes.

Your measurements

weight_1:[0,1] Continuous variable (real) Format: name:{lower_bound..upper_bound} weight_2:[0,1] Continuous variable (real) Format: name:{lower_bound..upper_bound} weight_3:[0,1] Continuous variable (real) Format: name:{lower_bound..upper_bound} threshold:[0,3] Continuous variable (real) Format: name:{lower_bound..upper_bound} time_spent:mean Metric mean (real) Format: [-]name:mean time_spent:se Metric standard error (real) Format: name:se 0.5 0.5 0.5 1.0 85.2 22.3 0.25 0.5 0.75 2.0 92.1 20.4 0.75 0.5 0.25 3.0 64.3 30.1

In the table, weight_1 can take values in the range 0 <= weight_1 <= 1. You indicate that a variable is real with a header suffix: :[lb,ub], where lb is the lower bound and ub is the upper bound. Compare this format to that for ordinal variables:

• weight:[0,1] -- A variable weight takes a real value in 0 <= weight <= 1
• position:{0..1} -- A variable position takes a value of 0 or 1
• threshold:[0,3] -- A variable threshold takes a real value in 0 <= threshold <= 3
• rank:{0..3} -- A variable rank takes a value of 0, 1, 2, or 3
• bias:[-1,1] -- A variable bias takes a real value in -1 <= bias <= 1
• level:{-1..1} -- A variable level takes a value of -1, 0, 1

Try reporting the table above to Cogneato: Click the copy icon , then paste your clipboard into the measurements paste bin below.

See this tutorial on Google Sheets

See this tutorial on Colab.

Tutorial: Mixed Variables

We saw in the previous tutorials that Cogneato supports multiple variable types, each described by a different header suffix:

• Categorical -- Unordered text labels, like :A,B, :top,side,bottom, or :low,medium,high
• Ordinal -- Ordered, numerical, integral values, like :{1..3}, :{-3..3}, or :{0..9}
• Continuous -- Ordered, numerical, real values, like :[0,1], :[-1,1], or :[1.41,3.14]

Cogneato is flexible enough to support arbitrary combinations of variable types, too. For example:

Your measurements

intensity:[0,1] Continuous variable (real) Format: name:{lower_bound..upper_bound} num_objects:{0..3} Ordinal variable (integer) Format: name:{lower_bound..upper_bound} scale:[1,3] Continuous variable (real) Format: name:{lower_bound..upper_bound} version:old,new Categorical variable (string) Format: name:value_1,value_2,… color:red,green,blue Categorical variable (string) Format: name:value_1,value_2,… position:{1..5} Ordinal variable (integer) Format: name:{lower_bound..upper_bound} views:mean Metric mean (real) Format: [-]name:mean views:se Metric standard error (real) Format: name:se 1.0 0 1.15 old red 2 100 30 0.5 3 2.2 old blue 1 110 35 0.22 1 2.5 new blue 5 60 22

The table above combines variables that are categorial (color, version), ordinal (num_objects and position), and continuous (intensity and scal) into a single experimental optimization.

Try reporting the table above to Cogneato: Click the copy icon , then paste your clipboard into the measurements paste bin below.

See this tutorial on Google Sheets

See this tutorial on Colab.

Tutorial: Number of Arms

When you run an A/B test you measure your metric once for version A and once for version B. A and B may be referred to as arms. You take one measurement per arm.

The design of an A/B test simply says, "Measure both arms" (until the standard errors are small enough). The design for an A/B/n/ test is similar: "Measure all the arms".

For more complex experiments -- like ones with mulitple parameters, especially ordinal or continuous parameters -- it might be impossible to measure, or even define, "all the arms". Cogneato makes careful, efficient decisions about which parameters it suggests to measure. These suggestions are the arms, and they appear in the design table, which you'll see after you click Analyze.

In some situations, you might need to measure the arms one at a time. In some situations you'll be able to measure more than one at a time. This will depend on your experimentation system's scale and flexibility, and the costs and risks involved with taking measurements.

Below the measurements paste bin, you'll see Number of arms. Set this before clicking Analyze.

Tutorial: Maximize or Minimize

Not all metrics are meant to be maximized. Metrics like cost, risk, latency, and bounce rate, for example, should be minimized.

If you wish to minimize a metric, place a negative sign before the metric name, like this:

Your measurements
level:low,medium,high Categorical variable (string) Format: name:value_1,value_2,… threshold:[1,3] Continuous variable (real) Format: name:{lower_bound..upper_bound} -cost:mean Metric mean (real) Format: [-]name:mean cost:se Metric standard error (real) Format: name:se low 2.1 $2,342.50 $123.12

The table indicates that the metric cost should be minimized with the negative sign in the header -cost:mean. When no negative sign is present, Cogneato will maximize a metric.

By the way, you may include dollar signs ($) and commas in your numbers for readability. They will be ignored by Cogneato.