Jump to: [ Overview ] [ Format ] [ Spreadsheets & notebooks ]

Documentation

Cogneato is a tool for experimental optimization.

Experimental optimization includes A/B/n testing and black-box optimization. It helps you find the optimal configuration, parameter settings, or design to optimize the metric you care about, whether it's clicks, revenue, latency, or anything else. Experimental optimization is regularly applied to

configurable software
ML-based products (ex., recommender systems)
web site design
cluster computing
trading simulations and systems
simulators of all kinds

Any project or system that is designed or engineered can benefit from experimental optimization.

Overview

Cogneato offers a simple interface to powerful algorithms for optimizing real-world, complex systems. You paste your measurements into the web interface, and Cogneato returns an analysis of them along with a design, i.e., a recommendation of what to measure next.

The analysis helps you understand how well you're doing and when you're done. The design ensures you take as few measurements as necessary to find the best settings for your system or product.

Format of measurements: Parameters and metrics

The header row of your measurements table gives Cogneato hints about the nature of the problem you're solving. For example:

The header row (bolded row) has two columns, which tell Cogneato:

version:A,B - I'm going to compare two versions (of my system, ad, model, etc.), which I will call "A" and "B".
revenue - I'm going to measure revenue for each version.

The names "version" and "revenue" are arbitrary. The parameter values, "A" and "B", are also arbitrary. These names just help make your table more readble.

What's required are the colon (":") and the comma. They help Cogneato understand your header.

version is a parameter, and revenue is a metric.

To get a feel for the format, see the tutorials on the home page.

Uncertainty

Since you're running an experiment, you'll probably take measurements that have some uncertainty. Uncertainty is characterized by standard error. Telling Cogneato about the standard error, which we'll abbreviate as se, helps Cogneato work more efficienly.

Let's say version A refers to a website landing page. You showed it to 107 users, and you recorded the revenue generated on each showing. You would compute the mean and standard error of those 107 revenue numbers and record them in a table like this:

The suffixes :mean and :se let Cogneato know which column is which. Note that we've also recorded a measurement for version B.

Parameter types: categorical, ordinal, continuous

The version parameter is a categorical parameter. Categorical parameters take values that are names (strings). You could imagine having more than two versions, say A, B, and C. You would describe them with this header column:

version:A,B,C

in the header. You could also use more descriptive names:

palette:reds,greens,blues

You could even use two distinct categorical parameters by making two columns:

palette:reds,greens,blues style:friendly,urgent,mysterious

Cogneato supports three types of parameters:

categorical - Categorical parameters take arbitrarily named values. Ex: version:A,B
ordinal - Ordinal parameters take integer (no decimal points) numerical values. Ex: pixels:{10...20} describes a parameter named pixels that takes values 10, 11, 12, ..., 20
continuous - Continuous parameters take numerical values within some range. Ex: percent:[25.5,75.0] describes a parameter named percent that takes values in the range 25.5 to 75.0

The "punctuation", i.e., the colon, brackets, braces, commas, and ellipses, are required to correctly describe the parameter types and ranges.

Spreadsheets & notebooks

You record your measurements in either a spreadsheet or a Jupyter notebook. These tools are convenient for the task of experimental optimization because:

Given that measurements take time (ex., hours, days) the measurements table will be updated infrequently and with few rows.
Spreadsheets and notebooks facilitate the analysis and communication of data.
Spreadsheets and notebooks are popular among people who run experiments. You might already be collecting your measurements using one of them.