Experimental optimization

Cogneato is a tool for experimental optimization. Optimization, here, means finding the best configuration or parameter settings of a system. Experimental means that it can take significant time to evaluate any given configuration. You evaluate a configuration by measuring some business or technical metric. For example:

A measurement could take minutes, hours, days, or even longer so it is important that you take as few as possible. Minimizing the number of measurements requires careful planning, called experiment design.

Using Cogneato for experimental optimization works like this: Cogneato provides expertise in experiment design and statistical analysis of measurements. The user brings their domain expertise to bear on measurement and decision-making (in the analysis step).

Below are some examples of experimental optimization, specifically Bayesian optimization and A/B testing, applied across diverse domains.

Many methods have been developed to tackle this problem in engineering and business contexts. Among them are A/B testing, DoE, RSM, and Bayesian optimization.

You can learn more from this book:

Experimentation for engineers: From A/B testing to Bayesian optimization A practical take on experimental optimization with examples from internet applications & advertising, high-frequency trading, and software engineering. Teaches A/B testing, multi-armed bandits, response surface methodology, and Bayesian optimization in a unified way, from the perspective of an engineer configuring or tuning a production system. Uses Python, NumPy, and Jupyter.

Use cases

Software engineering

• ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems
  "Experimental results show that ContTune reduces up to 60.75% number of reconfigurations under synthetic workloads and up to 57.5% number of reconfigurations under real workloads, compared to the SOTA method"
• Application of Bayesian Optimization in Router Port Testing: An Improved Port Scanning Technique
  "Bayesian optimization [...] can significantly improve the efficiency and accuracy of port scanning."
• Bayesian Optimization for auto-tuning GPU kernels
  "our search strategies generalize well and consistently outperform other search strategies by a wide margin."
• Autotuning PolyBench Benchmarks with LLVM Clang/Polly Loop Optimization Pragmas Using Bayesian Optimization
  "our autotuning approach outperforms the other compiling methods to provide the smallest execution time for the benchmarks"
• Bayesian Optimization of HPC Systems for Energy Efficiency
  "The resulting benchmark score enabled our system, named "kukai", to earn second place in the Green500 list in June 2017..."
• Machine-learning-assisted thin-film growth: Bayesian optimization in molecular beam epitaxy of SrRuO3 thin films
  "Our BO-based search method provides an efficient experimental design that is not as dependent on the experience and skills of individual researchers, and it reduces experimental time and cost, which will accelerate materials research."
• Constrained Bayesian Optimization with Noisy Experiments
  "accurate modeling of all the factors inside a processor is so difficult that the only reasonable way to compare the performance of two different configurations is by running A/B tests."
• [video] Automated Tuning of the JVM with Bayesian Optimization
  "One case saw a performance efficiency improvement of over 80%, measured by requests served per unit of GC time."
• [video] Performance tuning Twitter services with Graal and Machine Learning
  "This talk will show how much performance improvement we got by autotuning Graal."
• [video] Bayesian Optimization and Java Performance in Kubernetes
  "The Hotspot JVM has more than 100 tunables! ... This talk includes a demo and also the results that we have obtained so far."

Hardware Engineering

• Event-Triggered Safe Bayesian Optimization on Quadcopters
  "We evaluate ETSO on quadcopter controller tuning, both in simulation and hardware experiments. ETSO outperforms state-of-the-art safe BO, achieving superior control performance over time while maintaining safety."
• PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices
  "... a solution that optimizes configurations across individual hardware components using Constrained Bayesian Optimization in an energy-conserving manner. Our empirical evaluation uncovers novel facets of the energy-performance equilibrium showing that we can save up to 36 percent of energy for popular models"
• Battery Management and Component Optimization
  "The experimental results show that optimized mole fraction yields an increased conductivity by 9% compared to initial maximum value and thus reducing the battery resistance in synthesis stage."
• Safe Contextual Bayesian Optimization for Sustainable Room Temperature PID Control Tuning
  "The results of this work have an immediate im- pact on the room control loop performances and its related commissioning costs."
• Safe and Efficient Model-free Adaptive Control via Bayesian Optimization
  "We further demonstrate the proposed adaptive control approach experimentally on a rotational motion system."

Simulation

• Advances in Bayesian Optimization with Applications in Aerospace Engineering
  "We consider a finite budget of N = 40 iterations to conduct the optimization. At each iteration the value of the objective function (heat released) and the constraint (maximum temperature in the domain) are computed from the solution of a system of PDEs."
• Using Bayesian Optimization for Hardware Design
  "Applying this approach on an example where traditional methods to optimize blackbox functions don't apply, this method outperforms a naive approach in both number of samples and the quality of the solution found."
• Co-Design of Commercial Building HVAC using Bayesian Optimization
  "we develop a data-driven, black-box, co-design framework using Bayesian Optimization (BO) to co-optimize the system and control design variables for commercial building HVAC system."

Materials Science

• Self-Improving Photosensitizer Discovery System via Bayesian Search with First-Principle Simulations
  "...From a molecular space with more than 7 million molecules, 5357 potential high-performance PSs were discovered. Four PSs were further synthesized to show performance comparable with or superior to commercial ones."

• Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables
  "The chosen materials design examples ... materials selection and microstructure optimization for optimizing the light absorption of a quasi-random solar cell, and another on combinatorial search of material constitutes for optimal Hybrid Organic-Inorganic Perovskite (HOIP) design. It is found that [the methodprovides] ... substantial insight into the nature and effects of the qualitative factors."

Science

• Machine Learning to Support the ATLAS linac Operations at Argonne
  "A BO model was developed to maximize beam transmission using an initial data set of 20 randomly generated settings [...] successful [in] improving the beam transmission from ~40 to 70%.
• Bayesian optimization for fast and safe parameter tuning for SwissFEL
  "We showcase how we use our algorithm to tune up the FEL out- put of SwissFEL with up to 40 parameters simultaneously, and reach convergence within reasonable tuning times in the order of 30 minutes"
• Bayesian reaction optimization as a tool for chemical synthesis
  "Overall, our studies suggest that adopting Bayesian optimization methods into everyday laboratory practices could facilitate more efficient synthesis of functional chemicals by enabling better-informed, data-driven decisions about which experiments to run."
• Adaptive COVID-19 Forecasting via Bayesian Optimization
  "we propose a novel model-agnostic Bayesian optimization approach [3] for learning model parameters from observed data that generalizes to multiple application-specific fidelity criteria"

Medicine

• Optimizing Resource Allocation for Tumor Simulations over HPC Infrastructures
  "remove the burden of [guessing requirements] for [simulations of] tumor treatment methodologies ... [and] to [...] rapidly distinguish [...] drug combinations"
• Stagewise Safe Bayesian Optimization with Gaussian Processes
  "We demonstrate that STAGEOPT is more effective than existing safe optimization approaches, and is able to safely and effectively optimize spinal cord stimulation ther- apy in our clinical experiments."

Machine learning, hyperparameter tuning

• Practical Bayesian Optimization of Machine Learning Algorithms
  "We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks."
• Algorithms for Hyper-Parameter Optimization
  "The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported."
• Sequential Model-Based Optimization for General Algorithm Configuration
  "In these experiments, our procedure yielded state-of-the-art performance, and in many cases outperformed the previous best configuration approach."

Internet products

• Uber
• Netflix
• Twitter
• Meta (Facebook)
• LinkedIn
• Spotify