Taking another look at stochastic rounding to prevent stagnation in computer systems

Alignment of bits in algorithms for stochastic rounding based on sums. The random bits are added to the significand m_t followed by the truncation of it. How the bits are generated and added depends on the implementation—we may only add the k bits to the top k bits of the bottom part of the significand and then use a carry-out bit to control the rounding of m_r after the truncation, or we may pack the k random bits in a word as long as m_t and add it to m_t using integer arithmetic: the propagating carry will cause rounding in the top p bits. Credit: Royal Society Open Science (2022). DOI: 10.1098/rsos.211631

A small team of researchers from the U.S., the U.K. and France has taken a fresh look at the possibility of using stochastic rounding (SR) in certain computer applications to reduce stagnation. In their paper published in the journal Royal Society Open Sciencethe group describes their approach to surveying the use of SR in applications such as machine learning.

The most common approach is rounding numbers down that are less than a certain value, while those that are more than that value are rounded up. Rounding 3.4, for example, down to 3 and 3.7 up to 4. Here, the rounding value is .5. In such cases, numbers that fall on the rounding value are chosen at random. In this example, 3.5 could be rounded up or down, depending on the whim of the person or computer doing the rounding. Computers also have to perform rounding operations when handling infinite expressions such as π. For humans, rounding up or down can be a useful tool for quick estimations—adding up all the prices of goods in a shopping cart, for example, to ensure there is enough cash on hand when heading to checkout. Rounding is useful for computers, as well, because it allows for making calculations with mathematical constants such as π, but it also introduces a problem—stagnation.

With computers, stagnation occurs when long sums of small quantities, such as 0.1, are lost to rounding. There are a number of approaches to the problem in general applications, but stagnation is a major issue with machine learning applications. In this new effort, the researchers are looking at the possibility of using SR in such applications.

SR is a rounding approach for computers that has been around for more than a half-century, but has seen limited use. Programs using SR round a given number using probabilities that are based on its distance from that number. As an example, the number 2.6 has a 60% chance of being rounded to 3 and a 40% chance of being rounded to 2. Such probabilities can be used, the researchers note, because they are that percentage “along the way” to the target number. 2.6, for example, is 60% along the way to 3. In SR, the midpoint is considered to be equally likely to be rounded up or down. In such scenarios, the direction is still considered random. Using this approach, the researchers suggest, could prevent rounding from going in the same direction too often and help avoid stagnation. The catch, of course, is the lack of true random number generators on most computer systems. To overcome this problem, the researchers suggest the use of multiple types of rounding.

© 2022 Science X Network