Utilization is calculated by measuring the amount of times that amplitude exceeds a certain threshold, per frequency, averaged over the selected timespan.
This screenshot shows where you can select the timespan, the amplitude threshold, and where the utilization graph is displayed. The graph is a percentage, chopped up into 284 buckets (the amount of steps in a full sweep), and represented as a line:
The Channels Table also gives a Utilization percentage, this is simply an average of all of the steps that were in the respective 20 MHz channel:
Unfortunately we don't have a straight mathematical equation to provide, but to reiterate, we calculate utilization by averaging the amount of times that amplitude is detected over a certain amplitude threshold, averaged over the selected timespan (basically an average on an average).