Instead of reducing the accuracy by 5% per rank, each rank increases spread by a specific percent per rank depending on the weapon. This effectively reduces accuracy, but may reduce it by greater than 5% per rank (according to the Arsenal UI).
Damage increase value of this mod stacks additively with that of Serration to a total of +330% base damage.
Since the two stack additively and Heavy Caliber costs 2 more mod capacity for the same bonus, combining the two is slightly less efficient in terms of damage per capacity cost. Maxed Serration grants +11.79% bonus damage per 1 mod capacity, while maxed Heavy Caliber grants +10.31% bonus damage per mod capacity. Combining the two, Serration and Heavy Caliber grant +11% bonus damage per 1 mod capacity, which is +0.79% less bonus damage per capacity than Serration alone.
The bonus damage from using Heavy Caliber with Serration is double that of using Serration by itself, though it comes at the cost of an additional mod slot and Heavy Caliber's slightly higher "base" drain of 6.
For example, if base damage is 100 then Serration adds 165% bonus damage and Heavy Caliber adds an additional 165% bonus damage, for a total of 430 damage. Thus, bonus damage is doubled for roughly double (+114%) the drain cost.
Accuracy reduction varies by weapon, even for those with 100 accuracy. For example, Ogris (which has 100 accuracy, decreased to 1.7 at max rank) rockets fly very far outside of the reticule, while Soma (which has 28.6 accuracy, decreased to 15.5 at max rank) spreads bullets considerably but don't stray too far from the reticule.
With the exception of Ogris, the final accuracy after installing Heavy Caliber can be found by 100 / ((100 / base accuracy) + 0.27 * (Heavy Caliber rank + 1))