This paper presents an approach that provides optimization of both gearbox kinematic arrangement and gear tooth geometry to achieve a high-density gear transmission. It introduces dimensionless gearbox volume functions that can be minimized by the internal gear ratio optimization. Different gearbox arrangements are analyzed to define a minimum of the volume functions. Application of asymmetric gear tooth profiles for power density maximization is also considered.