Upgrading a detector from YOLO-small to YOLO-medium

You're considering upgrading your object detector from a 'small' variant (e.g. YOLO-S) to a 'medium' one (YOLO-M). What factors would you weigh to make that decision?