I'm not sure I agree with that line of thought, at least without seeing the math. Some things are simple like cables. Yes, they have to be bigger, that's obvious. How much weight does that actually add?
Look at the accumulator to start. For a given pack capacity using the same cell, assuming similar cooling efficiencies, heat wasted in the cells is going to be approximately equal in either low or high voltage configurations. How good are your cell to cell connections? Take a very low voltage system, let's say 100V. That's around ~24 series cells vs. ~140 for 600V. You have around the same number of total connections, but far fewer series cells to contribute to the IR, with better current sharing for more parallel cells. Total pack resistance is easy to drastically increase from cell connections that are sub-par.
On the motor side: For a given stator and rotor configuration, torque is going to be equal, neglecting difference in copper fill. You probably would gain some torque with a higher voltage wind since copper fill would be slightly better with more turns of smaller diameter wire, but is it significant?
I don't know enough about the inverter design to make any comments on it. I'd love to hear about why higher voltages helps/hurts inverter efficiencies, though!