You'll still be able to carry 160 pounds of equipment which is enough to carry a bunch of different weapons and still have some spare room for loot. If you don't care about Strong Back feel free to set it as low as 1. You will need at most 5 Strength and that only if you want the Strong Back perk.
So the SPECIAL stats are really only important as perk requirements. But it turned out to be just a slight inconvenience and I was playing on 'hard'. I gave my first character a 1 in Strength and Endurance because I wanted to challenge myself.
I was very dissapointed to see how little effect your SPECIAL stats acctually have on your character in the unmodded game.