Are you good at cleaning out your closet regularly?
I'm pretty good at keeping up with it (and Hot Mess from simply B organized's Laurie helps!).
I try to follow a 1 in, 1 or 2 out method. One easy way to do it is to actually WEAR the things you're unsure about. I've been hanging on to a sweater, and wearing it to work today sealed the deal that it's time for it to go. I'll wash it, take it for a trip to Clothes Mentor (or similar), then drop it off for donation if it doesn't sell.