I have a personal rule that, after five rejections, a piece should be shelved. That doesn't mean it will never see the light of day, but it does indicate to me that something is fundamentally wrong with the work and will need rethinking after it's lain fallow for a time. I find (to my surprise) that I'm not too proud to do a gut-renovation on a story.
The number five is arbitrary, but has felt right so far. By five rejections, I've usually figured out what's wrong, if not exactly how to repair it, thanks to my own contemplation and the comments of editors when I'm lucky enough to get any.
I may be making an exception soon, however, to the rule of five. One story that's been making the rounds keeps coming back with reactions along the lines of, "We really enjoyed your story. Good luck publishing it elsewhere." Everyone likes it, so I can't imagine how to fix it.