If you are still on your stock ball joints, you will have to replace them. I had the same problem with my passenger's side ball joint and I ended up having to replace it. I opted to change them all out.
The reason being, they are riveted into place from the factory. Once you cut the rivets, there is virtually no way to put the back on. I suppose you COULD drill out the remaining rivets and use bolts, but why not just replace them all at once?