It's more potential for behavioral issues as they grow. 4-6 weeks is a fear stage (they have another stage for a bit usually sometimes in the 6-10 month marks as well) and can cause lifetime dramas. You don't want to be too harsh since is a itty bitty puppy removed to soon but you can't expect it to understand why he was on the bed and couch when he was little and now you want him to get off. Stuff like that. Can cause anxiety and behavioral issues. The problem is he isn't up to date on his shots so going out in public isn't something you can do I would try to find owners and dogs you know and trust are healthy and vetted to help socialize him not just with your own older pup. My boy was removed from his litter early and is the best dog ever for me and a douche bag to everyone else and most other animals lol. I love him dearly but it's not always easy to manage. You seem like you are Starting to do research now so good for you!!!
I also have my big a chuck it ball when he would get mouthy. He chews them like gum there is always one close by. The ones without fuzz not like a tennis ball but the hard plastic chuckit balls.
Please post pictures. Not very nice talking about puppies and not posting a pic