what kind of milk were you giving him? cows milk is horrible for them and can cause more harm then anything, just as mentioned above they cant digest that well. You can give goats milk but that is more for nursing moms or pups at a year old there is no need for that really. { as a treat or something maybe }.
Sometimes they just get picky and will hold out until you add more to it , like kids throwing a temper tantrum. if you give in though you may have issues getting him to eat plain kibble again.
The best way I find if my dogs are just being picky or if they are ill is to offer something yummy like a peice of meat or hot dog or peice of cheese. If they turn that down I know something is up.
Has his bowel movements been normal? any chance of compaction? utd on all his shots and worming? I dont usually worry if my dogs wont eat a couple days UNLESS they are not drinking, or I notice weight loss. Anyways a vet check is never a bad idea , so If you have an appt already I would go see what they have to say.