Although I only went to level 3 on the skills. I didn't stick with armor that didn't max my physical resistance. You need to move fast on the dungeons. I skipped a few chests to use the stress points to just look around more.
Check out Cyderia's post. Follow it to the T and you will have no problem picking up all the badges in one sitting. 60 points in about an hour. Thank you!