if I empty the factory exhaust pipe from mufflers, will it cause more vibration on the bike ?
If you buy pipes designed for Drifter, no. If you buy any other pipes then, YES... unless you change the right rear peg mount which is also the pipe hanger. or make a rubber mount hanger.
https://www.vulcandrifterriders.com/rubbermount.html
https://www.vulcandrifterriders.com/tbracket.html