Quantum Advantages of Teleportation and Dense Coding Protocols in an Open System