Enhancement of coherent Smith-Purcell radiation at terahertz frequency by optimized grating, prebunched beams, and open cavity