The recent observations of rippled structures on the surface of the Orion molecular cloud (Berné et al. 2010) have been attributed to the Kelvin-Helmholtz (KH) instability. The wavelike structures that have been mainly seen near star-forming regions take place at the interface between the hot diffuse gas, which is ionized by massive stars, and the cold dense molecular clouds. The radiation pressure of massive stars and stellar clusters is one of the important issues that has been considered frequently in the dynamics of clouds. Here, we investigate the influence of radiation pressure, from the well-known Trapezium cluster in the Orion nebula, on the evolution of KH instability. The stability of the interface between the H ii region and the molecular clouds in the presence of radiation pressure has been studied using the linear perturbation analysis for a certain range of wavelengths. The linear analysis shows that the consideration of the radiation pressure intensifies the growth rate of KH modes and consequently decreases the e-fold time-scale of the instability. On the other hand, the domain of the instability is extended and includes more wavelengths, consisting of smaller ones rather than the case where the effect of the radiation pressure is not considered. Our results show that for λKH > 0.15 pc, the growth rate of KH instability does not depend on radiation pressure. Based on our results, the radiation pressure is a triggering mechanism in the development of the KH instability and subsequent formation of turbulent sub-structures in the molecular clouds near massive stars. The role of magnetic fields in the presence of radiation pressure is also investigated and it has resulted in the magnetic field suppressing the effects induced by radiation pressure.