A tuned mass damper (TMD) is a vibration control system used to reduce the structural responses to earthquakes and extreme wind loads. The performance of a TMD depends on its parameters, such as mass, damping coefficient, and stiffness. Therefore, several methods have been proposed to optimize the parameters of TMDs. This paper proposes a new method for optimizing TMDs’ parameters using the Mouth Brooding Fish (MBF) algorithm based on white noise excitations. The effectiveness of TMDs optimized using the proposed method and other methods in reducing the maximum displacement of a ten-story linear structure was compared. The results indicated that the proposed method could effectively find the optimum parameters of the TMD. The efficacy of elastic and elastoplastic TMDs optimized using the proposed method in the responses of linear and nonlinear 10- story structures was also investigated. According to the results, the optimal elastic TMD more effectively reduced the maximum displacement of linear and nonlinear structures than the optimal elastoplastic TMD. Besides, elastic and elastoplastic TMDs exhibited higher efficiency in reducing the maximum displacement of the linear structure than the nonlinear structure.