Abstract: In order to solve the shortcomings of low efficiency and large amount of calculation when traditional methods are applied to back analysis of geotechnical engineering, simplex algorithm and simulated annealing algorithm are directly combined as a hybrid optimization algorithm. The exact penalty function model is established based on the errors of measured and calculated values. The finite element program ABAQUS is embedded in the optimization algorithm program as a separate module, and the finite element optimization back analysis program is developed. In order to improve the accuracy of inversion parameters, a material model subroutine was developed and applied to engineering numerical model. The optimization method and material model subroutine are applied to the given engineering examples. The results show that the method is feasible and the search efficiency is significantly improved compared with the traditional method. It can be used for parameter inversion analysis of geotechnical engineering.