The PID parameters in atomic force microscope (AFM) play a decisive role in the performance of the system. The selection of better PID parameters can improve the overall stability and accuracy of the system. In order to solve the problem of inaccurate adjustment of PID parameters in AFM system, an optimal control strategy of PID based on longicorn beetle swarm optimization algorithm is proposed. The relay feedback is used to analyze AFM system, and the stable operation region of PID parameters is obtained. Under the constraint range, the optimal controller parameters and performance indexes are obtained by longicorn beetle swarm optimization algorithm. It can reduce the inaccuracy of parameter adjustment and a large number of parameter adjustment work in AFM system, and the corresponding optimal parameter values can be obtained by changing the performance indices according to the actual demand. The simulation results show that the control strategy can significantly improve the scanning stability and accuracy of AFM system, and effectively avoid the sample damage caused by probe overshoot oscillation.