KPCA could effectively obtain the nonlinear characteristic of the data, and the selection of kernel function parameters directly affects the fault detection capability of the kernel function itself. A linear combination of the polynomial kernel function and Gaussian radial basis kernel function, which are common kernel functions, is as the mixture of kernels. The fault detection rate is taken as the fitness function for optimizing the goal. The kernel parameters of KPCA are optimized by the artificial bee colony algorithm. The approach is applied to detect the fault of rotor unbalance, and firstly the motorized spindle is analyzed in time domain, then the nonlinear characteristics of sample data is extracted by KPCA, thus the fault could be detected on-line by monitoring T² and squared prediction error (SPE). According to the analysis of experimental data, ABC optimization algorithm could effectively improve the fault detection rate than the dichotomy and particle swarm optimization algorithm.