The small signal model of the modular multilevel converter (MMC) is an effective method to analyze the resonance problem of the MMC-HVDC system. When MMC is in steady-state operation, the arm current and capacitor voltage contain harmonic components of various frequencies, which have typical multi-frequency response characteristics, and traditional modeling methods can hardly reflect its complex multi-harmonic coupling characteristics. In this paper, the frequency domain small signal impedance model of MMC is established by the harmonic state space (HSS) method. The modeling process takes into account the influence of the main circuit parameters, controller parameters, circulating current suppressor and time delay, and can describe the internal multi-harmonic coupling characteristic of MMC, and the model order is low, it is easy to program and realize. Then, on this basis, using the equivalent impedance of the AC grid and the equivalent impedance of the converter, the wide-frequency oscillation mechanism and system stability of the AC and DC system are analyzed, and measures to improve the stability of the AC and DC system are proposed. Finally, based on the electromagnetic transient simulation model in MATLAB/Simulink, the established small-signal impedance model and the stability analysis of AC and DC system are verified.