In view of the stability problem of voltage source converter based high voltage direct current(VSC-HVDC) transmission system, the state space model of two-terminal VSC-HVDC is established under different control strategies of voltage source converter(VSC), and the oscillation modes of VSC-HVDC are comprehensively analyzed. The eigenvalue method is adopted to compute eigenvalue and damping ratio of 23 ranks state matrix of system without including phase-locked loop (PLL), and the main oscillation modes are screened out through mode frequency and participation factors. The results show that the system mainly contains the subsynchronous oscillation(SSO) modes of DC lines, SSO modes of q-axis current and supersynchronous oscillation modes of d-axis current in the AC side of converter controlled with a constant active power. When the active power controller of converter is controlled with a constant active power, the constant alternating voltage controller is better than the reactive power controller considering least oscillation modes and maximum damping ratio; when VSC is controlled with a constant direct voltage, the effects of the constant alternating voltage controller and the constant reactive power controller on the oscillation mode are basically the same; most oscillation modes of VSC-HVDC are mainly related to the constant direct voltage controller and the state variables of VSC controlled with a constant active power, which are affected by the control loop of the converter.