The increase of converters in the multi-terminal flexible HVDC transmission system makes the topological structure of the network increasingly complex, and the problem of resonance instability becomes more and more prominent. The traditional DC admittance analysis method cannot effectively analyze the coupling effect of the AC and DC sides, and cannot distinguish the system instability problems caused by different reasons such as the AC side, the DC side, or the AC and DC coupling effect. Based on AC/DC hybrid admittance, a stability evaluation model of multi-terminal flexible HVDC electric power transmission is established. On this basis, the generalized Nyquist criterion based on Determinant is introduced to judge the stability of the system and classify the different causes of system instability, including the interaction of AC side, DC side and AC/DC side coupling. A method has been proposed for effectively identifying the causes of unstable systems.Different impedance reshaping strategies are proposed to stabilize the system instability caused by different reasons. The results showed that the proposed method can effectively identify the causes of system instability and maintain system stability.