This paper aims to solve the task of generating yes or no questions, which generates yes/no questions based on given passages. These questions can be used for evaluation automatically. We propose a double phases generation network that can identify specific phrases related to facts from the input passage and use them as auxiliary information for generation. Specifically, the 1st-phase prediction uses the extracted phrases as assistance to generate an initial question. Then, the 2nd-phase prediction utilizes an attention network to focus on the relevant phrases related to the initial question in the passage to generate questions that are more relevant to the specific facts contained in the initial question. Extensive experiments we performed on BoolQ dataset demonstrate the effectiveness of our framework.