Paul Ryan's biggest defense against being deposed as speaker is that being in charge of the House Republican Caucus a terrible impossible job that no one wants. People will criticize him and threaten his ouster, but unless someone is really gunning for the job, it is Ryan's until he either resigns or until the Republicans lose their majority in the House.
