Let $|X|=n>1$. As the action is transitive, there is only one orbit. By Burnside's lemma, this is equal to the average number of points fixed by an element of $G$.
The identity of $G$ fixes all the $n$ points. If every element of $G$ fixed at least one point, the average would be grater than $1$.