Was Cain rejected because he brought the wrong sacrifice, or did he bring the wrong sacrifice because he was an unbeliever?
That Cain was an unbeliever is beyond doubt. The last two Scripture references to him state this succinctly: “Cain, who was of that wicked one” (1 John 3:12) and “Woe unto them! for they have gone in the way of Cain” (Jude 11).
It is also clear that he brought the wrong sacrifice. The description of Abel’s sacrifice as “more excellent” (Heb 11:4) has led some to deduce that Cain’s sacrifice was “excellent,” and Abel’s even more so. That is not so. The phrase translated “more excellent” is only one word in the original, and it does not contain the word “excellent.” Indeed, of the 56 times it occurs in the New Testament, this is the only time it is translated “more excellent.” When used to compare quality (as here), it simply means “greater.” An example of it being translated “greater” is in Matthew 12:41, where the Lord Jesus, speaking of Himself, says, “A greater than Jonas is here.” No one would conclude from this that Jonah was “excellent,” nor should they make such a deduction regarding Cain’s sacrifice from Hebrews 11:4.
Further, it is explicitly stated that “the Lord had respect unto Abel and to his offering: But unto Cain and to his offering he had not respect” (Gen 4:4, 5). For both men it is stressed that the Lord did not just look at the person; He also looked at the offering, and assessed it. Cain’s offering was found wanting. So the answer to the first part of the question – “Was Cain rejected because he brought the wrong sacrifice?” – is “Yes.”
As for the second part – “Did he bring the wrong sacrifice because he was an unbeliever?” – the answer must be “Yes” to this also. The statement in Hebrews 11 verse 4 regarding Abel brings this out clearly: “By faith Abel offered unto God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts: and by it he being dead yet speaketh.” The reason that Abel brought a greater sacrifice was that he came by faith. Thus, testimony was given as to his righteousness – that testimony as to the acceptability of His gifts being made by none other than God Himself; and it is on account of his faith, evidenced in the sacrifice that he offered, that he continues to speak. This verse inextricably links the sacrifice that Abel made to the state of his heart. The converse must be equally true. Cain’s offering was a result of, and a testimony to, the unbelieving condition of his heart.
So we do not have to choose between the two alternatives in the question. Both are true. However, it was Cain’s unbelief that was the core problem. His wrong sacrifice was the outward evidence of his unbelieving heart; and God, Who knows all hearts, rejected him.