At least AJ and Greg stayed to fulfill their commitment and obligation. Oh, wait ... they were not threatened to lose their starting position!!!
One fact about Shea is that he gets intimidated when there is competition for the QB position. Patterson most likely felt that there was a good chance that he would not start at QB next year at Ole Miss, especially since Ta'amu performed so well in the role in addition to Luke not guaranteeing him and his father the QB position without competition.
If you look at past history, Shea Patterson has not been at one place for more than two seasons since he was in the 7th grade. He runs when he is threatened and feels like he is good enough to have the job without working for it.