I had the same issues and stopped at that point… but to be fair, they were following the source material.
Which they were not required to do. They didn’t kill off others at the same point and invented Daryll.
Yes, they really meandered plot wise, but what are you going to do over eleven years? Really enjoyed the first season best.