Your circle grows or shrinks depending on the level of your season. Everyone that comes with you, can't go with you. In other words, some people may fall off and that's ok. If God wanted them with you, he would've opened the door for them just like he did for you.
