Most people: afraid of the zombie horde because they can kill you.
Phyllis: afraid of the zombie horde because it’s a large, loudish group of people and he’d really rather not deal with crowds
Also, most of Phil’s resistances are because he lives in a crappy apartment, and can’t afford hospital visits or breaks. Although the poison one is because sometimes food is a little bad but he can’t afford more yet.