Somehow it seemed that the elders who were from the soul-eater organization were gathering more and more, so Sword Sect elders always denied these invitations and it was only at a meeting that Mount Fei Sect organized because none of those elders were invited.
These elders from the soul-eater organization were clever, and they used these refusals from the Sword Sect elders to show that they thought they were better than the others and didn't want to be around the elders of the weaker sects.
If it wasn't for Zhang Yi's warning these elders would go to a few meetings just to put an end to those rumors that could be bad for the sect's reputation, but knowing what might happen, the elders would prefer Sword Sect to look bad than fall into a trap.