hmm, going off the plugins page, we're not the only ones, although I have a feeling all the settings are 'set correctly' its the plugin devs fault.
To leave it should we /sw leave.
will look into this further
[EDIT]
The issue is that falling into the void kills you, ie the plugin has no "has fallen off while in arena" (like survival games). I will try to compensate for this with a portal to spawn or command blocks, since then it might not count as having left.
HUMMM
Also, no grass underneath, its void.