I rinse my arms with water, thats all. I am never scared unless fragging zoanthids or palys. However, there are plenty of things that can cause allergic reactions, etc... so there is always a risk...
I would wear gloves when moving rocks to prevent cuts and infections... moving corals around or mounting them... no gloves!