You cannot remove the author of an experiment through the security policy or group association.
The author is in a higher hierarchy than the previous two resources, so to limit access to an experiment to its own author, a user with full control of the experiment needs to remove the author from the Sharing section of the experiment.
In addition, an author cannot self-eliminate as the author of his own experiment.