Action link: add / remove from one/many connections, not just set

If I've understood action links correctly (and experimented correctly) it looks like they can't be used to add / remove from a field that's the "many" end of a one/many or a many/many connection.

I have a table that's a collection of date/times (blocks of time that are used for interviews).

I want applicants to be able to specify the blocks of time that they are free to interview, so I have a many/many connection between the block table and the applicant table.

In a table for applicants I can show all the blocks that are available, and I can add an action link that sets the value of the connection to the logged in user.

However, this doesn't *add* them to the many/many relation, it wipes out whatever's already there, and replaces it.

So it doesn't work.

So what I think I need is a more nuanced version of an action that add the connected user to the relation if they're not present, and an action that can remove them if they are.