Many to many setup

I am trying to set up a structure that allows users to have different permissions for different plans they are working on. For example a user may be able to fully edit Plan A but can only edit certain parts of Plan B.

My idea to achieve this was an intermediate table Plan Users that is connected to both users and plans and includes the check boxes for permissions.

Input form (inserts new Plan User connected to Plan)

This works great - the problem I am having is that there is nothing to stop me adding the same user multiple times to the same plan with different permissions.

From a database point of view the primary key should be Plans and Users in the intermediate table, but not sure how to enforce or set this up?

Anyone have any ideas how I can restrict an end-user from adding a user to plan that they have already been assigned to?