How to add a delimeter to smartparens properly

When I write ' in org mode it automatically closes it with another ' . But when I delete it (say I typed ' by mistake), then the automatically added ' remains. This is opposed to other similar signs like " and ( which their closing bracket is deleted automatically as well. Is there a way to fix it also for ' ?

I tried to add (sp-local-pair 'org-mode "'" "'") but it did not fix the issue.