Yes, and you are far from alone in dealing with it.
What you are experiencing is one of the most common relationship strains that comes with incarceration, and it has very little to do with anything you are actually doing. An inmate sitting in a cell 23 hours a day has one thing in unlimited supply: time. And time without distraction, without purpose, without control over anything in their environment, has a way of turning into anxiety that needs somewhere to go.
You become that somewhere.
The mind inside a jail cell fills silence with worst-case scenarios. It constructs detailed, convincing stories about what the person on the outside must be doing, who they are talking to, why they seemed distracted on the last call. None of it is based on evidence. All of it feels completely real to the person thinking it. I have described it before as writing a science fiction novel in your own head and then getting angry at the people in it.
The painful irony is that the accusations tend to be worst with the partners who are most faithful. The person working extra shifts to keep the bills paid, turning down invitations to stay home and write letters, keeping to themselves exactly the way you described, that person gets accused because the inmate loves them enough to be terrified of losing them.
There is no quick fix for this. Maintaining your innocence calmly and consistently, without matching the emotional intensity of the accusation, is about the best tool available. Keeping communication frequent and warm helps too, because silence and distance feed the anxiety that drives the accusations.
It is one of the hardest parts of loving someone through a sentence, and the fact that it is normal does not make it any less exhausting.