Is he just using me cause he has no one else at this moment

You know this question could be answered either way. "Yes, he might be", or "no he really cares". Let's assume he likes talking with you and seeing you at the visitation. He likes that you are showing him attention. There is a chance that this relationship can grow beyond where it would have if he was out. Maybe you get to know each other better and he becomes more into you. Life is too short, you have to let this play out.

I'm sure you have great instincts (your inner feeling already knows), you have to trust yourself enough to make the right decision. Give it some time, try not to be too hard on yourself. He might be a rascal and a dirty dog too, but I vote for giving it a try and see how it plays out. You are in control of this relationship and every inmate knows this fact. Since the inmate loses control of what goes on in the outside world and he will often try to keep that control over the phone or letters. Just trust yourself and what you see and feel.