Stamps, envelopes, paper, and pens are all available through the commissary at most correctional facilities, and the pricing is generally reasonable rather than the extreme markup you see on food and hygiene items.
On why you cannot simply mail stamps and envelopes directly to your inmate, the answer comes down to security. Stamps and envelopes sent from the outside are harder to verify and control than items purchased through the facility's own commissary system. Some facilities do allow stamps to be mailed in under specific conditions, but most restrict it because contraband has historically been introduced through seemingly innocent mail items including stamps themselves.
The commissary system exists partly to solve this problem. Everything sold through it has been vetted and approved by the facility, which means your inmate can get what they need for correspondence without the security concerns that come with items arriving from outside.
For inmates who have no money on their books, the system has a provision for that too. Indigent inmates are entitled to basic writing materials including paper, a pen, envelopes, and postage to send legal mail and maintain basic correspondence. It is not unlimited, but nobody gets completely cut off from the ability to communicate in writing because of a zero balance.
The most practical way to make sure your inmate has what they need for correspondence is to keep a reasonable amount of money on their commissary account so they can purchase supplies when they need them. Even a small balance goes a long way for basic writing needs.