Yes, inmates can send as many letters as they want through the regular postal system. There are no restrictions on how many outgoing letters an inmate can write, and the mail goes out through USPS just like any other letter.
The cost falls entirely on the inmate's end, not yours. Postage stamps are available through the facility commissary and come out of the inmate's trust account when purchased. A standard first class stamp covers a regular letter anywhere in the country, and the inmate buys those the same way they buy anything else from commissary, through their weekly or biweekly commissary order.
For inmates who have no money on their books, the facility is required to provide basic letter writing materials and postage for what are called indigent inmates. That typically covers a certain number of free stamps per week along with paper and envelopes, enough to maintain basic correspondence even without outside financial support. The threshold for indigent status and the exact amount provided varies by facility and system, but the principle that an inmate without funds should still be able to communicate with the outside world is a recognized standard across most correctional systems.
On your end as the recipient, you pay nothing. The letter arrives in your mailbox like any other piece of mail. There is no collect mail system for physical letters the way there is for phone calls.
If you want to make sure your person always has stamps and writing materials available without relying on the indigent supply, keeping a modest amount of money on their commissary account covers it easily.