Even though you've commissioned the work, the artist will own the copyright unless you both agree something else. It is implicit in your commission that you will be able to use the painting for the purpose you commissioned it, that is, as an album cover, and that use would normally also include using the same picture to advertise the album. However you should talk to the artist about any wider use, such as posters etc, because selling copies of the artwork on its own constitutes a different use and is a right which belongs to the artist. Don't assume that he will be OK with it.
I suggest that to avoid any misunderstandings it would be sensible to put your agreement on paper; indeed if you want the artist to transfer the copyright to you (this is known as assigning copyright) then it must be in writing.
You don't need an elaborate contract, but you do need to make it precise about what is being agreed to. And of course you both should sign the agreement. Here are two examples of standard contracts you could buy if you really want to do things propoerly, although as I say, this may well not be necessary: Artwork Agreement and Assignment of copyright
Yes, as you can see I was confused!
Making a painting from a photograph or another piece of art is something which happens a lot and can often cause problems over copyright if you do not own the copyright in the original. This is because in UK copyright law it is still infringement to copy something even if it's done in another medium. The test is to see if a substantial amount of the original has been copied. Since I assume you want people to recall the Bob Dylan album, it seems likely that a copy would take a substantial part of the original. What is substantial is generally the essence or main message of the work being copied.
On that basis, I think that doing this for your private use would probably be bordering on infringement, but carries a relatively low risk because few people will be aware of the existence of the painting. But to try and sell copies commercially would be foolhardy unless you get permission. The most likely owner of copyright in an album cover will be the record company that put it out. They should certainly be amenable to the idea of a licence to make spin-off merchandise which is how they would treat this sort of venture.