Set the picture size as a ratio. Use width on the top of the ratio and height on the bottom:
3 in/5 in
Then set the size of the enlarged photo to a ratio in the same way
x in/8.5 in
Set the ratios equal to each one another using the variable x as the unknown width.
3/5=x/8.5
Solve for x by multiplying both sides by 8.5
x=5.1