Here is a simple photo tutorial on curving the border of a rectangular shape image.
1. Fire your photoshop and open a rectangle image such as the example shown below
2. draw a rectangle, with the rounded rectangle tool. The rectangle will be place where the image shows
3. Double click on the layer with the rectangle and the layer style panel will prompt out. In the Advanced Blending area, slide the Fill Opacity percentage down to 0% and click 'OK'.
4. Back in the Layers palette, select the 'background' image(the dog) and right click it. Select duplicate layer and click 'OK'.
You should get a new layer of the image.
5. In the layer palette, search for this camera look like icon, 'Add layer mask', and click on it. Be sure that the 'background copy' is selected and you will get the following display.
6. now turn off 'background' image visibility view to off. viola~