1
Pick out an image and be sure that it is one single layer. If it is a background layer to begin with, promote it to a raster layer. This can be done under "layer" in your menu bar.  

See the image to the left:  Under effects go down to geometric effects and click on circle.  Your square or rectangular image will now be a circle.

2
Once you have circled the image, create one black layer and one white layer underneath it so that you can see what the edge of the circle looks like on a light and a dark background.

View your circled image with each of the backgrounds so that you can see the rough, pixelated edge which would show later in your completed artwork if you don't get rid of it now.

3
Now you will want to pick out the correct selection type. Open up the selection tool. Under selection type, pick Circle.  

4
Start with your tool cursor as close to the center of your circled image as you can.  Now while holding down a left click on your mouse, draw outward until the selection is just slightly smaller than your circled image.  It probably won't come out perfect, but you will fix that soon. Notice how my first example is out a bit off center.

The Move tool is highlighted here.  To move the selection into place, click on the Move tool.  Place the move tool inside the selection, and while you hold down the right click on your mouse, pull the selection with the mouse into the right position. You are trying to get an even edge outside of the selection area.  That is the part that will be smoothed.

The second example in this section  is after I moved the selection into place with the Move tool.

5
Using the Selection Modifications:

After you have your selection circle centered, look at it to see if there is an edge hanging outside the selection so that there is room for smoothing the edge of your image. My second image in the section above is just about right.

If the selection area is too big, you will want to use the contract function you will find under Selections>Modify. See the example at the left. And if the selection area is too small, you will want to use the expand function. and expand the selection.  

When you have the selection looking like my image in the section above, then you are ready to go into Selections again and choose Inside/Outside Feather.

6
In the Inside/Outside feather, you want to choose Both and feather amount 1. Depending on the particular image you are working with, your may have to redo this at a higher feather amount. But don't use more than you need.
Also, when using feathering to smooth edges, at times you might want to just feather inside or just feather outside.  If you try it one way and it doesn't work, try it another way.

You have one more thing to do so that you can delete the rough edge.  You must go to Selections again and Invert the selection.  

Now,  click delete and the rough outer edge will be removed and you should be left with a smooth edge.

7
Now look at your circled image with the white background and with the black background.
Are you happy with the edge now?  You can always set the feather higher if you want to.  But be careful that you don't get the edge too blurred.