Removing the salt and pepper noise is an active research area in image processing. In this paper, a two-phase method is proposed for removing the salt and pepper noise, while preserving the edges and fine details. In the first phase, the noise candidate pixels that are likely to be contaminated by noise are detected. In the second phase, only the noise candidate pixels are restored using an adaptive median filter. In terms of noise detection, a two-stage method is utilized. At first, a thresholding is applied on the image for the initial estimation of the noise candidate pixels. Since some pixels in the image may be similar to the salt and pepper noise, these pixels are mistakenly identified as noise. Hence, in the second step of the noise detection, the pixon-based segmentation is used to identify the salt and pepper noise pixels more accurately. Pixon is the neighboring pixels with similar gray levels. The proposed method is evaluated on several noisy images, and the results show the accuracy of the proposed method in the salt and pepper noise removal and outperforms to several existing methods.