We propose a new technique for impulse noise filtering that can remove the impulse noises from color as well as gray scale images. We operate on the HSI (Hue-Saturation-Intensity) color model. Our algorithm has three Phases. In first Phase, we take a window W of size N×N (say, 3×3) and form two groups: group of color and group of colorless pixels. We select the group that has the higher count of pixels in W. This allows us to remove the noise due to the colorless pixels from the color pixels and vice-versa. In the second Phase, if the selected group is a collection of colorless pixels then we find the median pixel based on increasing order of Intensity values and we call this as a candidate pixel.