If you select color image, the brush tip image is converted to grayscale. Any layer mask applied to the image doesn’t affect the definition of the brush tip.
If you want
to define a brush with soft edges, select pixels with gray values. (Colored
brush shapes appear as gray values.)