Path tracer fireflies

I'm trying to make a path tracer but I have no idea how do I solve fireflies problem when sampling from env map. Env maps usually contain really bright and small areas and when some ray hits that really bright region we go a firefly. My image looks this:render with fireflies

I tried removing them by clamping max luminance of the pixel in the shader like following:

const float maxLuminance = 10.0f;float lum = dot(hitValue, vec3(0.212671f, 0.715160f, 0.072169f));if(lum > maxLuminance){    hitValue *= maxLuminance / lum;}

But this results in very bland and dark image:render without fireflies

So my question is: How do I keep those sun shadows from Env map like on the first image (best visible on the green wall) but eliminate all the noise?