Simple: outperformed and outclassed by alternate technology.

Man-portable rockets provide superior performance in a lighter package. If you need more firepower than the man-portable variety, vehicle mounted rockets are much deadlier than the recoilless rifle.

Consider the LAW rocket. Lightweight, capable of penetrating most armor that the recoilless rifle was capable of penetrating and easier to carry, as it did not require a long, cumbersome tube.

Moving up in technolgoy, we get to the TOW, M47 Dragon and other anti-tank weapons that are guided in flight and therefore much more accurate than the RR. There are also larger, even more accurate weapons systems that can be mounted on vehicles.

 

Most common recoilles rifle calibers were 73,82 and 107mm
We got disposable one shot 84mm AT4s and 83mm SMAWs today.

You can bring these up on rooftops as easy as medium MGs and blow up or weaken walls or doors.

 

My MOS in the Corps was 0351, one of the listed weapons of that MOS was the 106 Recoilless Rifle. I was on one in Gitmo in 1972. I shot it once at 3 55 gallon barrels welded together floating out in the sea, probably about 500 yards out. I just shot over the top of them. The rifles we had mounted on early 1950s style jeeps, with a holddown slot in the windshield and bracket. We also had Mule vehicles, like early ATVs. The idea was for two rifle crews to work together on a tank from both sides. Shoot the aiming .50 sighting barrel with a tracer round, make aiming adjustments if necessary, then fire the 106 round. The crew immediately had to manuever the gun back on the Mule and get out fast, because if you didn't kill the tank, it had a returning round at you immediately. You telegraphed your position with muzzle flash, and dust kicked up.
To go against tanks was probably a suicide mission.
It was used some in Vietnam for direct fire support of your immediate area.
Semper Fi