A vehicle's transmission (drive train) is geared to allow for optimum fuel efficiency when equipped with the proper and required tire and wheel size. It is not so much the type of tire, rather the complete diameter of the wheel and tire; which if not the manufacturer required size, may cause an increase in emissions, or in a best case scenario, not allow the engine RPM and wheel speed (MPH) to maintain the correct ratio (in rotation) as required by the smog test parameters.
We do not recommend installing any size of wheel and tire (overall diameter) other then what a vehicle's manufacturer has recommended/required. We do recommend however that you ensure proper tire pressure before the smog test. Low tire pressure will put a greater load on your vehicle when it's being driven on the dynamometer during the ASM smog check. Over inflated tires will not pose an emissions problem but will cause instability while being driven on the dyno; so we can't recommend that as well. As far as mud tires, besides a possible wheel size issue, a smog technician may refuse to smog check the vehicle because of fear of damage to the station's dynomometer's bearings.