You could use the 4" but you may make more issue for yourself by using it rather than not. Lets use a bong as an example, most all of us have used one so this will be the best explaination i can do probably. To get the smoke to fill the bong you must first inhale the air, making for a negative pressure, now you are not blowing on the bowl to get air and smoke to come through, you are just sucking on the other side. Your room would do the same thing, by evacuating the air from the room it will build a negative pressure that will pull in fresh air from your vent(s). So rather than dealing with 2 fans you have made 1 do the same thing and made sure that you have accomplished the negative pressure. By using the intake fan you could cause a positive pressure in the room, because the filter, hood and ducting is all reducing the amount of air the 6" fan is actually moving. So if the 4" fan is open and moving more air than the 6" that is restricted you have now made a positive pressure and containing smell will be impossible.
It is recommended that you have twice the size of passive intake as active exhaust, so you would need 2 6" openings for fresh air this way, but it is how most of us deal with sealing in the smell.