fix nails in Vinyl Flooring?
If possible take up the old floor ( best to do) and put down a new sub floor commonly called luan. But this can go over the old subflooring. 1/4 thick and smooth on one side. Try not to use screws since the screw head can show through the vinyl. Use a staple gun or hand stapler. Rent this anywhere tools are rented. The staple is designed not to come back up. You can get a hand one or pneumatic gun.Then do the vinyl after the seams are treated.. Any questions you can e mail me through my avatar.. GL
Mine has been nailed down 40 years now and not a nail pop is showing on a sheet of vinyl.. What is done these days is you can buy 1/4 inch plywood sheets (I like plywood over PDF) and use a screw gun and put down another sub-floor. using floor screws and make sure they are sunken slightly. Yes that means a lot of wood again. OR SOME of the nails are causing you grief. You can grind off the heads with an angle grinder(or pull those useless nails.as they are doing nothing anyways). They won't be a problem as they were not holding the floor down anyways. You may want to nail in another nail. In that case make sure it is Longer than what you pulled out. A 2 1/2 to3 galvanized nail and pound it down so the head is below the surface some(like a dimple) When new floor is laid down you use glue and that fills the dimples in the floor. The problem stems from floor joist spacing is too great so there is a great flex when one walks on the floor. If the floor joists are spaced far apart then the floor plywood needs to be thicker to keep the flexing to a minimum. You got wide joist spacing and minimal T&G plywood thickness. The nails can only come up because of the weight placed on the floor in the middle(between the joists) is like a claw hammer pulling nails. If you have forced air heating it is easy to see how thick the floor is because most vents are placed in the floor under a window. Take off the grate and try to see past the metal of the heating duct to see how thick the plywood is. Each layer in plywood is 1/8 In my jurisdiction , 5/8 is the minimum floor thickness of the plywood.. I lay ceramic or slate or granite floor tile. Regulation for that is 1 thick So 5/8+3/8=1inch. Then I pour a float on the floor(basically concrete) so now it is dead level and rigid. So that is an inch 1/2 thick before the 1/4 inch tile go on top.
If you can't raise the floor and don't want to remove all the nails then your best bet is to put screws in next to the nails. This will hold the subfloor tightly down and help to stop the nails from creeping up. intermittent moisture will still swell and shrink the wood though which will pop up the nails eventually.
I would replace the sub-floor with 1/2 inch plywood and screw it down with 1 5/8 drywall screws.