There are many factors, including thorough wood preparation to remove machine marks, swirls, scratches & dents, etc. But when it comes down to it, the one factor that separates a great finish from an average finish is smoothness.

There are two significantly different types of finish: penetrating and film-building. A penetrating finish is one that doesn't harden, so all the excess has to be wiped off after each coat. Oil finishes are penetrating finishes. Oil finishes include boiled linseed oil, tung oil, and a mixture of varnish and one or both of these oils.

All finishes that harden are film-building finishes. They can be built to a greater thickness on the wood by leaving each coat wet on the surface to dry. The procedure for rubbing is different for oil and film-building finishes; I'll discuss both.

 Oil Finishes:

You can create a fairly smooth oil finish by sanding between coats using very fine grit sandpaper (#320 grit or finer). Be sure to allow each coat to fully cure, which means leaving overnight in a warm room. Some oil finishes, such as Watco Danish Oil and Deft Danish Oil, instruct to apply coats within an hour or two; following these directions won't produce good results. You can create an ultimately smooth oil finish by sanding each coat while it's still wet on the surface using very fine grit sandpaper. Then wipe off the excess and allow what's left to cure overnight.

Here's the procedure.  → Sand the wood to remove machine marks and other flaws.

→ Wipe or brush on a wet coat of oil and keep the surface wet for several minutes, rewetting any areas that become dull because the finish has soaked in.

→ Wipe off all the excess. Be sure to hang your wet rags to dry, or drape them singly over the edge of a trash can, so they can't spontaneously combust.

→ Allow the finish to dry overnight in a warm room.

→ Wipe or brush on a second coat of oil and sand the surface while it's still wet in the direction of the grain using #600-grit wet/dry sandpaper. Sand over all areas with three or four back-and-forth strokes. There's no gain sanding more than this.

→ Wipe off the excess oil and allow the surface to dry overnight.

→ Apply a third coat of oil and again sand wet. Remove the excess and allow overnight drying. This is usually all you need to do to achieve an ultimately smooth finish, but you can repeat the procedure with a fourth coat, and with as many additional coats as you want.

One caveat: Sanding an oil finish wet (or even sanding dry between coats) is risky if you have stained the wood. You might sand through some of the color, especially at edges. Sand lightly and carefully.

Film-building Finishes:  Film-building finishes include varnish, lacquer, shellac, water-based finish and two-part catalyzed finishes. Both varnish and waterbased finish have a version called polyurethane. This is the regular finish (alkyd or acrylic) with some polyurethane resin added. Catalyzed finish is also available in one part called pre-catalyzed lacquer. Except for varnish, each of these finishes hardens within a couple of hours in a warm room so several coats can be applied in a day. Varnish, on the other hand, requires overnight drying between coats.

The Sealer Coat:  The first coat you apply of any of these finishes is called the sealer coat. It stops up the pores and seals the wood. It also leaves the wood feeling rough, so you should always sand the sealer coat smooth. (Though you could skip the sanding and still achieve smoothness at the end by sanding just the last coat, it's easier to sand the sealer coat because it's thin.)

Varnish (not including polyurethane varnish) and lacquer are more difficult to sand than other finishes because they tend to gum up the sandpaper. So manufacturers provide a special product called sanding sealer to use as a first coat under these finishes. Sanding sealer is varnish or lacquer with a soap-like lubricant included. Sanding sealer powders when sanded. If you are finishing a large project such as a set of cabinets with varnish or lacquer, it will be worthwhile to use a sanding sealer for your first coat. But if your project is small, requiring little sanding, it's better to avoid using sanding sealer because it weakens the overall protection of the finish. The included soap weakens the moisture barrier and makes this layer softer than the finish itself. Instead of using sanding sealer to gain easy sanding, you can thin the finish itself about half with the appropriate thinner (mineral spirits for varnish or lacquer thinner for lacquer). The thinner layer of finish hardens faster so it is easier to sand sooner.

If you are finishing a wood with resinous knots (such as pine), or you are refinishing wood with silicone contamination (it causes the finish to roll up in ridges) or animal-urine or smoke odors, use shellac as the sealer coat. Shellac blocks off these problems (but it is not easier to sand). There's no reason to use shellac otherwise. No matter what you use for the sealer coat, sand it after it dries using a grit sandpaper that creates smoothness efficiently without causing larger-than-necessary scratches- most often a grit between #220 and #400.

Sanding Between Coats: It's always best to sand lightly between every coat of finish to remove dust nibs. This is done easily using very fine-grit sandpaper: #320. Using a stearated or dry-lubricated sandpaper is best because it clogs least. This sandpaper has the same soap like ingredient as sanding sealer and is usually available at auto-body supply stores.  Sand just enough so you can no longer feel the dust nibs.

Rubbing the Finish:  When you have applied all the coats you want, usually three or more including the sealer coat, it's time to make the surface feel smooth. If the dust nibs aren't bad, you can usually improve the feel significantly by simply rubbing lightly with a folded brown-paper bag. As long as you have allowed the finish to harden well (so you can no longer smell any odor when you press your nose against it), the bag will level the nibs without damaging or changing the sheen of the finish.

To create a more perfect and attractive surface, rub it with #0000 steel wool or gray Scotch-Brite. Rub in the direction of the grain. Rub the three or four inches nearest the ends using short strokes (so you are less likely to rub over the edges and cut through). Then rub the entire length, being careful to stop just short of the edges. You can achieve even better results by using a soap-and-water or mineral-oil lubricant with the steel wool or Scotch-Brite. Oil causes the abrasive pad to scratch the least, but cutting will be slower and the gloss attained higher. You can try one and then the other on the same surface to see which you like best. Most professionals use soap and water.

Be careful using any lubricant because if you cut through you won't see the damage until the lubricant dries. Rubbing with an abrasive pad, with or without a lubricant, improves the feel and appearance, but it doesn't remove the flaws; it just rounds them over and disguises them with fine scratches. To achieve the ultimate rubbed finish you have to level the finish first and then rub it.

Leveling and Rubbing:  Leveling a finish is a mechanical exercise employing the same exact procedure as sanding wood, with two differences: You use finer grits of sandpaper and you use a lubricant with the sandpaper to prevent clogging. Here is the procedure. Using a flat sanding block to back your sandpaper, sand the surface until it is perfectly flat. Use a grit sandpaper that cuts through the flaws efficiently without creating larger-than-necessary scratches that then have to be sanded out, usually a grit between #320 and #600.

Use wet/dry sandpaper (black in color) and a lubricant of mineral oil, mineral spirits or a mixture of the two. The oilier the lubricant, the slower the cutting and the less likely the sandpaper will clog. (I find that sandpaper clogs quickly with a water or soap-and-water lubricant, but you can use one of these also.)

On unfilled, open-pored woods you may need to apply more than three coats so you don't sand through. Because finishes differ in solids content and thus build, and because everyone applies finishes differently, you will need to experiment on scrap wood to learn the number of coats necessary so you don't sand through. Think in terms of four to seven.

A finish has no grain so you don't need to sand with the grain. In fact, you can sand in circles, which I find easier, and you can sand cross-grain near the ends (to keep from sanding through the finish at the edges). Each time you advance to a finer grit sandpaper, change directions (circles, with the grain, across the grain) until you reach your finest grit which should go with the grain. By removing the sanding sludge with naphtha or mineral spirits, you will be able to see clearly when you have removed all the scratches from each previous grit sandpaper (a big advantage over sanding wood). You will see your progress better if you use gloss finish rather than satin.

After sanding a little, scrape off the sludge from parts of the surface using a plastic spreader. If you see shiny troughs or spots, the surface isn't level. When the surface is an even satin sheen overall, it is level and you can move to a higher grit sandpaper to remove the coarser scratches.  Once the surface is level, sand or rub it with finer and finer grit abrasives until you achieve the sheen you want. Begin by sanding up to at least #1000 grit, continuing to back your sandpaper with a flat block or a felt or sponge pad. Then rub with #0000 steel wool, or with pumice and a mineral-oil lubricant using a felt or cloth pad.

If you want a higher gloss, sand up to #2000 grit (P2000 or higher) and then rub with rottenstone and a mineral-oil lubricant using a felt or cloth pad. Or use any other abrasive rubbing compound.