Okay, I rode the bike today after adjusting the tube height evenly and that didn't make a difference.  Whoever said it wasn't the problem is correct. But step one is down and now onto step two.  I'll strip the  headset down and check the bearings.  Should be able to get that done and ride it by the weekend. 

Yep, that a good next step & hope you find a reason. Look forward to next post.

You can have new swingarm bushings and still have a problem if it's not shimmed right.  If you can get the rear wheel in the air, try to pull wheel side to side. 

It doesn't take much of a loose swingarm to make a bike squirrely.  Also make sure your rear wheel axle is good and tight!  With a wrench.  I've had the 360's and 400's move the rear wheel around with the axle only slightly loose.

Here's what I discovered.  When I installed the upper bearing race, #2,  in the lower section of the frame neck, I picked up some powder coat material that compressed between the bearing race and the bearing shoulder.  Hope that makes sense.  The bearing race was crooked even though it was tight.  I disassembled everything, cleaned and reinstalled. It feels much better. I haven't ridden it yet but I confident this is the issue.
 

Curious if you installed races following part numbers on parts diagram?

I just checked's and it has no side to side movement.  I torque almost everything so I know the axle's okay. Hopefully the issue is resolved with the front fork bearings .  I've worked on restoring this bike off and on for 3 years and with so much time lapsed  I start questioning everything I've done when there's a problem. 

Sounds promising. Hope same hasn't happened to the lower race. Both races might be accidentally interchangeable too--not sure,-- except top one is shallow & lower is deep. Different ball bearing size of course.

Sorry, my bad, reading again. Upper race in lower section--understood sorry.

Constant issue with frames being powder coated is threads, apertures and earthing points not being masked off. Someone here in the UK has had a near identical problem, but with the swing arm bushes.

The races are definitely installed in the correct order per the diagram. My only problem was that I didn't clean the powder coat from the raceway before installing the race.  When I disassembled the steering head, I had 2 immediate clues. One was where the race had peeled the coat back to bare metal when installed, and the second was my new fresh grease looked old and black because it was mixed with crushed powder coat.. Without any cables or wires attached to the steering forks the difference was obvious.