Could also be a bent or worn shift fork or worn fork pin roller/drum slot.

Sounds like I need to open it up further to diagnose. That will be my next step and i'll report back with what I find

I had a somewhat similar issue with my CT2. I would shift into 2nd gear and begin accelerating and all of a sudden I would drop back into neutral. At first I figured that I just wasn't getting a positive shift (didn't shift up hard enough) but after hearing some serious gear grinding one time I just always skipped second gear cause I didn't want to mash up the transmission to all hell. After splitting the case, I found that the shift fork was just really worn down.  

Best of luck with the bike! 

From diagrams looks like there's three shirt forks do you remember which one it was?

Is there any measurements to compare to or can you just tell by looking at them?

Second gear is made up of #17 dogs engaging with #14 in link below.
www.partzilla.com/catalog/yamaha/motorcy...972/ct2/transmission
So it's for #3 in below link.
www.partzilla.com/catalog/yamaha/motorcy...2/ct2/shift-cam-fork
But as alnarv said in post #11[& i didn't think about], rollers on the pins & the drum slots.
I've broken those rollers on a 250/360 & in later years the did away with them.
I've had the high point of the drum track get worn or chipped away on the bigger motors & its the high point where it stops in gear.
In saying 'that, second gear would be the last i'd expect trouble with.
Any wear on the Fork is always caused by something else in the begining, like rounded dogs pushing on selector to try & escape or other things mentioned.

I agree with RT325, the dogs and the forks work together. If the fork doesn't fully engage the dogs, they slip past each other under load (like when accelerating), rounding off the edges. That puts greater pressure on the fork which can bend or wear badly. As the dogs round off, they will slip apart more and more easily. (a vicious circle) Normally, something else has to start that ball rolling.

Later bikes have a gear indexing adjustment, and if yours has the same thing and its not properly adjusted, could be causing your problems.

Have to concur with RT.  Engagement dogs worn or something moving around when it should not.  Engagement dogs would be my first choice, followed by bent or bady worn selector forks.

Drum can't move to the right if the washer & retainer is under the plastic cap.
If it 'could' move right it reduces 2nd depth of engagement.
I'm sure you said all the bits are under the plastic cover.

Yes all of the bits are under the plastic cover.

I have the clutch side cover off right now and I adjusted the shift shaft lever that engages the dowel pins. Not sure technical terms but the spacing on the dowels and shift shaft arm was pretty far off.

Was also missing the bottom washer for the flat bearing that's behind the clutch boss and between the basket. Not sure if that has any affect really but getting that replaced. The one under #2 in the diagram

www.partzilla.com/catalog/yamaha/motorcycle/1972/ct2/clutch