I know this is beyond the scope of a simple forum post. I'd appreciate any links to previous discussions or resources for issues with this. I started my music journey with typical blues rock electric guitar, and then for the last decade or so I have been focused on bluegrass guitar. In both of those genres I could get away with noodling around in the pentatonic scale of whatever key the song was in. That also technically works for some of the more simple jazz tunes, but it doesn't sound correct.
I do understand music theory and even jazz theory, and I can analyze a solo and tell you what's happening. But as soon as the song starts playing I revert back to spacing out and noodling around on pentatonics for the one chord. At best I can regurgitate a few licks or follow along with a few arpeggios as the chords change.
I am familiar with some of the accepted advice which would be to practice arpeggios all over the neck following the chords. Is that really the best way to proceed, or is there some different exercise? Or is the best practice to do things like transcribe and learn solos - that way you internalize the way the great players have navigated the changes and you internalize it without realizing you are doing it?