When to Harvest Gingko Leaves

I am presently in Korea. There are thousands of Gingko Biloba trees around. I'm wondering if I can pick the leaves for my own use? I've heard that the leaves have a toxin in them which has to be removed. Is this true?

We have not heard anything about toxins being present in ginkgo leaves. Christopher Hobbs, in his book, "Ginkgo, Elixir of Youth", says that ginkgo leaves of have history of safe use going back thousands of years. When you use whole leaves within the normal dose range, Hobbs says that there is "no danger of negative side effects." However the highly purified extracts can cause minor gastric upset in a small percentage of individuals.
The fruits can cause skin rashes. The fresh pulp contains chemicals similar to those found in poison ivy and poison oak. In China the fruits are processed to remove the irritating pulp.

I also read that the leaves in spring (a light green) and in fall (yellow) contain different active ingredients, so it's good to harvest both for use. Maybe this is to do with the percentage of active ingredients that is recomended in the extract?

Hobbs has this to say about harvesting ginkgo leaves: the best time to harvest is in the fall just as the leaves begin to turn yellow. This is a trade off between the best time for flavanoid content (after the leaves turn colour) and the best time for ginkgolides and bilobalide (in late summer just before the leaves turn yellow).

I'm in a small town, so the trees seem to be relatively clean!
What would you advise?