Some scientists think this is a result of the North American extinction event during the Pleistocene. It has been suggested it may have been eaten by the giant ground sloth that went extinct. Or perhaps mammoths or mastodons may have eaten it. Another suspect is the equine that went extinct - a theory reinforced because domestic horses introduced by the European settlers will eat the fruit, hence another name, Horse Apple.
When first found by explorers over 200 years ago, the trees' natural range was very restricted, essentially in the Red River Valley. It has subsequently spread, because the settlers found it is almost indestructible and impervious to disease and insects. The Plains Indians greatly valued it for its strength, and they would travel long distances to the Red River Valley to get the wood, hence the names B'ois D'arc (bow wood) or Osage Orange (from the Indian tribe.) Hedge Apple comes from its planting as fence rows by pioneers.
Now, MY theory - based upon the fact the trees were found along tributaries of the Red River (there is a B'ois D'arc Creek in Northeast Texas), and that is not the area you might find horses or mammoths (but yes, maybe sloths - don't mess up my theory with facts) is that the seeds were still being propagated by an almost extinct mammal...
Google the movie "The Legend of Boggy Creek" and the Fouke Monster to see what I mean.
I know there are a few of these trees here in Mount Pleasant, and this evening I happened to drive past one, so I stopped and loaded a plastic grocery bag of with a half dozen of this... Bigfoot Fruit.
I put some in the fridge, and I have one on my desk. Maybe it's time to write the story.