How can i trust a language model to extract chemical data from lab notes?

[SofiaB]

I’ve been trying to use a large language model to help parse and categorize decades of unstructured lab notes in my field, but I’m hitting a wall with its tendency to confidently generate plausible but incorrect chemical properties. It’s making me question how we can ever trust this kind of automated literature mining for actual hypothesis generation without introducing subtle, persistent errors into the foundation of a research project.

[AbigailS]

Yeah I’ve been there. We started with a big batch of notes and a model that seemed helpful until it started asserting that a compound had a melting point of 250 C and it was clearly a misread. We built a validation layer that flags any chemical properties not supported by at least two sources or not in our catalog, and we force a human check before any table goes to the hypothesis stage. That helped, but it slowed things down and we still had some stubborn wrong labels creeping in. The confidence scores helped a little in keeping expectations in check.

[Eric_J]

I tried to rely on retrieval augmented generation and a set of rule checks. It saved time on rough categorization, but we found the model would still latch onto a plausible property and keep pushing it even after we corrected the source. The mistake persisted in the archive because the notes had ambiguous shorthand and inconsistent units. We ended up decoupling the model from the final property extraction and used it only for coarse topic tagging, while keeping a separate database of verified facts.

[Gary_M]

One concrete moment: we mislabeled a solvent's polarity due to a synonym in the notes; the model treated 'polar' and 'polar aprotic' as the same. Took weeks to chase that leak. We paused the automated pass and implemented a glossary and cross-reference plan with a chemist, but it wasn't fully solved. I don't feel confident it's the real problem: maybe the notes themselves are noisy or the problem is that we expect a single pass to clean everything.

[NathanIH]

I keep thinking maybe the root issue isn't the model at all but the data it sees. The notes span decades and mix lab slang with old shorthand, and if we don't standardize that first the model will fill gaps with plausible stories. We started a small pilot with a clean subset and measured how often the model hallucinated, but the numbers were not encouraging. Do you think the data quality problem is the real bottleneck, or is there another hidden trap in automated literature mining?