Silent Wrong Data: Eight Ways a Research Pipeline Lied Without a Single Visible Error

None of these failures surfaced an error. No alert, no visible exception, no bad log line — just quietly wrong answers delivered with full confidence.

Updated 4 October 2026 with technical corrections.

The most instructive data bug we hit in the past year did not throw an exception. It did not log an error. It returned a dictionary, fully populated, carrying 297,497 synonyms and 473,381 cross-references. The response time was normal. The downstream join ran to completion. We only noticed because a researcher asked why a query about a human cancer cell line was returning results that matched a mouse hybridoma.

A single line of XML parsing code was reading an element as an attribute. The field that served as the join key was blank for all 50,000 rows in the dataset. When the filter ran, it matched everything, and the first row in the table happened to be a mouse hybridoma. The data looked healthily populated. Every call succeeded. The answers were consistently, silently wrong.

That was one incident. There were seven others like it — different mechanism, same pattern: the pipeline completed without complaint, the data it produced was wrong, and the wrongness propagated into answers that the system delivered with full confidence. None of them involved schema drift, null propagation, or stale sources (those are covered in our earlier post on data infrastructure quality for AI readiness). These are a different class of failure: structural corruption that the type system, the exception handler, and the monitoring dashboard all agreed was fine.

Here is a catalogue of all eight, with what happened, why nothing errored, the fix, and the verification that would have caught each one.

Failure 1: Case Normalisation Erased the Only Discriminator

Our ingestion pipeline resolves biological specimen identifiers by name — a common step when joining data from sources that use slightly different spellings or capitalisations. We resolved names with name.strip().upper() before every lookup. Afterwards, 79 names still refused to resolve.

Characterising them revealed the problem: all 79 were pure case-variant collisions. INS-1 and Ins-1 are different organisms — rat insulinoma versus a human line. Li-7 and LI-7 are different species — human hepatocellular carcinoma versus a mouse hybridoma. rMC-1 and RMC-1 are two different rat lines; K562-R and K562-r are two different K562 derivatives — distinct cell lines that differ only in case. Every one of the 79 entries resolves uniquely on an exact-case match. Case was the discriminator, and the normalisation discarded it before the lookup ran.

# Before — normalise up front, silently collapse case variants
normalised = name.strip().upper()
match = registry.get(normalised)          # "INS-1" and "Ins-1" fold to the same key; wrong match is silent

# After — exact-case primary index + separate case-folded index for fuzzy lookup
# Build phase (once, at registry load):
case_index = {}                      # folded key -> list of ALL entries that fold to it
for key, entry in registry.items():
    case_index.setdefault(key.strip().upper(), []).append(entry)

# Lookup phase — most-specific first:
match = registry.get(name) or registry.get(name.strip())
if not match:
    candidates = case_index.get(name.strip().upper(), [])
    if len(candidates) == 1:
        match = candidates[0]    # "hela" -> "HELA" -> [HeLa] -> resolved
    # len > 1: ambiguous collision — surface all candidates, do not pick one silently

The broader lesson: before applying any normalisation, ask what information the transform discards. In biological identifiers, case, hyphens, whitespace, and Greek letters all carry meaning. IFN-α vs IFN-γ, TNF-α vs TNF-β, Ins-1 vs INS-1 — the thing you stripped was the only thing that distinguished them.

Failure 2: Guessing Enum Values Sent 44% of Rows to the Wrong Bucket

When integrating clinical development stage data from a public drug-target database, the pipeline needed to map stage labels to confidence weights. We wrote the mapping from intuition:

# Guessed values — three of six keys absent from a 305-row live sample
STAGE_WEIGHTS = {
    "PHASE_4":    0.95,
    "APPROVED":   0.95,   # actual key is APPROVAL
    "PHASE_3":    0.85,
    "PHASE_2":    0.70,
    "PHASE_1":    0.60,
    "PRECLINICAL": 0.50,
}

The live API, sampled across 305 rows in September 2026, used APPROVAL (not APPROVED), PREAPPROVAL, PHASE_1_2, PHASE_2_3, and UNKNOWN. Three of our six keys did not appear in that sample. (Re-sampled in October 2026, the API also returned PHASE_4 and WITHDRAWAL — enum sets drift, which is exactly what the unrecognised-value log below is for.) The key APPROVAL covered 44% of the sampled rows. Every drug in that bucket fell through to the default — which we set to 0.5, below even PHASE_1's 0.6.

The result: at the time, querying the ranking of drugs targeting HER2 by confidence returned pertuzumab (an approved therapy) at 0.5, below an obscure phase-1 compound at 0.6. Because the confidence score carried a source annotation, the inversion was authoritatively attributed to the data source. The data source was right. The mapping was wrong.

Failure 3: Reading the Producer Code Instead of Capturing the Payload

We built four data extractors for a knowledge graph. One was written against a captured payload — a real API response logged to disk and used as the ground truth for the consumer. The other three were written by reading the producer's source code and inferring the return shape. Two of those three had defects that code inspection had already "confirmed."

The first defective extractor read columns named fdr and pvalue. The real dataset uses adj_p_val and p_value. The column names were not wrong — they were simply never there. Every edge in that extractor landed on the fallback confidence of 0.6 regardless of statistical significance. A gene with adjusted p-value of 1×10⁻¹² and one at 0.49 were recorded as equally confident. Because the knowledge graph's edge merge keeps the maximum confidence seen, 0.6 would have become a permanent floor — unreachable by any downstream update.

The second defective extractor had the right column names but wrong semantics. The source is multi-species, and the extractor keyed nodes on bare gene symbol. A record from a non-human species would silently overwrite a human node's Ensembl ID, because the property merge was an unconditional set rather than a conditional update.

The trap with code inspection is that it shows you the keys on the path you happened to read. It does not tell you which of nine possible return shapes your extractor will receive, what error returns look like, or whether the identifiers in the payload join to anything in your graph.

Failure 4: The Hand-Curated Lookup Table Was Data, Not Code

A 14-row alias table mapping informal gene names to canonical HGNC symbols was written from memory. It felt too small to audit carefully — 14 rows, straightforward mappings. The table contained one entry: BCL2L → BCL2. This is wrong. BCL2L is an HGNC alias for BCL2L1 (Bcl-xL), a different gene from BCL2. Bcl-xL and Bcl-2 are related anti-apoptotic proteins but they have different tissue expression, different inhibitor profiles (venetoclax is Bcl-2-selective and spares Bcl-xL; navitoclax hits both), and different toxicological implications.

A query about Bcl-xL activity would have been answered with Bcl-2 evidence, and the response would have annotated the canonical symbol as BCL2 to justify it. Two other rows in the same table were ambiguous rather than wrong: NEU is also a previous symbol of NEU1, and PD1 is also an HGNC alias for SNCA (and SPATA2). These were not caught because nobody queried the registry.

# Before — written from memory, unvalidated
GENE_ALIASES = {
    "BCL2L":    "BCL2",    # WRONG — BCL2L is an alias for BCL2L1, not BCL2
    "NEU":      "ERBB2",   # AMBIGUOUS — NEU also resolves to NEU1
    "PD1":      "PDCD1",   # AMBIGUOUS — PD1 is also an HGNC alias for SNCA (and SPATA2)
}

# After — each row validated against the HGNC registry
# GET https://rest.genenames.org/search/BCL2L (Accept: application/json)
# → alias_symbol for BCL2L1, not BCL2
GENE_ALIASES = {
    "BCL2L":    "BCL2L1",  # confirmed via HGNC
    ...
}

Failure 5: A Blank Join Key Returns Everything, Not Nothing

This is the one from the opening paragraphs. Our ingest of a public biological specimen database's XML export produced a 50,000-row table. Each entry has an accession identifier that serves as the primary key for all downstream joins. Our parser read that identifier with:

# Wrong — reads an XML attribute that does not exist at this element
accession = elem.get("accession", "")   # "" for every row

In the actual XML schema, the accession is a child element, not an attribute on the root cell-line element:

<!-- Actual XML structure (simplified) -->
<cell-line category="Hybridoma" ...>       <!-- category IS an attribute -->
  <accession-list>
    <accession type="primary">CVCL_XXXX</accession>
  </accession-list>
</cell-line>

# Correct — reads the child element, mirrors how adjacent fields were parsed
acc = elem.find("accession-list/accession[@type='primary']")
accession = acc.text.strip() if acc is not None and acc.text else ""

Because elem.get("accession", "") on an element with no such attribute returns an empty string rather than raising an error, every one of 50,000 rows got a blank accession. The assembly function that filters rows by accession value then matched everything — because every row matched the empty-string key. The first row in the table was a mouse hybridoma entry. Every query returned that row, with the whole-table counts of 297,497 synonyms and 473,381 cross-references attached as though they belonged to it.

Failure 6: The Counter Was Measuring Write Events, Not Entities

A knowledge graph service exposed a statistics endpoint that reported node and edge counts by entity type. The gene counter stored on a live table read 2,015. The actual number of distinct gene nodes on that table was 435. The total node counter read 3,137; the actual was 784 — a 4× inflation.

The counter incremented on every upsert, not on every creation. A single research query re-upserts the same gene node from multiple sources as it processes results. That gene gets counted each time. The more the system was used, the more wrong the number got — the exact opposite of how people intuitively expect a metric to behave.

There was a second problem layered underneath. The code that read the counter had a subtle serialisation bug: it was passing typed DynamoDB values ({"S": …}) to table.meta.client, which on a boto3 resource Table still applies the resource's serialiser — so they were serialised twice and caused a ValidationException. That exception was caught by a broad except block that logged at DEBUG level and returned a fallback value computed as int(table.item_count * 0.6). The write-event counter had accumulated 3,137 against 784 actual nodes (2,015 against 435 gene nodes), but the read path never reached it — the serialisation failure caused every get_stats() call to fall back to int(table.item_count * 0.6). The number reaching users and the agent's tool calls was a fabricated 60/40 split of raw item count; the already-inflated write-event counter was not even being read.

# Before — increments on every write regardless of whether the item is new
def upsert_node(entity_type, entity_id, props):
    table.put_item(Item={"pk": entity_id, **props})
    counters.increment(entity_type)     # fires on update AND creation

# After — increment only when the item did not previously exist
def upsert_node(entity_type, entity_id, props):
    response = table.update_item(
        Key={"pk": entity_id},
        # keep the GSI key attributes top-level so search still finds the node (Failure 7)
        UpdateExpression="SET entity_type = :t, entity_id_lc = :lc, #d = :d",
        ExpressionAttributeNames={"#d": "data"},
        ExpressionAttributeValues={":t": entity_type, ":lc": entity_id.lower(), ":d": props},
        ReturnValues="ALL_OLD"
    )
    if not response.get("Attributes"):  # no item existed before this write → new creation
        counters.increment(entity_type)

Failure 7: An Entity ID Without the Type Prefix Is Permanently Invisible

A knowledge graph uses a DynamoDB composite GSI where the sort key index is constructed as {entity_type}:{search_term}. This means every search query is structured as a begins_with filter over that composite pattern. A node whose entity ID does not start with its entity type is written successfully, passes the provenance gate, appears in scans — and can never be returned by any search.

While writing new extractors, we used what felt like natural sub-kind prefixes as the entity ID prefix. For entities of type evidence_item, we wrote IDs like study:record_001 and result:record_002, because "study" and "result" felt like more descriptive identifiers. We caught it before the extractor shipped. Had it shipped, the node would have been stored with no error raised — and every subsequent search for evidence items would have run:

# Search function — always prefixes the query with entity_type
Key("entity_type").eq("evidence_item") &
    Key("entity_id_lc").begins_with("evidence_item:")

# A node stored as  pk = "study:record_001"  with entity_type = "evidence_item"
# will NEVER be returned — the begins_with condition fails silently

# Wrong — sub-kind used as prefix
entity_id = f"study:{record_id}"          # entity_type is "evidence_item"
entity_id = f"result:{record_id}"         # entity_type is "evidence_item"

# Correct — entity_type is always the prefix, sub-kind follows the colon
entity_id = f"evidence_item:study_{record_id}"
entity_id = f"evidence_item:result_{record_id}"

The permanence matters here. The graph had no delete route and TTL was disabled. A node written with the wrong ID prefix would not have been just temporarily invisible — it would have been unreachable for the lifetime of the data store, and fixing the extractor code afterwards would not have repaired the stored nodes.

Failure 8: Null Values Silently Excluded Records From Results

A dose-response data service returned an empty result set for a compound that was confirmed present in the underlying data. The underlying database had one record for that compound. The service said it found a record but could not retrieve it. The researcher reported a data gap.

The service filtered results before returning them:

# Before — silently excludes records where curve fit did not converge
valid_records = [r for r in results if r.get("potency_metric") is not None]
return valid_records

Records where the curve-fit algorithm did not converge carry a null for the potency metric field. Those records are real data — a null potency metric means "curve did not converge," not "compound was not tested." The filter was removing them without logging or signalling the exclusion. The caller received an empty list, saw that the total match count was 1, and could not reconcile the two facts.

# After — include all records; surface the exclusion explicitly
all_records = results
converged = [r for r in results if r.get("potency_metric") is not None]

if len(converged) < len(all_records):
    logger.info(
        "%d of %d records have null potency_metric (curve did not converge); returning all",
        len(all_records) - len(converged),
        len(all_records)
    )

# Return all records; let the caller decide how to handle non-converged entries
return all_records

Patterns Across All Eight

The common thread across all eight is that the failure mode was designed in, not accidental. Case normalisation is a useful operation in many contexts — it just should not be unconditional. Enum defaults are useful — they just must not shadow a known category. A counter is useful — it just must measure what its name says it measures. In each case the error was not in the operation itself but in the assumption that the operation was safe to apply without first verifying what it would throw away.

The verification step in every case is small — a single test, a one-time API query, an assertion at write time. The cost of skipping it is paid later, quietly, by users who get wrong answers that the system delivered with full confidence.

Disclaimer: The views and opinions expressed in this post are my own and do not represent those of my employer or any organisation I am associated with.