Multi-Tenant FinOps: The Three Silent Bugs That Made Our AWS Cost Data Untrustworthy

We built the identity join, ran it against real data, and got 1.8% coverage. It looked like a directory sync problem. It was a .lower() call — and two more silent bugs followed.

Updated 4 October 2026 with technical corrections.

The Join That Came Back Nearly Empty

We built a multi-tenant cost dashboard to track AWS spend across research teams. The design was straightforward: take per-user usage records, join them to Azure Active Directory group membership to find each user's team, then attribute costs accordingly. In development, with synthetic test data, coverage was excellent. Totals matched. We deployed to production.

We ran the join for the first time against real production data. Coverage: 1.8%.

It looked exactly like a data problem — an incomplete directory sync, a stale mapping table, a missing field. We spent two hours checking the source data. Everything looked fine. Then we compared a single email address from the usage records against the same person's record in the AD membership table.

Usage records: firstname.lastname@company.com
AD table: FirstName.LastName@Company.com

It was a .lower() call. The join was case-sensitive. That was Bug 1. The other two bugs were waiting quietly behind it.

Three bugs. All silent. All there from day one. None of them affected the cost totals — only the attribution.

The Multi-Tenant Attribution Problem

Before diving into the bugs, it's worth articulating what we were actually trying to do, because "multi-tenant cost attribution" sounds straightforward until you build it.

AWS gives you cost allocation tags (you tag resources with owner/team/project), account-level cost aggregation and, for some shared services, native splitting — EKS split cost allocation data apportions node cost to pods, and Bedrock can attribute spend to the calling IAM principal. For a simple setup — one team per account, or one team per tag — this is sufficient. For shared infrastructure it isn't, even with the native splits: they hand you cost per pod or per caller, and mapping that to teams is still your job.

On shared EKS clusters, a single node might serve three teams simultaneously. The same S3 bucket hosts data for several research groups. A Bedrock model serves queries from every team in the organization. You can't tag a node as belonging to one team. You need to allocate the cost across teams based on usage signals — and that allocation introduces fan-out: one dollar in must equal exactly one dollar distributed out.

The identity problem is separate: you need to connect "who used this resource" to "which team do they belong to." That requires an identity join — matching usage records against your directory service (in our case, Azure Active Directory groups via a sync service). That join is where Bug 1 lived.

Bug 1: The Mail Casing Collapse

The join logic was conceptually simple: take the email address from the per-user usage record, look it up in the AD group membership table, find the team.

In development, this worked perfectly. We'd seeded our test data with lowercase emails throughout. Nearly every join hit — 96% coverage felt great.

In production, a directory sync ran and updated the AD group membership table from real Entra data. Entra's mail field is mixed case. Some users have firstname.lastname@company.com, others have FirstName.LastName@Company.com, and the same domain appears with three different capitalizations depending on when the account was created and whether it had been migrated.

In our environment, the email addresses in the usage records were consistently lowercase — but nothing guarantees that, so normalise both sides rather than relying on the source being consistent. Our join was case-sensitive. The result: 1.8% of users matched, because only a tiny fraction of Entra accounts happened to have lowercase mail fields that aligned with the lowercase values in our usage records.

The fix is two lines:

# Before: case-sensitive join
df = costs.merge(groups, left_on='user_email', right_on='mail')

# After: normalize both sides
costs['email_lower'] = costs['user_email'].str.lower()
groups['mail_lower'] = groups['mail'].str.lower()
df = costs.merge(groups, left_on='email_lower', right_on='mail_lower')

We also added a fallback: if mail doesn't match after normalization, try userPrincipalName. Some accounts have a valid UPN even when the mail field is inconsistent. That coalesce brought join coverage to 94.9%. Of all group members, 61% had activity in the last 30 days — the remaining 39% are real people who simply had no activity in the selected window, not missing accounts or service accounts.

The deeper lesson: never join on identity fields without normalizing case on both sides, and never test with synthetic data that doesn't reflect your real directory's encoding quirks. A test with 50 carefully-crafted synthetic users will not find a bug that only exists in real Entra data with three domain spellings.

Bug 2: Row Ratio ≠ Value Ratio in Membership Fan-Out

The second bug involved a different kind of attribution failure. Many of our researchers belong to multiple AD groups — a platform team, a project team, and a department group simultaneously. Our attribution logic credited their full spend to each group they belonged to.

The consequence: one user in three groups had their usage counted three times. The sum of all per-group credits exceeded the actual total spend. We had created spend out of nothing.

We measured it: membership rows outnumbered unique users by 7.9%. But because multi-group members tend to be heavier users (they're on more projects, they have broader access), the credit inflation was 12.6% over 30 days, 13.9% over 90 days. Row inflation 7.9%; credit inflation 13.9% — the gap is the signature of above-average users being the ones with multiple memberships.

We caught this with a conservation assertion added after the fan-out step:

def attribute_costs(costs_df: pd.DataFrame, membership_df: pd.DataFrame) -> tuple[pd.Series, float]:
    """Attribute user costs to their AD groups (deduplicated share per group)."""
    membership_df = membership_df.drop_duplicates(['user_id', 'group_id'])
    input_total = costs_df['amount'].sum()

    # Fan-out: join costs to membership — each user's spend appears in each of their groups
    fanned = costs_df.merge(membership_df, on='user_id', how='left')

    # Deduplicate: each user contributes a proportional share to each group they belong to
    # (a user in 3 groups contributes 1/3 of their spend to each, not their full spend;
    # a user in no group keeps their full spend for the unattributed bucket)
    n_groups = fanned['user_id'].map(membership_df.groupby('user_id')['group_id'].nunique()).fillna(1)
    fanned['user_share'] = fanned['amount'] / n_groups
    # Orphan costs: users with no group mapping get an explicit unattributed bucket, not dropped
    unattributed = fanned[fanned['group_id'].isna()]['user_share'].sum()
    deduped = fanned.dropna(subset=['group_id']).groupby('group_id')['user_share'].sum()

    # Conservation invariant: deduplicated total plus unattributed must match input
    output_total = deduped.sum() + unattributed
    delta = abs(output_total - input_total)
    epsilon = input_total * 0.0001  # 0.01% tolerance for floating-point drift
    if delta > epsilon:
        raise ValueError(
            f"Cost conservation violated: input={input_total:.2f}, "
            f"output={output_total:.2f}, delta={delta:.4f}"
        )

    return deduped, unattributed

The fix: compute each group's share as a per-user proportion — a user in three groups contributes one-third of their spend to each group, not a full copy. Orphan costs (users with no group mapping) are preserved in a separate unattributed bucket rather than silently dropped. With the per-user share approach, the sum of deduped plus unattributed equals the original input total.

Bug 3: CloudWatch Period Alignment

The third bug was subtler and, if anything, more insidious because it didn't affect totals — it affected time series.

We were querying CloudWatch for daily metrics using Period=86400 (one day in seconds). This seemed correct: we want daily granularity, one day is 86,400 seconds, done.

The problem — observed behavior, not explicitly documented in the API reference — is that Period=86400 doesn't align to UTC midnight. It aligns to the rounded StartTime of your query. If you query starting at 14:37 UTC, your "daily" buckets are 14:37 → 14:37 the next day. Change your start time and the buckets shift.

In practice this means that if your start time is derived from "now", running the same query at 09:00 and again at 17:00 gives "Monday" a different 24 hours each time: the same day shows different numbers depending on when you ran it, and events from two calendar days can share one bucket.

# Broken: Period aligns to StartTime, not midnight
response = cloudwatch.get_metric_data(
    MetricDataQueries=[{
        ...,
        'MetricStat': {
            'Period': 86400,  # one "day"
            ...
        }
    }],
    StartTime=datetime.now(timezone.utc) - timedelta(days=7),  # e.g. 14:37 UTC
    EndTime=datetime.now(timezone.utc),
    ScanBy='TimestampAscending'
)
# Each datapoint covers StartTime + n*86400, not midnight boundaries

# Fixed: use fine-grained periods, aggregate to calendar days in application code
response = cloudwatch.get_metric_data(
    MetricDataQueries=[{
        ...,
        'MetricStat': {
            'Period': 300,  # 5-minute granularity
            ...
        }
    }],
    StartTime=datetime(2026, 8, 24, 0, 0, 0),  # Explicit midnight UTC
    EndTime=datetime(2026, 8, 31, 0, 0, 0),
    ScanBy='TimestampAscending'
)

# Aggregate to calendar days ourselves
df['date'] = pd.to_datetime(df['timestamp']).dt.floor('D')
daily = df.groupby(['team', 'date'])['cost'].sum().reset_index()

The fix: use fine-grained periods (5 minutes) and aggregate to calendar days in application code, always starting at explicit UTC midnight. The CloudWatch period alignment issue disappears when you own the aggregation.

Building the AD Group Fan-Out That Actually Works

After fixing all three bugs, the picture changed significantly. Here's what the working system looks like:

We sync AD group membership from Entra on a scheduled basis. The sync normalizes mail to lowercase and coalesces to UPN as fallback. The cost attribution pipeline:

1. Fetches per-user usage records, normalised to lowercase email
2. Joins on normalised email → group membership
3. Allocates shared resource costs by dollar weight, not row count
4. Asserts conservation at every allocation step
5. Aggregates to calendar days using application-level bucketing
6. Explicitly preserves "unattributed" as a first-class bucket — never silently drops

Join coverage after fixes: 94.9% — up from 1.8%. Separately, of all group members, 61% had activity in the last 30 days; the remaining 39% are real people with no activity in the selected window, not missing accounts or service accounts. Both metrics matter: the join coverage tells you how complete your mapping is; the activity rate tells you how much of the directory population is actually represented in the selected period. Infrastructure without owner tags appears explicitly as "unattributed" rather than being spread across teams or dropped.

What Makes Multi-Tenant Cost Data Trustworthy

Looking back at the three bugs, they share a pattern: each one corrupted the data silently, produced plausible-looking output, and only became visible when someone checked the output against an independent number.

The system we ended up with has four properties that prevent this class of failure:

Conservation assertions at every split. Any operation that redistributes a cost pool raises an error if the total deviates beyond floating-point tolerance. Use if delta > epsilon: raise ValueError(...) rather than assert — Python strips assertions under -O, which many container images enable. It catches both the fan-out bug and any future allocation logic that violates the conservation law.

Identity joins with normalized fields. Every join on email, username, or any identity field normalizes case on both sides before comparison. The normalization is applied at both ingestion (so new data is stored clean) and at join time (so older data already on disk is handled correctly — a write-time-only fix leaves existing records wrong).

Metric buckets owned by the application. We don't rely on cloud-native period alignment for time series that need to align to calendar boundaries. We query at fine granularity and bucket ourselves using UTC midnight as the day boundary, everywhere, consistently.

Unattributed is a real bucket. We never silently drop costs that can't be attributed. Unattributed costs are explicitly surfaced in the dashboard. If unattributed jumps, that's a signal — not a rounding error.

There's one more thing worth adding: store owner identity as email, not display name. We initially stored team owners as display names. LDAP resolution takes results[0] — and names are not unique. We had two "John Smith" entries in our directory. Every time we looked up the owner of a particular cost center by name, we might get the wrong person. Email is unique. Use email as the identity key; map to display name only at render time.

Disclaimer: The views and opinions expressed in this post are my own and do not represent those of my employer or any organisation I am associated with.