Amazon Bedrock tokenomics data lake: Running Amazon Bedrock across many

Amazon Web Services launched an open-source reference architecture and deployment toolkit designed to aggregate Amazon Bedrock invocation logs across multiple linked accounts into a single, queryable data lake. The pattern, detailed in AWS Machine Learning Blog's report, addresses a major blind spot for engineering teams operating large model fleets where aggregate monthly cloud invoices fail to expose per-call prompt contents, token efficiency, or invocation failures across individual departments.

This guide examines the five-stage pipeline architecture, the required AWS CloudFormation templates, and the SQL join strategies that tie raw invocation telemetry directly to granular financial metrics. Platform teams, machine learning engineers, and cloud architects running multi-account AWS environments can use these concrete implementation steps to track model utilization, audit user inputs, and isolate inefficient prompts across their organizations.

01Centralized log ingestion and transformation pipeline

The observability data lake moves each model invocation record across a coordinated pipeline spanning source accounts and a designated central account. When an application initiates a call to an Amazon Bedrock model, native account-level invocation logging captures the model identifier, the caller AWS Identity and Access Management identity, input and output token counts, and the raw prompt and response strings. A local Amazon CloudWatch subscription filter immediately forwards these structured events directly to a CloudWatch Logs destination located in the central account without granting the source account direct permissions on the primary storage pipeline.

Upon arrival in the central account, a dedicated AWS Lambda function decompresses and decodes the incoming CloudWatch log batches while discarding non-invocation operational messages such as setup confirmation notices. The function unpacks the payload, extracts the nested caller identity to the top level of the JSON document, and forwards the cleaned events downstream. Amazon Data Firehose buffers these records, converts the raw JSON payloads into columnar Apache Parquet files, and writes them into an Amazon Simple Storage Service bucket partitioned under structured date prefixes.

02Queryable telemetry and invocation metadata capture

Exposing the aggregated telemetry through an AWS Glue Data Catalog table allows teams to query the data lake with standard SQL in Amazon Athena. Flattening the raw event structures gives operators visibility into the precise prompt text submitted by an application, the generated response, and the exact input and output token volumes used during execution. The cataloged schema also retains metadata detailing the specific model selected, the calling account identifier, the invocation timestamp, and the acting IAM role.

This level of granular capture also tracks the model stop reason, surfacing critical runtime indicators like max_tokens cutoffs when responses terminate prematurely due to length constraints. Instead of relying exclusively on high-level usage totals, platform administrators can audit model distribution across separate engineering teams and pinpoint oversized prompts that consume unnecessary resources. The resulting Athena views present token counts, cache token metrics, stop conditions, and text previews as queryable columns in a dedicated workgroup.

03Staged deployment across linked cloud accounts

Deploying the architecture requires the AWS Command Line Interface configured with target account credentials, CloudFormation capabilities for named IAM roles, and a local Python 3 environment equipped with the Boto3 library for verification testing. The roll-out sequence follows a strict dependency order: operators must deploy the central account CloudFormation stack first to establish the central receiving pipeline. This primary stack outputs the Amazon CloudWatch Logs destination Amazon Resource Name, which source accounts require, alongside an AWS QuickSight role ARN for future dashboard integration.

Once the central pipeline exists, administrators deploy the secondary CloudFormation template into every linked account running Amazon Bedrock workloads, passing the central destination ARN as an input parameter. Because model invocation logging operates as an account-level setting rather than a native CloudFormation resource, teams must activate Bedrock logging manually once per account through an API call, directing output strictly to the provisioned CloudWatch log group. Operators then invoke a test prompt, wait approximately one minute for Amazon Data Firehose to flush its write buffer, confirm Parquet objects in the S3 bucket, and execute the pre-saved Athena query to instantiate the flattened view.

04Financial attribution using cost data joins

Organizations can connect the technical execution telemetry directly to detailed billing records by joining the flattened Athena dataset with AWS Cost and Usage Report 2.0 tables. Using the IAM role ARN as the common relational key, analysts can reconcile real-time token counts against actual daily cloud spend. This query structure maps prompt overhead directly to distinct business units, individual accounts, and specific software components operating within the organization.

The financial join transforms operational monitoring into true cost attribution, revealing the exact dollar amounts incurred by specific application prompts and identifying the costliest interactions across the enterprise. Platform managers can isolate usage spikes, calculate day-over-day expense trends by team, and evaluate whether expensive frontier models are being invoked where smaller, cost-effective models would suffice. The sample SQL queries included in the accompanying GitHub repository provide the baseline logic needed to run these combined attribution reports.

05Fleet visibility for multi-account engineering teams

This centralized data lake design specifically resolves the governance challenges faced by organizations operating decentralized Amazon Bedrock workloads across fragmented cloud accounts. When teams manage separate projects in isolation, account-level CloudWatch and S3 sinks keep log data trapped inside functional silos, preventing cross-organization auditing or standard security reviews. Consolidating the stream into a single repository ensures compliance officers and infrastructure leads maintain a permanent record of all inputs, outputs, and system interactions.

With universal prompt auditing in place, engineering leaders can enforce token efficiency guidelines and detect behavioral anomalies before billing cycles close. The architecture eliminates cross-account IAM credential distribution by decoupling ingestion filters from storage buckets through the native CloudWatch destination interface. As generative workloads expand across multiple engineering branches, the consolidated lake establishes a foundation for long-term prompt performance analysis, financial forecasting, and fleet-wide model optimization.