Automated Data Management Using AWS Serverless Framework

National Consumer Panel Automates Data Management using a Serverless Framework on AWS

IM电竞网站 helped the National Consumer Panel (NCP) automate the management of consumer panel data by designing and developing a serverless framework using  Amazon Web Services .


The  National Consumer Panel  (NCP) is a joint venture between Nielsen and IRi – the two leading consumer insight providers in the United States. They are a market research panel that focuses on measuring consumer attitudes and behavior. Since 1987, NCP has been gathering panel member’s opinions and shopping data to help manufacturers and retailers make informed decisions on what products to develop, improve or sell in the market.


NCP came to WCI with the challenge of handling the processing, validation and auditing of consumer panel data, feedback, receipts and reward payment.  Each consumer panel member is invited to participate in special programs and surveys and in return gain rewards.  The goal of the WCI solution was to simplify and automate the entire process.


The first step in the process was selecting Amazon Web Services (AWS) which allowed NCP to quickly scale to demand without expensive hardware purchases. Utilizing a serverless architecture & native AWS services proved to be the most efficient way of preparing for a workload that could be highly unpredictable.

Architecture Creation & Validation

  • WCI started by building API Gateway endpoint to receive receipt requests captured in the existing NCPMobile app by panelists.
  • With the use of S3, Lambda and SQS, we created tasks for Turk workers responsible for transcribing panelist receipts.
  • Build Lambda function triggered by API gateway endpoint to create HIT tasks for transcribing receipts.
  • For each received receipt create Mechanical Turk HIT for which input screen was built and manually tested. In order to audit the process, Lambda was used to suspend the autopay service
  • For each step of the process a persisted record was created and updated with the use of DynamoDB.
  • Data is streamed from the DynamoDB tables to an Aurora instance using Kinesis Streams.
  • The data from the Aurora instance is used to populate a Quicksight dashboard for analysis on the number of transcriptions being done and the number of unique workers working on those tickets.

The engagement with NCP required utilizing a broad array of services on AWS, including:

Amazon Elastic Compute Cloud (Amazon EC2)Amazon Relational Database Service (Amazon RDS), Amazon Glue, Amazon Athena, Kinesis Streams, Amazon CloudWatch, Amazon Simple Queue Service (Amazon SQS), AWS Lambda, Amazon Virtual Private Cloud Peering (VPC-Peering), Amazon QuickSight, AWS Identity and Access Management (IAM) and Amazon Simple Storage Service (Amazon S3).


NCP was able to automate the management of consumer panel data using a serverless framework, which enabled them to save reduce labor costs and avoid significant investments in hardware.

Not sure how to begin your journey adopting AWS? Let WCI be your guide.

Explore AWS Solutions

About IM电竞网站

WCI is a consulting services company focused on the application of data through the use of  Business Intelligence  and  Data Warehouse  technologies that bring increased business performance for our clients. WCI has serviced well over 300 companies through our knowledge and expertise of how to make data valuable to decision-makers.

Get Started

Want a  FREE  on-site discovery session with your team?  Receive a  FREE data management evaluation session with one of our veteran data architects. Reap the benefits of an expert’s outlook on taking control of your business insights and data. Stop missing out on data-driven opportunities,  and start making smarter, more profitable decisions today.

友情链: 新IM电竞平台首页 im电竞直播大厅-新出的电竞竞猜 | im电竞官网 _ im电竞在线官网首页-电竞体育平台排名前十 | IM电竞平台注册 - im电竞官方app下载 电竞体育官网首页 | IM电竞登录-im电竞滚盘官网_电竞体育真人竞猜注册送 | im电竞永久官网-im电竞比分网官网登录|电竞体育平台送彩金 | im电竞比分网~亚博IM电竞足球~电竞竞猜贴吧最新 | im电竞下注 IM电竞专业电竞平台-电竞竞猜推荐平台 |