Are You Unlocking the Power of Your Data? Enhance your Analytics with Mactores Aedeon Data Lake

Mactores Aedeon Data Lake Reference Architecture

• Data Sources: Aedeon Data Lake supports a variety of data sources and integration with third-party data providers, internal systems such as SAP, Oracle EBS, and Salesforce, or file uploads through an API or workflow job. The sources are managed by independent data producers with the necessary Information and Access Management (IAM). The location and source of data are abstracted from the data consumer, which makes it easy to integrate, modify and manage multiple data sources.
• Ingestion: Amazon Glue, Amazon Kinesis, and Amazon Managed Streaming for Apache Kafka (MSK) serverless services are used to batch and real-time data at scale. The cataloged data is immediately available for discovery, cleansing, modeling, enrichment, obfuscation, or masking.  Aedeon Ingestion platform supports real-time data analytics, which accelerates the value of your data to make real-time decisions.
• Data Manipulation and Storage: Aedeon Data Lake offers a storage platform backed by Amazon S3. Organizations can choose between storage tiers depending on the storage volume, cost, processing performance, business value, and applicable compliance regulations. Instead of manually configuring and managing server configurations, users can take advantage of the Aedeon integrations service for automated resource provisioning, scaling, and serverless infrastructure operations. Aedeon supports tight integration with LDAP/Active Directory and Amazon Key Management service to ensure that data storage complies with all organizational security policies for information access and management.
• Platform Operations: Aedeon Data lake platform runs complex operations in the cloud and requires organizations to take the necessary governance and security measures. Aedeon Data Lake follows the five pillars of the AWS Well-Architected Framework: operational excellence, high security, consistent reliability, maximum performance, and continuous cost optimization. These goals are achieved by integrating Amazon infrastructure management services, including Amazon CloudWatch, CloudTrail, and Managed Apache Airflow. Aedeon enables customers to automate sensitive data discovery and gain visibility into data from ingestion sources to identify and protect sensitive columns, datasets, or information that may be part of the ingestion source.
• Data Governance: Aedeon Data lake is designed to provide high-quality, clean, and reliable business-ready information for various data analytics use cases. Aedeon data lake enables appropriate data governance mechanisms. It does not turn a data lake into data swamps when users scale storage from multiple data sources, in numerous structural formats, in large volumes, and in real-time. Data quality, catalog management, lineage, and data governance are maintained using serverless technologies, reducing the administrative overhead of data ingestion and data pipelines(ETL). A standardized visualized interface allows users to catalog data and manage data pipelines(ETL) workflows from the graphical interface without writing the code or needing to engage experts.
• Infrastructure: Aedeon Data Lake allows users to scale Amazon S3 storage and Amazon EKS infrastructure resources to meet evolving demands for analytics workloads. Amazon Auto Scaling ensures that the resources are automatically adjusted to your analytics requirements while maintaining organizational policies for cost optimization and compliance. Customers can build their containers efficiently and scalable manner using the EKS offering to add custom functionality to the Aedeon data lake.
• Analytics: Aedeon Data Lake offers various analytics working environments for business analytics, ad-hoc analytics, and data science use cases. Organizations can plan for long-term resource deployments cost-effectively and efficiently. Aedeon Data Lake integrates the Amazon SageMaker and Amazon Athena services for data science workloads where machine learning engineers may require temporary access to high-performance computing and large storage resources.
• Consumption: Aedeon Data Lake is designed to simplify the data analytics process so that data analysts, data stewards, data forensics experts, and business decision-makers can focus on innovation instead of having to design, develop, run, and manage the complex set of underlying services. Advanced AI/ML and analytics tools from AWS are integrated to take care of infrastructure management and advanced analytics use cases within the appropriate data governance and security policies. Intuitive interfaces and visualizations make it easy for decision-makers to extract valuable knowledge and insights from raw data stored and analyzed with the Aedeon Data Lake.

Result: Efficient, Secure, and Scalable Analytics

Figure 2: Mactores® Aedeon™ Data lake a comprehensive  data platform

• Data Quality: Quality issues such as duplication of attributes when integrating multiple data sources or remediating data quality issues by remediating with the data producers or source are managed and resolved with an end-to-end data quality lifecycle maintained by the Aedeon Data lake.
• Self-Service Data Preparation: The data is integrated from multiple sources and available for immediate processing. Built-in AWS services manage ETL workflows, requiring minimal user intervention, customizations, or coding.
• Data Catalog: The data catalog provides a unified metadata repository across data of various structures and format types. This data catalog allows users to interactively discover and query data directly from the central storage repository.
  
• Managed Ingestion: Data ingestion from multiple data producers and sources can be managed according to identity and access management, data quality, and governance policies.
• Metadata Management: Aedeon tools allow users to capture helpful metadata and simplify management tasks. Efficient and accurate metadata management prevents your data lake from transforming into a swamp.
• Data Lineage: As one of the essential components of data quality and governance strategy, Aedeon allows users to analyze data lineage to ensure that trustworthy insights are derived from the data.
  
• Data Privacy and Security: Ensure compliance with stringent regulations, especially when dealing with sensitive Personally Identifiable Information (PII) data, GDPR compliance, or FedRAMP data.
• Data Lifecycle Management: Manage data across all phases of the end-to-end analytics pipeline. Automatically enforce and manage policies while focusing your efforts on developing innovative and insightful data analytics use cases.

• Mactores ETL Migration Accelerator: for Amazon EMR, Amazon Glue, and Apache Spark.
• Mactores Data Warehouse Migration Accelerator: for Amazon Redshift and Apache Hive.
• Mactores Hadoop Migration Accelerator For Amazon: for Amazon EMR, Apache HBase, and Apache Spark
• Mactores Streaming Migration Accelerator: for Amazon MSK and Amazon Kinesis.