#Merits and demerits of data warehouse | Explore Tumblr posts and blogs

timseigfield · 1 year ago

Text

Data Lake vs Data Warehouse: 10 Key difference

Today, we are living in a time where we need to manage a vast amount of data. In today's data management world, the growing concepts of data warehouse and data lake have often been a major part of the discussions. We are mainly looking forward to finding the merits and demerits to find out the details. Undeniably, both serve as the repository for storing data, but there are fundamental differences in capabilities, purposes and architecture.

Hence, in this blog, we will completely pay attention to data lake vs data warehouse to help you understand and choose effectively.

We will mainly discuss the 10 major differences between data lakes and data warehouses to make the best choice.

Data variety: In terms of data variety, data lake can easily accommodate the diverse data types, which include semi-structured, structured, and unstructured data in the native format without any predefined schema. It can include data like videos, documents, media streams, data and a lot more. On the contrary, a data warehouse can store structured data which has been properly modelled and organized for specific use cases. Structured data can be referred to as the data that confirms the predefined schema and makes it suitable for traditional relational databases. The ability to accommodate diversified data types makes data lakes much more accessible and easier.

Processing approach: When it is about the data processing, data lakes follow a schema-on-read approach. Hence, it can ingest raw data on its lake without the need for structuring or modelling. It allows users to apply specific structures to the data while analyzing and, therefore, offers better agility and flexibility. However, for data warehouse, in terms of processing approach, data modelling is performed prior to ingestion, followed by a schema-on-write approach. Hence, it requires data to be formatted and structured as per the predefined schemes before being loaded into the warehouse.

Storage cost: When it comes to data cost, Data Lakes offers a cost-effective storage solution as it generally leverages open-source technology. The distributed nature and the use of unexpected storage infrastructure can reduce the overall storage cost even when organizations are required to deal with large data volumes. Compared to it, data warehouses include higher storage costs because of their proprietary technologies and structured nature. The rigid indexing and schema mechanism employed in the warehouse results in increased storage requirements along with other expenses.

Agility: Data lakes provide improved agility and flexibility because they do not have a rigid data warehouse structure. Data scientists and developers can seamlessly configure and configure queries, applications and models, which enables rapid experimentation. On the contrary, Data warehouses are known for their rigid structure, which is why adaptation and modification are time-consuming. Any changes in the data model or schema would require significant coordination, time and effort in different business processes.

Security: When it is about data lakes, security is continuously evolving as big data technologies are developing. However, you can remain assured that the enhanced data lake security can mitigate the risk of unauthorized access. Some enhanced security technology includes access control, compliance frameworks and encryption. On the other hand, the technologies used in data warehouses have been used for decades, which means that they have mature security features along with robust access control. However, the continuously evolving security protocols in data lakes make it even more robust in terms of security.

User accessibility: Data Lakes can cater to advanced analytical professionals and data scientists because of the unstructured and raw nature of data. While data lakes provide greater exploration capabilities and flexibility, it has specialized tools and skills for effective utilization. However, when it is about Data warehouses, these have been primarily targeted for analytic users and Business Intelligence with different levels of adoption throughout the organization.

Maturity: Data Lakes can be said to be a relatively new data warehouse that is continuously undergoing refinement and evolution. As organizations have started embracing big data technologies and exploring use cases, it can be expected that the maturity level has increased over time. In the coming years, it will be a prominent technology among organizations. However, even when data warehouses can be represented as a mature technology, the technology faces major issues with raw data processing.

Use cases: The data lake can be a good choice for processing different sorts of data from different sources, as well as for machine learning and analysis. It can help organizations analyze, store and ingest a huge volume of raw data from different sources. It also facilitates predictive models, real-time analytics and data discovery. Data warehouses, on the other hand, can be considered ideal for organizations with structured data analytics, predefined queries and reporting. It's a great choice for companies as it provides a centralized representative for historical data.

Integration: When it comes to data lake, it requires robust interoperability capability for processing, analyzing and ingesting data from different sources. Data pipelines and integration frameworks are commonly used for streamlining data, transformation, consumption and ingestion in the data lake environment. Data warehouse can be seamlessly integrated with the traditional reporting platforms, business intelligence, tools and data integration framework. These are being designed to support external applications and systems which enable data collaborations and sharing across the organization.

Complementarity: Data lakes complement data warehouse by properly and seamlessly accommodating different Data sources in their raw formats. It includes unstructured, semi-structured and structured data. It provides a cost-effective and scalable solution to analyze and store a huge volume of data with advanced capabilities like real-time analytics, predictive modelling and machine learning. The Data warehouse, on the other hand, is generally a complement transactional system as it provides a centralized representative for reporting and structured data analytics.

So, these are the basic differences between data warehouses and data lakes. Even when data warehouses and data lakes share a common goal, there are certain differences in terms of processing approach, security, agility, cost, architecture, integration, and so on. Organizations need to recognize the strengths and limitations before choosing the right repository to store their data assets. Organizations who are looking for a versatile centralized data repository which can be managed effectively without being heavy on your pocket, they can choose Data Lakes. The versatile nature of this technology makes it a great decision for organizations. If you need expertise and guidance on data management, experts in Hexaview Technologies will help you understand which one will suit your needs.

#data lake #data warehouse #data lake vs data warehouse #cloud #technology

0 notes

siva3155 · 6 years ago

Text

300+ TOP SAP HANA Interview Questions and Answers

SAP HANA Interview Questions for freshers experienced :-

1. What Is Sap Hana? SAP HANA stands for High Performance Analytical Appliance- in-memory computing engine. HANA is linked to ERP systems; Frontend modeling studio can be used for replication server management and load control. 2. The Two Types Of Relational Data Stored In Hana? The two types of relational data stored in HANA includes Row Store Column Store 3. What Is The Role Of The Persistence Layer In Sap Hana? SAP HANA has an in-memory computing engine and access the data straightaway without any backup. To avoid the risk of losing data in case of hardware failure or power cutoff, persistence layer comes as a savior and stores all the data in the hard drive which is not volatile. 4. What Is Modeling Studio? Modeling studio in HANA performs multiple task like Declares which tables are stored in HANA, first part is to get the meta-data and then schedule data replication jobs Manage Data Services to enter the data from SAP Business Warehouse and other systems Manage ERP instances connection, the current release does not support connecting to several ERP instances Use data services for the modeling Do modeling in HANA itself essential licenses for SAP BO data services 5. What Are The Different Compression Techniques? There are three different compression techniques Run-length encoding Cluster encoding Dictionary encoding 6. What Is Latency? Latency is referred to the length of time to replicate data from the source system to the target system. 7. What Is Transformation Rules? Transformation rule is the rule specified in the advanced replication setting transaction for the source table such that data is transformed during the replication process. 8. What Is The Advantage Of Slt Replication? SAP SLT works on trigger based approach; such approach has no measurable performance impact in the source system It offers filtering capability and transformation It enables real-time data replication, replicating only related data into HANA from non-SAP and SAP source systems It is fully integrated with HANA studios Replication from several source systems to one HANA system is allowed, also from one source system to multiple HANA systems is allowed. 9. How You Can Avoid Un-necessary Information From Being Stored? To avoid un-necessary information from being stored, you have to pause the replication by stopping the schema-related jobs 10. What Is The Role Of Master Controller Job In Sap Hana? The job is arranged on demand and is responsible for Creating database triggers and logging table into the source system Creating Synonyms Writing new entries in admin tables in SLT server when a table is replicated/loaded

SAP HANA Interview Questions 11. What Happens If The Replication Is Suspended For A Longer Period Of Time Or System Outage Of Slt Or Hana System? If the replication is suspended for a longer period of time, the size of the logging tables increases. 12. What Is The Role Of The Transaction Manager And Session? The transaction manager co-ordinates database transactions and keeps a record of running and closed transactions. When transaction is rolled back or committed, the transaction manager notifies the involved storage engines about the event so they can run necessary actions. 13. How You Can Avoid Un-necessary Logging Information From Being Stored? You can avoid un-necessary logging information from being stored by pausing the replication by stopping the schema-related jobs. 14. How Sql Statement Is Processed? In the HANA database, each SQL statement is implemented in the reference of the transaction. New session is allotted to a new transaction. 15. Name Various Components Of Sap Hana? SAP HANA DB SAP HANA Studio SAP HANA Appliance SAP HANA Application Cloud 16. How To Perform Backup And Recovery Operations? During a regular operation, data is by default stored to the disk at savepoints in SAPHANA. As soon a there is any update and transaction, logs become active and get saved from the disk memory. In case of power failure, the database restarts like any other DB returning to the last savepoint log state. SAP HANA requires backup to protect against disk failure and reset DB to the previous state. The backups simultaneously as the users keep performing their tasks. 17. Define Slt Configuration? Configuration is the meaningful information to establish a connection between source, SLT system and SAP HANA architecture as stated in the SLT system. Programmers are allowed to illustrate a new Configuration in Configuration and Monitoring Dashboard. 18. What Is Stall? The waiting process for data to load from the main memory to the CPU cache is called Stall. 19. Define Different Types Of Information Views.? There are primarily three types of information views in SAP HANA, which are all non-materialized. Attribute view Analytic view Calculation View 20. What Are Configuration And Monitoring Dashboard? They are SLT Replication Application Servers to provide configuration information for data replication. This replication status can also be monitored. 21. What Is Logging Table? Logging table records all replicated changes in the table, which can be further replicated to the target system. 22. How To Define Transformation Rules In Hana? Using advanced replication settings, transformation rules are specified to transfer data from source tables during replication process. For instance, setting rules to covert fields, fill vacant fields and skip records. These rules are structured using advanced replication settings (transaction IUUC_REPL_CONT) 23. The Role Of Transaction Manager And Session? SAP HANA transaction manager synchronizes database transactions keeping the record of closed and open transactions. When a transaction is committed or rolled back, the manager informs all the active stores and engines about the action so that they can perform required actions in time. 24. How Is Sql Statement Processed In Sap Hana? Each SQL statement in SAP HANA is carried out in the form of a transaction. Every time, a new session is allocated to a new transaction. 25. Define Master-controller Job? A Master-controller job is responsible to build database logging table in the source system. It further creates synonyms and new entries in SLT server admin when the table loads / replicates. 26. How Users Can Avoid Un-necessary Storage Of Logging Information? Pause the replication process and terminate the schema-related jobs. 27. Is The Table Size In Source System And Sap Hana System Same? No 28. When To Change The Number Of Data Transfer Jobs? The number of data transfer jobs change when the initial loading speed or latency replication time is not up to the mark. At the end of the initial load, the number of initial load jobs may be reduced. 29. List The Merits And Demerits Of Using Row-based Tables? Merits: No data approach can be faster than row-based if you want to analyze, process and retrieve one record at one time. Row-based tables are useful when there is specific demand of accessing complete record. It is preferred when the table consists of less number of rows. This data storage and processing approach is easier and effective without any aggregations and fast searching.Demerits: The data retrieval and processing operations involve the complete row, even though all the information is not useful. 30. List Advantages Of Column-based Tables.? Allows smoother parallel processing of data as the data in columns is stored vertically. Thus, to access data from multiple columns, every operation can be allocated to a separate processor core. Only specific columns need to be approached for Select query and any column can be used for indexing. Efficient operations since most columns hold unique values and thus, high compression rate. 31. What Table Type Is Preferred In Sap Hana Administration: Column-based Or Row-based? Since analytic applications require massive aggregations and agile data processing, column-based tables are preferred in SAP HANA as the data in column is stored consequently, one after the other enabling faster and easier readability and retrieval. Thus, columnar storage is preferred on most OLAP (SQL) queries. On the contrary, row-based tables force users to read and access all the information in a row, even though you require data from few and/or specific columns. 32. What Is The Main Sap Hana Database Component? Index Server consists of actual data engines for data processing including input SQL and MDX statements and performs authentic transactions. 33. The Concept Of Persistence Layer? The persistence layer in SAP HANA handles all logging operations and transactions for secured backup and data restoring. This layer manages data stored in both rows and columns and provides steady savepoints. Built on the concept of persistence layer of SAP’s relational database, it ensures successful data restores. Besides managing log data on the disk, HANA’s persistence layer allows read and write data operations via all storage interfaces. 34. Define Modeling Studio In Sap Hana Administration.? Modeling Studio is an operational tool in SAP HANA based on Eclipse development and administration, which includes live project creation. The SAP HANA Studio further builds development objects and deploys them, to access and modify data models like HTML and JavaScript files. It also handles various data services to perform data input from SAP warehouse and other related databases. Responsible for scheduling data replication tasks. 35. List The Different Compression Techniques In Hana? Run-length encoding Cluster encoding Dictionary encoding 36. What Is Slt SLT expands to SAP Landscape Transformation referring to trigger –based replication. SLT replication permits data transfer from source to target, where the source can be SAP or non-SAP while the target system has to be SAP HANA with HANA database. Users can accomplish data replication from multiple sources. The three replication techniques supported by HANA are: SLT SAP Business Objects Data Services (BODS) SAP HANA Direct Extractor Connection (DXC) 37. What Is Latency? The time duration to perform data replication starting from the source to the target system is known as latency. 38. What Are The Various Components Of Sap Hana Administration? SAP HANA Studio SAP HANA Application Cloud SAP HANA Cloud Sap HANA DB The Sap Hana Training Videos and Certification Course can open the doors to a stellar career for you. 39. List Advantages Of Using Sap Hana Database.? With the HANA technology, you can create gen-next applications giving effective and efficient results in the digital economy. By using single data-in memory, SAP HANA supports smooth transaction process and fault-tolerant analytics Easy and simple operations using an open-source, unified platform in the cloud High-level Data Integration to access massive amounts of data Advanced tools for in-depth analysis of present, past and the future.Interested in learning SAP HANA? Well, we have the comprehensive Sap Hana Course to give you a head start in your career. 40. Parallel Processing In Sap Hana? Using the columnar data storage approach, the workload in SAP HANA is divided vertically. The columnar approach allows linear searching and aggregation of data rather than two-dimensional data structure. If more than one column is to be processed, each task is assigned to diverse processor. Operations on one column are then collimated by column divisions processed by different processors. 41. You mean I have to buy a HANA only 2.5x smaller than my big Oracle RDBMS? What about archiving and data ageing? Yes, in some instances you may have to buy a HANA appliance that is only 2.5x smaller than it would be under Oracle. And data ageing isn't part of the 1.0 release, but SAP is certainly working on it pretty hard. Let's hope they release something faster than you need to buy a bigger HANA appliance! 42. What's the wider market opportunity for IMDB? This is the interesting thing - no one knows yet, and few analysts seem to have cottoned on that the wider market opportunity might be huge. Think not just SAP applications but any third party that requires ultra-high speed. Think not just an appliance but a development platform. Time will tell. 2. SAP HANA database hardware 43. What hardware is supported right now? Talk to your hardware vendor - all of the major vendors e.g. HP, IBM, Dell, have HANA offerings now. Technically HANA will run on any Intel x64 based system from your laptop through to the big 40-core, 2TB RAM servers. It is however only supported on a small number of big rack-mount servers like the Dell R910 and HP DL980. 44. Why doesn't HANA run on blades? It's unclear but probably because the blades don't yet offer the same performance. HANA is optimized for the Intel X7560 CPU and will run fastest on this. And for instance, the Dell M910 blade can only run 2x X7650 CPUs and 512 GB RAM in this configuration, which probably explains the limitations. What's certain is that HANA will eventually run on blades - it's born to run on blade technology! 45. Does SAP make their own IMDB/HANA hardware? Yes, but only in the labs so far. There are no public plans to compete against IBM/HP/Dell in this space, but it may make sense for SAP to enter the appliance market, especially in the context of Data Centres and even more so in the context of the SAP Business by Design cloud offering, which will run on IMDB. 46. How big does HANA scale? Theoretically at least - very well. The biggest single-server HANA hardware will run most mid-size workloads - 2TB of in-memory storage is equivalent to 5-20TB of Oracle storage. The way that HANA works means that it is possible to chain multiple systems together - meaning that scalability has thus-far been determined by the size of customers' wallets. Do note that whilst SAP talk up "Big Data" quite a lot, HANA currently only scales to the small-end of Big Data, which refers to the kind of huge datasets that Face Book or Google have to store - not Terabytes, but rather Peta bytes. 47. What storage subsystem does HANA use? This varies from vendor to vendor but it is shared network attached storage (NAS). Both regular magnetic disks and SSD storage can be used for the backup of the database (HANA runs in memory remember, so disk storage is just for backup, and later, for data ageing). Note that you require 2x storage that you have RAM, which is 2x the database size - i.e. storage size = 4x database size. In most cases there is additional ultra-high speed SSD storage for log files. 48. What source databases does HANA support in real-time? If you use Sybase Replication Server (SRS) for near real-time data then you need to watch out for licensing still (SAP have license deals pending). If you run DB2 then you're fine but with Oracle and Microsoft SQL Server there are some license challenges if you buy your license through SAP, because you may have a limited license that does not allow extraction. Talk to SAP for further information on this. 49. What source databases does HANA support for batch loads? If you use SAP Business Objects Data Services 4.0 for bulk loads then pretty much anything. BO-DS is a very flexible Extract, Transform & Load tool that supports many databases - check out the specs for more details. 50. What additional limitations does Sybase Replication Server present? SRS has additional restrictions which are worth bearing on mind. It can only replicate Unicode data and does not support IBM DB2 compressed tables. SAP HANA Questions and Answers pdf Download Read the full article

0 notes

siva3155 · 6 years ago

Text

300+ TOP SAP HANA Interview Questions and Answers

SAP HANA Interview Questions for freshers experienced :-

0 notes