#sql db replication

[khayalonsebunealfaz]

Advanced Database Management in Full Stack Development 

Introduction 

A Full Stack Development project's foundation is effective database management. Data transactions and application performance are guaranteed by the capacity to build, optimize, and scale databases. Proficiency with SQL and NoSQL databases, indexing, query optimization, and high availability replication are essential for modern applications. To create scalable systems, developers also need to concentrate on cloud integration, backup plans, and database security. In order to guarantee maximum performance and dependability in full stack applications, this paper examines sophisticated database management strategies. 

Choosing the Right Database: SQL vs. NoSQL 

Choosing the right database is essential for the scalability of the program. Strong data consistency, defined schema, and ACID compliance are features of SQL databases (MySQL, PostgreSQL, and Microsoft SQL Server). Applications needing relational data storage, financial transactions, and sophisticated searches are best suited for them. NoSQL databases, such as MongoDB, Cassandra, and Firebase, offer distributed architecture, high scalability, and customizable schemas. Large-scale, unstructured data processing, such as real-time analytics and Internet of Things applications, is best handled by these. Database efficiency is increased by combining NoSQL for dynamic content and SQL for structured data. 

Optimizing Query Performance and Scaling 

Applications that have poorly optimized queries operate slowly. Data retrieval is accelerated by the use of indexing, query caching, and denormalization. In high-traffic applications, partitioning huge tables improves read/write performance. Performance is enhanced via read and write replicas, which disperse database loads. Sharding lowers latency by dividing big databases into smaller portions that are distributed across several servers. In full stack applications, database interaction is streamlined by using ORM (Object-Relational Mapping) technologies like SQLAlchemy, Hibernate, or Sequelize. 

Database Security and Backup Strategies 

In Full Stack Development, data availability and security must be guaranteed. Unauthorized access is avoided by putting role-based access control (RBAC) into practice. User information is protected by using hashing methods, SSL/TLS, and AES to encrypt important data. Data loss may be avoided by point-in-time recovery, disaster recovery plans, and routine database backups. AWS RDS, Google Cloud Firestore, and Azure Cosmos DB are examples of cloud-based databases that provide fault tolerance, replication, and automated backups to guarantee data dependability and integrity. 

Conclusion 

Building scalable, high-performance applications in Advanced Full Stack Development requires a solid understanding of database administration. System efficiency is increased by selecting between SQL and NoSQL databases, optimizing queries, and protecting data storage. A Full Stack Development Training curriculum gives developers hands-on experience while teaching them sophisticated database approaches. Database management internships provide professionals practical experience in handling data in the real world, preparing them for the workforce. A successful career in Full Stack Development is ensured by investing in database competence. 

[cert007]

Microsoft Azure Cosmos DB DP-420 Practice Exam For Success

If you’re planning to take the DP-420: Designing and Implementing Cloud-Native Applications Using Microsoft Azure Cosmos DB exam, your journey just got easier! Cert007 has introduced an updated Microsoft Azure Cosmos DB DP-420 Practice Exam tailored to ensure your success. With realistic scenarios, detailed explanations, and comprehensive coverage of exam objectives, this resource is a must-have for aspiring professionals. Let's dive into what makes the DP-420 certification a game-changer for your career in cloud-native applications.

What is the DP-420 Exam?

The DP-420 exam is a key step toward earning the Microsoft Certified: Azure Cosmos DB Developer Specialty. This certification validates your expertise in designing, implementing, and monitoring cloud-native applications using Azure Cosmos DB.

Who is this for? Developers, architects, and IT professionals eager to deepen their knowledge of Azure Cosmos DB and gain a competitive edge in cloud computing. The certification showcases your ability to build resilient, high-performance solutions that meet modern business needs.

Exam Prerequisites and Skills Needed

To ace the DP-420 exam, you need solid experience in developing Azure-based applications and a deep understanding of Azure Cosmos DB technologies. Skills include:

Writing efficient SQL queries for the NoSQL API.

Creating indexing policies tailored to application needs.

Interpreting JSON for database operations.

Proficiency in C#, Java, PowerShell, and Azure resource management.

Why Choose Azure Cosmos DB?

Azure Cosmos DB is a globally distributed, multi-model database service designed for high availability and low latency. Its flexibility and scalability make it a top choice for cloud-native applications. Whether you're managing IoT data or delivering real-time analytics, Azure Cosmos DB offers unparalleled performance.

Key Responsibilities of a DP-420 Certified Professional

Designing and Implementing Data Models

As a certified professional, you’re expected to create efficient data models tailored to specific application needs. This involves identifying the right schema, avoiding anti-patterns, and ensuring scalability. For instance, choosing between a denormalized or normalized design can impact performance significantly.

Optimizing and Maintaining Azure Cosmos DB Solutions

Optimization ensures your solution meets performance, availability, and cost goals. From fine-tuning queries to managing throughput, maintenance is a continuous task that keeps your applications running smoothly.

Integrating Azure Cosmos DB with Other Azure Services

Integration amplifies the power of your solutions. Common integrations include Azure Functions for serverless computing, Azure Logic Apps for workflows, and Azure Event Hubs for data streaming.

Detailed Breakdown of Skills Measured in the DP-420 Exam

Designing and Implementing Data Models (35-40%)

Designing data models is one of the most critical skills for the DP-420 exam. This section evaluates your ability to structure data effectively, ensuring scalability and performance. Candidates must:

Define an appropriate data schema for their application needs.

Leverage Azure Cosmos DB’s NoSQL model to create flexible and efficient data designs.

Optimize data queries for minimal latency and maximum throughput.

Best Practices:

Understand your application's access patterns before designing the schema.

Use partitioning strategies to evenly distribute data and workload.

Avoid over-normalization in NoSQL databases to reduce the number of expensive joins.

Data Distribution Design (5-10%)

Azure Cosmos DB provides global distribution capabilities, allowing data to be replicated across multiple regions. In this section, the exam tests your understanding of:

Designing partitioning strategies to support scalability and performance.

Leveraging Azure’s multi-master capabilities for write operations.

Managing consistency levels to balance performance and data accuracy.

Pro Tip:

Partition keys play a vital role in data distribution. Selecting an inappropriate key can lead to hot partitions, which degrade performance.

Solution Integration (5-10%)

This skill area focuses on integrating Azure Cosmos DB with other Azure services to build robust and scalable solutions. Examples include:

Using Azure Logic Apps to automate workflows.

Employing Azure Functions for event-driven data processing.

Integrating with Azure Synapse Analytics for advanced data analysis.

Key Consideration:

Integration efforts should prioritize seamless connectivity, ensuring minimal latency and secure data transfer.

Optimizing Solutions (15-20%)

Optimization is about squeezing the maximum performance out of your Cosmos DB solution while keeping costs manageable. The DP-420 exam tests your ability to:

Fine-tune indexing policies to improve query performance.

Manage throughput using autoscaling and reserved capacity.

Identify and resolve bottlenecks in application performance.

Quick Tip:

Run query diagnostics regularly to identify inefficient queries and adjust indexing policies accordingly.

Maintaining Solutions (25-30%)

Long-term maintenance is critical to ensuring reliability and availability. This section evaluates your ability to:

Monitor Azure Cosmos DB metrics using Azure Monitor and Application Insights.

Handle backups and implement disaster recovery plans.

Ensure compliance with security and regulatory requirements.

Pro Tip:

Use Azure Advisor to receive personalized recommendations for cost optimization and performance improvements.

Tips to Succeed in the DP-420 Exam

Study Strategies for the DP-420 Exam

A structured study plan is crucial. Start with Microsoft’s official documentation and pair it with the Cert007 practice exam for hands-on preparation. Dedicate time to each skill area based on its weight in the exam.

Practical Experience is Key

Theory alone won’t cut it. Work on real-world projects using Azure Cosmos DB to understand how different features and services come together in practice.

Time Management During the Exam

With a set time limit, it’s vital to allocate time wisely across questions. Prioritize questions you know well and revisit the tougher ones later.

Conclusion

The DP-420 certification is a powerful way to showcase your expertise in designing and implementing cloud-native applications using Azure Cosmos DB. By focusing on key skill areas like data modeling, distribution, optimization, and maintenance, you’ll be well-prepared for the challenges of this exam. Pair your preparation with Cert007’s updated DP-420 practice exam to simulate real-world scenarios, learn from your mistakes, and approach the exam with confidence.

[suhailms]

Mastering Database Administration with Your Path to Expert DB Management

In the age of data-driven businesses, managing and securing databases has never been more crucial. A database administrator (DBA) is responsible for ensuring that databases are well-structured, secure, and perform optimally. Whether you're dealing with a small-scale application or a large enterprise system, the role of a database administrator is key to maintaining data integrity, availability, and security.

If you're looking to build a career in database administration or enhance your existing skills, Jazinfotech’s Database Administration course offers comprehensive training that equips you with the knowledge and hands-on experience to manage databases efficiently and effectively.

In this blog, we’ll explore what database administration entails, why it's an essential skill in today's tech industry, and how Jazinfotech’s can help you become an expert in managing and maintaining databases for various platforms.

1. What is Database Administration (DBA)?

Database Administration refers to the practice of managing, configuring, securing, and maintaining databases to ensure their optimal performance. Database administrators are responsible for the overall health of the database environment, including aspects such as:

Data Security: Ensuring data is protected from unauthorized access and data breaches.

Database Performance: Monitoring and optimizing the performance of database systems to ensure fast and efficient data retrieval.

Backup and Recovery: Implementing robust backup strategies and ensuring databases can be restored in case of failures.

High Availability: Ensuring that databases are always available and accessible, even in the event of system failures.

Data Integrity: Ensuring that data remains consistent, accurate, and reliable across all operations.

Database administrators work with various types of databases (SQL, NoSQL, cloud databases, etc.), and they often specialize in specific database management systems (DBMS) such as MySQL, PostgreSQL, Oracle, Microsoft SQL Server, and MongoDB.

2. Why is Database Administration Important?

Database administration is a critical aspect of managing the infrastructure of modern organizations. Here are some reasons why database administration is vital:

a. Ensures Data Security and Compliance

In today’s world, where data breaches and cyber threats are prevalent, ensuring that your databases are secure is essential. A skilled DBA implements robust security measures such as encryption, access control, and monitoring to safeguard sensitive information. Moreover, DBAs are responsible for ensuring that databases comply with various industry regulations and data privacy laws.

b. Optimizes Performance and Scalability

As organizations grow, so does the volume of data. A good DBA ensures that databases are scalable, can handle large data loads, and perform efficiently even during peak usage. Performance optimization techniques like indexing, query optimization, and database tuning are essential to maintaining smooth database operations.

c. Prevents Data Loss

Data is often the most valuable asset for businesses. DBAs implement comprehensive backup and disaster recovery strategies to prevent data loss due to system crashes, human error, or cyber-attacks. Regular backups and recovery drills ensure that data can be restored quickly and accurately.

d. Ensures High Availability

Downtime can have significant business impacts, including loss of revenue, user dissatisfaction, and brand damage. DBAs design high-availability solutions such as replication, clustering, and failover mechanisms to ensure that the database is always accessible, even during maintenance or in case of failures.

e. Supports Database Innovation

With the evolution of cloud platforms, machine learning, and big data technologies, DBAs are also involved in helping organizations adopt new database technologies. They assist with migration to the cloud, implement data warehousing solutions, and work on database automation to support agile development practices.

3. Jazinfotech’s Database Administration Course: What You’ll Learn

At Jazinfotech, our Database Administration (DBA) course is designed to give you a thorough understanding of the core concepts and techniques needed to become an expert in database management. Our course covers various DBMS technologies, including SQL and NoSQL databases, and teaches you the necessary skills to manage databases effectively and efficiently.

Here’s a breakdown of the core topics you’ll cover in Jazinfotech’s DBA course:

a. Introduction to Database Management Systems

Understanding the role of DBMS in modern IT environments.

Types of databases: Relational, NoSQL, NewSQL, etc.

Key database concepts like tables, schemas, queries, and relationships.

Overview of popular DBMS technologies: MySQL, Oracle, SQL Server, PostgreSQL, MongoDB, and more.

b. SQL and Query Optimization

Mastering SQL queries to interact with relational databases.

Writing complex SQL queries: Joins, subqueries, aggregations, etc.

Optimizing SQL queries for performance: Indexing, query execution plans, and normalization.

Data integrity and constraints: Primary keys, foreign keys, and unique constraints.

c. Database Security and User Management

Implementing user authentication and access control.

Configuring database roles and permissions to ensure secure access.

Encryption techniques for securing sensitive data.

Auditing database activity and monitoring for unauthorized access.

d. Backup, Recovery, and Disaster Recovery

Designing a robust backup strategy (full, incremental, differential backups).

Automating backup processes to ensure regular and secure backups.

Recovering data from backups in the event of system failure or data corruption.

Implementing disaster recovery plans for business continuity.

e. Database Performance Tuning

Monitoring and analyzing database performance.

Identifying performance bottlenecks and implementing solutions.

Optimizing queries, indexing, and database configuration.

Using tools like EXPLAIN (for query analysis) and performance_schema to improve DB performance.

f. High Availability and Replication

Setting up database replication (master-slave, master-master) to ensure data availability.

Designing high-availability database clusters to prevent downtime.

Load balancing to distribute database requests and reduce the load on individual servers.

Failover mechanisms to automatically switch to backup systems in case of a failure.

g. Cloud Database Administration

Introduction to cloud-based database management systems (DBaaS) like AWS RDS, Azure SQL, and Google Cloud SQL.

Migrating on-premise databases to the cloud.

Managing database instances in the cloud, including scaling and cost management.

Cloud-native database architecture for high scalability and resilience.

h. NoSQL Database Administration

Introduction to NoSQL databases (MongoDB, Cassandra, Redis, etc.).

Managing and scaling NoSQL databases.

Differences between relational and NoSQL data models.

Querying and optimizing performance for NoSQL databases.

i. Database Automation and Scripting

Automating routine database maintenance tasks using scripts.

Scheduling automated backups, cleanup jobs, and index maintenance.

Using Bash, PowerShell, or other scripting languages for database automation.

4. Why Choose Jazinfotech for Your Database Administration Course?

At Jazinfotech, we provide high-quality, practical training in database administration. Our comprehensive DBA course covers all aspects of database management, from installation and configuration to performance tuning and troubleshooting.

Here’s why you should choose Jazinfotech for your DBA training:

a. Experienced Trainers

Our instructors are seasoned database professionals with years of hands-on experience in managing and optimizing databases for enterprises. They bring real-world knowledge and industry insights to the classroom, ensuring that you learn not just theory, but practical skills.

b. Hands-On Training

Our course offers plenty of hands-on labs and practical exercises, allowing you to apply the concepts learned in real-life scenarios. You will work on projects that simulate actual DBA tasks, including performance tuning, backup and recovery, and database security.

c. Industry-Standard Tools and Technologies

We teach you how to work with the latest database tools and technologies, including both relational and NoSQL databases. Whether you're working with Oracle, SQL Server, MySQL, MongoDB, or cloud-based databases like AWS RDS, you'll gain the skills needed to manage any database environment.

d. Flexible Learning Options

We offer both online and in-person training options, making it easier for you to learn at your own pace and according to your schedule. Whether you prefer classroom-based learning or virtual classes, we have the right solution for you.

e. Career Support and Placement Assistance

At Jazinfotech, we understand the importance of securing a job after completing the course. That’s why we offer career support and placement assistance to help you find your next role as a Database Administrator. We provide resume-building tips, mock interviews, and help you connect with potential employers.

5. Conclusion

Database administration is a critical skill that ensures your organization’s data is secure, accessible, and performant. With the right training and experience, you can become a highly skilled database administrator and take your career to new heights.

Jazinfotech’s Database Administration course provides the comprehensive knowledge, hands-on experience, and industry insights needed to excel in the field of database management. Whether you’re a beginner looking to start your career in database administration or an experienced professional aiming to deepen your skills, our course will help you become a proficient DBA capable of managing complex database environments.

Ready to kickstart your career as a Database Administrator? Enroll in Jazinfotech’s DBA course today and gain the expertise to manage and optimize databases for businesses of all sizes!

[govindhtech]

Deep Dive into Azure Cache for Redis: A Developer’s Guide

Azure Cache for Redis

Scale performance easily and economically as app needs and traffic grow. With the advantages of a fully managed service, handle thousands of concurrent users with near-instantaneous performance by adding a fast cache layer to the application design.

Better speed and throughput to manage millions of requests per second with latency as low as sub-milliseconds. You can concentrate on development with Azure’s fully managed service that has automated patching, updates, scalability, and provisioning. RedisJSON, RedisTimeSeries, RedisBloom, and RediSearch module integration, enabling data streaming, search, and analysis. Strong features include Redis on Flash, clustering, active geo-replication, and up to 99.999 percent availability. What can be constructed using Azure Cache Redis Reduce latency and increase application performance To speed up your data layer via caching, use Azure Cache for Redis. At a reasonable cost, adding Azure Cache for Redis may boost latency performance by over 1,000 percent and enhance data throughput by over 800 percent. It’s a fantastic method for quickly scaling apps and enhancing user experience without paying for a costly database redesign.

Applications may run faster using a distributed cache

Provide your data layer with the ability to extend throughput at a cheaper cost than with larger database instances, complementing database services like Azure SQL Database and Azure Cosmos DB. By using a common cache-aside structure, you may store and exchange session states, static content, and database query results while improving the nimbleness and scalability of your application.

Effectively store session information Save, retrieve, and update web session data such as output pages and user cookies quickly. Enhance your application’s responsiveness and capacity to manage growing loads with less web-computing resources to improve performance. To maximize user data dependability, take use of data permanence and automated data duplication. To satisfy the demands of even the biggest companies, use clustering to scale up to terabyte sizes and synchronize session data worldwide with active geo-distribution.

Act as an intermediary for messages between services To develop queue or publish/subscribe structures, use Azure Cache for Redis. Send messages in real time and expand online communication frameworks like SignalR. For data in transit, use industry-standard TLS encryption, and set up appropriate network isolation using Azure Private Link.

Boost app performance on Azure with Redis Enterprise

With Azure Cache for Redis’s Enterprise and Enterprise Flash tiers, which were created in collaboration with Redis Inc., you can get revolutionary support for well-liked Redis Enterprise capabilities. To build globally distributed caches with up to 99.999% availability, use active geo-replication. Utilize the RediSearch, RedisBloom, and RedisTimeSeries modules that come with the package to add new data structures that improve analytics and machine learning. And run Redis on fast flash storage by using the Enterprise Flash tier, which offers huge cache sizes at a cheaper cost.

Azure Redis Cache Savor Redis as a completely managed solution Use an Azure fully managed service to expedite deployment. Redis data structures, such as hashes, lists, and sorted sets, are natively supported by Azure Cache for Redis, which is powered by the open-source Redis software. Azure Cache for Redis is compatible with applications that utilize Redis. Import and export Azure Cache for Redis database files to expedite the migration process.

Summary Leading supplier of in-memory data storage solutions, Redis, announced today a change to its open-source licensing policy. Redis is switching from the BSD 3-Clause License to a dual-license approach, allowing developers to choose between the first-ever Server-Side Public License (SSPLv1) and the second-ever Redis Source Available License (RSALv2).

Azure Redis cache pricing Azure Cache for Redis lets you utilize a safe open-source cache. Microsoft’s dedicated solution provides rapid data access to construct highly scalable and responsive applications. Redis’ comprehensive feature set and ecosystem are combined with Microsoft’s trustworthy hosting and monitoring.

FAQs What steps must I take in order to utilize Azure Cache for Redis? The impending Redis license changes will not impact Azure Cache for Redis, including Azure Cache for Redis Enterprise and Enterprise Flash. Users may utilize the service without any action.

RSALv2: What is it? Redis has provided a permissive, non-exclusive, royalty-free license known as RSALv2, or Redis Source Available License 2.0, which allows the usage, distribution, and modification of certain software components created to operate with Redis, subject to the terms and restrictions specified in the agreement.

SSPLv1, what is it? With a changed Section 13, the SSPL is a variation of the GNU Affero General Public License (AGPL). Any organization providing software under SSPL license as a service is required to make available all of the service’s source code, which includes management, user interfaces, APIs, automation, monitoring, backup, storage, and hosting software. This way, users can use the provided source code to run their own instances of the software.

Is the license for RSALv2 and SSPLv1 still valid for Azure Cache for Redis? No, clients may only use Azure Cache for Redis in accordance with Microsoft’s terms of service. The usage of Azure Cache for Redis is not subject to the RSALv2 and SSPLv1 licenses.

Will updates and bug fixes for my Azure Cache for Redis instance be provided going forward? Indeed, even after the aforementioned license announcement, updates and bug fixes will still be available for Azure Cache for Redis, Azure Cache for Redis Enterprise, and Enterprise Flash.

Will the Redis version that is available change as a result of Azure Cache for Redis? The basic community versions of Redis and Redis Enterprise will still be supported by Azure Cache for Redis, Azure Cache for Redis Enterprise, and Enterprise Flash.

In light of this license news, what steps do I as an Azure Cache for Redis client need to take? As of right now, our Microsoft and Azure clients are not needed to take any action in relation to the license announcement.

Read more on Govindhtech.com

[jcmarchi]

Cloud Databases: Trends and Predictions for 2024 - Technology Org

New Post has been published on https://thedigitalinsider.com/cloud-databases-trends-and-predictions-for-2024-technology-org/

Cloud Databases: Trends and Predictions for 2024 - Technology Org

What Are Cloud Databases?

Cloud databases are an online solution for data storage and management, offering a flexible, scalable, and cost-effective alternative to traditional database systems. They are hosted on a cloud computing platform by a service provider, freeing businesses from the cost and overhead of maintaining the physical hardware and infrastructure.

Cloud database – abstract artistic impression. Image credit: vectorjuice via Freepik, free license

Cloud databases offer several advantages. They provide the ability to access your data from anywhere, at any time, and from any device with an internet connection. They also offer flexibility, as they can easily be scaled up or down (or in some cases, in and out), depending on your data storage needs. Plus, with most cloud database service providers offering a pay-as-you-go model, you only pay for the storage you actually use.

However, like any technology, cloud databases also have their challenges. Data security and privacy are among the top concerns, especially with the increasing number of data breaches. There’s also the issue of data governance, particularly in terms of compliance with data protection laws and regulations.

The Key Types of Cloud Databases

SQL Databases

SQL, or Structured Query Language, is a type of database that uses structured query language for defining and manipulating the data. SQL databases are known for their reliability, ease of use, and compatibility with many applications. They are ideal for businesses that handle structured data and require complex queries.

However, SQL databases can be challenging to scale, making them less ideal for businesses that handle large amounts of data. They also have a fixed schema, which means you need to define the structure of your data before you can store it. This can be a drawback if your data’s structure frequently changes or if you handle unstructured data.

Running SQL databases in the cloud involves utilizing cloud-based services that support relational database structures. Cloud providers offer managed services that handle routine database maintenance tasks such as provisioning, backups, patching, and scaling. Examples of these services include Amazon RDS, Google Cloud SQL, and Azure SQL Database. These platforms enable users to run SQL databases without the complexity of managing the underlying infrastructure.

NoSQL Databases

NoSQL databases, on the other hand, are designed for scalability and flexibility. They can handle large amounts of structured, semi-structured, and unstructured data, making them ideal for businesses that handle big data. NoSQL databases also have a flexible schema, allowing you to store data without defining its structure first.

However, NoSQL databases are not as mature as SQL databases. They may not offer the same level of functionality, and there might be compatibility issues with some applications. Additionally, NoSQL databases are not as straightforward to use as SQL databases, and they may require more technical expertise.

NoSQL databases in the cloud leverage the inherent scalability and flexibility of cloud computing. Cloud-based NoSQL services like Amazon DynamoDB, Google Firestore, and Azure Cosmos DB provide a serverless experience where the database automatically scales to meet the application’s demands without the need for manual intervention. This can be particularly beneficial for applications with unpredictable workloads. NoSQL databases in the cloud also offer replication across multiple geographical regions to ensure high availability and data redundancy.

In-Memory Databases

In-memory databases store data in the system’s main memory instead of on a disk, resulting in faster data processing times. They are ideal for applications that require real-time analytics, high performance, and high availability.

However, in-memory databases can be expensive, especially for large amounts of data. They also have limited storage capacity, as they rely on the system’s main memory, which is typically smaller than a disk.

Cloud providers offer in-memory databases as services, like Amazon ElastiCache and Azure Redis Cache, which deliver high-performance data storage and retrieval by leveraging RAM. These services manage the complex administration of in-memory databases, allowing users to focus on the application logic. The use of cloud resources also means that businesses can start with a smaller deployment and scale as needed while paying only for the resources utilized.

Time-Series Databases

Time-series databases are designed to handle time-series data, which are data points collected over time. They are ideal for applications that require the analysis of trends and patterns over time, such as stock market analysis, weather forecasting, and IoT applications.

However, time-series databases are not suitable for all types of data. They are specialized databases, and using them for non-time-series data can result in inefficiencies.

Services like Amazon Timestream or InfluxDB Cloud allow users to run time-series databases entirely in the cloud. These databases are optimized for time-stamped or time-series data, offering efficient ingestion, compression, and querying. The cloud-based time-series databases provide scalability to handle massive influxes of data points and the ability to query across large datasets rapidly. They are often integrated with other analytics and IoT services, allowing for real-time monitoring and decision-making applications.

NewSQL Database

NewSQL databases are a newer type of database that combines the best of SQL and NoSQL databases. They offer the scalability and flexibility of NoSQL databases while maintaining the reliability and ease of use of SQL databases.

However, as a newer technology, NewSQL databases might not be as mature or stable as other types of databases. They might also require more technical expertise to use effectively.

NewSQL databases combine the benefits of traditional SQL systems with the scalability of NoSQL systems and are offered through cloud platforms like Google Spanner or CockroachDB. These databases are designed to scale horizontally across cloud infrastructure, providing global distribution and high availability. They maintain strong consistency across distributed nodes, making them suitable for transactional workloads that require accuracy and integrity.

Cloud Databases: Trends and Predictions for 2024

In the next few years, cloud databases will continue to evolve, bringing about newer trends and technological advancements. Let’s explore these expected developments and how they can affect your data management future.

1. Blockchain Databases and Distributed Ledger Technology

Blockchain databases and distributed ledger technology are set to make a big impact on how cloud databases operate. Blockchain databases, with their ability to decentralize data, provide an added layer of security, making them exceptionally resistant to cyber threats.

Blockchain technologies not only reduce the risk of a single point of failure but also enhance data transparency and traceability. With distributed ledger technology, you can expect greater trust in data integrity, as every transaction is recorded and visible across the network.

As you plan your business’s digital strategy, consider how you can integrate blockchain databases into your operations. Their potential to foster data security and transparency can significantly benefit industries like finance, supply chain, healthcare, and more.

2. AI and Machine Learning Integration

AI and machine learning are no longer buzzwords, but integral components of modern cloud databases. These technologies enable you to extract valuable insights from your data, aiding in predictive analysis, trend identification, and decision-making processes.

Imagine having the ability to forecast customer behavior, market trends, or operational issues without having to build a complex data science architecture. With AI and machine learning directly integrated into your cloud databases, you can do just that.

By leveraging integrated AI capabilities, you can transform your organization’s data into actionable insights and drive strategic decision-making, with a fraction of the cost and effort of setting up previous-generation machine learning solutions.

3. Serverless Databases

The concept of serverless databases is another trend that’s gaining traction. Unlike traditional databases that require server management, serverless databases enable you to focus on your core business functions without worrying about infrastructure management.

Imagine a scenario where you don’t have to worry about server provisioning, patching, or scaling. With serverless databases, is a reality. These databases automatically manage these tasks, allowing you to scale seamlessly and pay only for the resources you use.

The adoption of serverless databases can result in cost savings, improved efficiency, and the ability to focus on data-driven insights rather than infrastructure management. As you strategize for the future, consider how serverless databases could streamline your operations and enhance your data management capabilities.

4. Handling Multiple Data Models

As the volume and complexity of data increases, the need for databases that can handle multiple data models also grows. These multi-model databases can store diverse data types like documents, graphs, and key-value pairs in a single, integrated backend.

Multi-mode databases let you query across different data types and access insights in real-time. These databases provide flexibility in data storage and retrieval, leading to improved efficiency and data accessibility.

Incorporating multi-model databases into your strategy can enhance your ability to handle complex data and derive meaningful insights. Whether it’s analyzing customer behavior from diverse data points or integrating data from various sources, multi-model databases can provide unprecedented versatility.

5. Advances in Encryption Technologies

As data breaches become increasingly common, advances in encryption technologies are more critical than ever. These technologies protect your data by converting it into an unreadable format, ensuring that it remains secure even if it falls into the wrong hands. Modern encryption systems operate end-to-end, ensuring that data is encrypted when stored in your database, across transfer channels, and on the end-user’s device as well.

With encryption, your sensitive data is shielded from cyber threats, providing you with peace of mind. As you plan your data strategy, consider how advances in encryption technologies can enhance your data security. From end-to-end encryption to post-quantum encryption, these technologies can provide robust protection for your data in the cloud.

6. Databases at the Edge

Edge computing is another trend that’s set to revolutionize cloud databases. By processing data closer to its source, edge computing reduces latency, improves performance, and enables real-time data analysis.

Edge-based databases let you process and analyze data instantaneously, right where it’s generated. By integrating edge computing with your cloud databases, you can enjoy faster data processing, reduced network congestion, and improved operational efficiency.

Incorporating edge computing into your data strategy can provide your business with a competitive edge, enabling real-time insights, faster decision-making, and enhanced customer experiences.

7. Data Sovereignty and Compliance

Data sovereignty and compliance are critical considerations for businesses operating in the cloud. With regulations like GDPR and CCPA in effect, it’s crucial to ensure that your data is stored and processed in compliance with legal requirements.

As you plan your data strategy, consider how you can ensure data sovereignty and compliance. From selecting cloud providers with robust compliance certifications to implementing data governance practices, there are several ways to uphold data sovereignty and compliance in the cloud.

As we look towards 2024 and beyond, it’s clear that cloud databases will continue to play a pivotal role in data management. From blockchain databases to AI integration, serverless databases to multi-model databases, and advances in encryption to edge computing and data sovereignty, these trends and advancements are set to revolutionize how businesses handle their data.

Author Bio: Gilad David Maayan

Gilad David Maayan is a technology writer who has worked with over 150 technology companies including SAP, Imperva, Samsung NEXT, NetApp and Check Point, producing technical and thought leadership content that elucidates technical solutions for developers and IT leadership. Today he heads Agile SEO, the leading marketing agency in the technology industry.

LinkedIn: https://www.linkedin.com/in/giladdavidmaayan/

[tuckermatthew108]

What is SQL Server Data Replication – Process and Tools

Most organizations have a strong and flexible Database Management System (DBMS) in place to handle massive amounts of data generated in routine operations. It helps users to access databases, change and manipulate data, and generate reports to take cutting-edge business decisions. There are four types of DBMS - Hierarchical, Network DBMS, Relational DBMS, and Object-Oriented Relation DBMS of which Relational DBMS is related to SQL. Relational DBMS defines databases in the form of tables and have pre-fixed data forms that support MySQL, Oracle, and Microsoft SQL Server database. Coming to the definitions, what is SQL? It is Structured Query Language and is the main platform for dealing with Relational Database. SQL is used to update, insert, search, and delete database records. Other benefits of SQL include optimizing and maintaining Relational databases.

Replication is the technology that processes, copies, and distributes database objects from one database to another. It is also used to distribute data to different locations and remote and mobile users. It is done through the Internet, dial-up connections, wireless connections, and local and wide area networks. Before the era of SQL server data replication, organizations had a single server for multiple users at various locations resulting in performance and availability problems and maintenance issues. Now SQL server replication tools help to maintain database copies at multiple locations. SQL Server data replication improves the performance of SQL Server databases and the availability of information across an organization. It ensures that businesses can integrate and load data into data warehouses, migrate data to cloud storage platforms, and copy and distribute data across multiple database servers in remote locations. These activities are regardless of whether a third-party strategy is created by a replication expert or the process is implemented using built-in MSSQL replication utilities. Enterprises can also accelerate, augment, and reduce the cost of SQL data replication using SQL change data capture technology (CDC). SQL Server data replication CDC also makes sure that data-driven enterprises integrate data in real-time and realize faster time-to-insight. It is because changes and updates that are made by individuals at various locations are synchronized to the main server quickly as SQL Server replication distributes the relevant part of the tables and views only and not the entire database. There are three types of SQL Server data replication. The first is Snapshot Replication. A “snapshot” of the data in one server is taken which is then moved to another server or another database in the same server. Another is Transactional Replication where after the initial snapshot replication, only the changes, updates, or deleted records on the publisher are forwarded to the subscriber/s. Finally, there is Merge Replication where data from multiple sources are merged into a single database. SQL Server data replication synchronizes data across enterprises and replication to SQLCE 3.5 and SQLCE 4.0 are supported by both Windows Server 2012 and Windows 8.

[powersmpp]

High availability vs Load balancing vs redundancy in messaging application SMPP

This article written based on our  experience on real world in messaging domain. Though the writing may not be very impressive, but we can assure you that a complete reading  will give you enough information on how to Scale with Speed.

I often get requests from our clients for high availability of their application. While requesting for high availability most of the time people assume that it will also act as load balancer and scale their application. But it is not true. Load balancer is actually a prerequisite to implement high availability. Before I discuss further on load balancer, let me briefly tell you that high availability precisely means running your application from multiple geo-location which can be within a country or can be in two different continents. Choosing the ideal scenario depends on your customer base. If you are catering for only domestic audience then its better to host your services within the same country to avoid any undue latency.

Coming to redundancy, the scope is very wide. Degree of redundancy is relative and varies man to man. An organisation my feel that having their application running from two different geo-location is redundant, whereas at the same time another organization may like to put some more servers in different continents to make their application even more redundant for global audience. Some may be happy with a single load balancer whereas at the same time, some may think of adding multiple load balancer to achieve a near zero downtime. In short redundancy is a relative choice and should be decided based on criticality and affordability.

Load balancer is usually configured to distribute traffic based on round robin algorithm to all the healthy and connected nodes. It also does a health check at regular intervals to keep a track of healthy nodes. For pure TCP traffic such as SMPP, you need a TCP based load balancer and for web servers you can use Nginx as load balancer.

Challenges: The biggest challenge of setting up high availability application are regular updates. Any new update needs to be pushed to all the locations. If you are using a docker container, there are tools available to make your life easier. For RESTful API updates are straight forward, but in the case of large web application, it is highly recommended to use CDN for all the assets, to make the updates smoother.

Now that your application is fully redundant, its time to discuss about other components such as caching Db, SQL databases, any other service running important job, file system (logs), upstream connectivity etc. Can we make everything redundant? Answer is yes with some riders. Lets go one by one.

Database: Enough work has been done by Opensource community to make major databases highly scalable. For SQL database, sharding(horizontal scaling) should be treated as a buzzword and should be adopted as a last resort especially when you have exhausted all the options to increase your write. A quick suggestion is to use SSD drives to increase your write performance. The moment you decide to scale horizontally, its day zero for you and entire application may require an architectural change. So my humble suggestion is to scale horizontally as long as possible.

Humble suggestions on designing Database Architecture :

One: Never limit yourself running single instance to save some cost. From day one, you must opt for a replication server sitting at different geo-location. At initial stage we try to keep application and Db on same machine to save some cost. But, what happens if you are attacked with a ransomware, when you are just getting started? Your replication server will come to your rescue. There will be some downtime, but it will save you from going into depression. So thumb rule is to use a replication server from day one of your business. Important Note: RAID and replication is not same. RAID can safeguard you from physical damage of disk but not from ransomware.

Two: Try to keep your Db server on a different machine for better performance and added security.

Three: When you reach the write bottleneck (IOPS), try to use one of your replication machines as  read node. Even if it fails to handle the desired load of traffic, do not waste your time in adding a new machine to your cluster. If you have more than two machines and make one of them as a proxy. (Details on setting up of Proxy is out of the scope of this blog, but it’s not that complicated if you know how to do it.

Final Say: Scaling your DB is a decision rather than know how. Go for vertical scaling followed by cluster. Sharding should be avoided.

[japanese-influenced-blog1]

SQL Server Version - a Quick Outline

The Argument About SQL Server Version

The demo edition of the computer software is in a position to save scanned MDF file in STR format but should you need to export recovered data to SQL Server database you've got to buy license edition of MS SQL Recovery Software. To be aware of the software more deeply it's possible to download the demo version of the software totally free of cost. It is possible to only get the internet edition through application hosting providers. There are some techniques to set the edition of SQL Server that's installed. There are several different versions of SQL Server 2016 and all of them are available for download for everybody that has a valid MSDN subscription. It's currently available as a free download so that you can manage any instance without requiring a complete license. 

Manual technique to fix corrupted MDF file isn't so straightforward and several times it isn't able to repair due to its limitations. In that case, exitnetcat and you're going to observe the file. If you make the key database file read-only in order to acquire a read-only database and attempt to open it in a greater version, you'll receive an error 3415. The database doesn't use any 2012 specific capabilities. A MySQL database for beginners course is a great place to begin your education. Storage Engine MySQL supports lots of storage engines. Most APIs concerning user interface functionality aren't offered.

Expand Databases, then right-click the database that you would like to shrink. The database is currently upgraded. You may use an existing SQL Server database if it's already set up on the system, or put in a new instance as a portion of the SolidWorks Electrical installation. 

Begin all your SharePoint solutions, now you've successfully migrated your database. You may now begin creating databases. SQL database has become the most common database utilized for every kind of business, for customer accounting or product. Choose the edition of MS SQL Database you desire. The 2008 server cannot restore SQL Server Version backups made by a 2012 server (I've tried). SQL Server 2016 have several new capabilities. SQL Server 2005 is a good example. It also includes an assortment of add-on services. Nowadays you have your whole SQL Server prepared to rock and roll so you may install SharePoint.

There are 3 things you will need to obtain a SQL Server ready. It allows you to define a Magic Table. In such case, it chooses the plan that is expected to yield the results in the shortest possible time. It also provides the optimistic concurrency control mechanism, which is similar to the multiversion concurrency control used in other databases. On the other hand, it enables developers to take advantage of row-based filtering. 

Life, Death, and SQL Server Version

As stated by the SQL Server development group, the adjustments to the database engine are made to supply an architecture that will endure for the subsequent 10 decades. Clicking on the database will permit you to observe the results in detail. Probably one reason the Oracle RDBMS has managed to stay on top of mighty RDBMS is associated with its product updates which are closely tied to changes on the market. In SQL Server there are two kinds of instances. You might have associated your DB instance with an ideal security group once you created your DB instance. 

When you make a new user, you need to add this to sudo user group which provides the new user with administrator privileges. You've learned how to make a new user and provide them sudo privileges. So, you've got to make a new user and provide its administrative access. The 1 thing about the main user is it has each of the privileges in the Ubuntu. If you're root user and will need to bring the public key to the particular user, change to that user by making use of the below command. If you own a client that doesn't understand the (localdb) servername then perhaps it is possible to connect over named pipes to the case. 

The steps are given for the entire process of migrating SQL Server Database to lessen version. There is considerably more to upgrading your application than simply upgrading the database you should be conscious of. The computer software is a comprehensive recovery solution with its outstanding capabilities. Enterprise is suited to demanding database and company intelligence requirements. 

SQL Server Version Can Be Fun for Everyone

Data change replication is a practice of dynamic synchronization. The Oracle TIMESTAMP data type can likewise be used. Even though the data is masked, it's not necessarily in the format that might have been desired. Scrubbed data can impact the efficacy of testing by skewing query benefits. 

The sequence of actions essential to execute a query is known as a query program. You can also locate the version number utilizing mysqladmin command line tool. Knowing the different SQL Server build numbers is a significant item of management info.

[computingpostcom]

Welcome to our guide on how to install ArangoDB on Ubuntu 20.04/18.04. ArangoDB is a multi-model, open-source database with flexible data models for graphs, documents, and key-values. It is designed for high-performance applications which use SQL-like query language or JavaScript extensions. ArangoDB is ACID compliant and it is designed to scale horizontally with a few mouse clicks. On top of that, it also provides an internal API that is used to communicate with clients. ArangoDB doesn’t support SQL as it comes with its own declarative language called AQL (ArangoDB Query Language). AQL is a single SQL-like query language for all data models of the database. Key features of ArangoDB It is fully open source and released under Apache 2.0 License. Has a flexible data modeling which allows you to model your data as combination of key-value pairs, documents or graphs – perfect for social relations Comes with a Powerful query language (AQL) for modifying and retrieving data. You can run queries on multiple documents or collections with optional transactional consistency and isolation Support for Replication and Sharding: You can set up the database in a master-slave configuration or spread bigger datasets across multiple servers You can use ArangoDB as an application server and fuse your application and database together for maximal throughput. Good storage for performance and data durability through its support for modern storage hardware, like SSD and large caches. ArangoDB can be easily deployed as a fault-tolerant distributed state machine, which can serve as the animal brain of distributed appliances Install ArangoDB on Ubuntu 20.04/18.04 The process of installing ArangoDB on Ubuntu 20.04/18.04 is straightforward. This guide will demonstrate a single instance installation of ArangoDB but you can also do: Master/Slave installation Active Failover setup Cluster installation Multiple Datacenters & Standalone Agency setup The packages are readily available on an Apt repository. The only manual work required is adding the apt repository to your system using the commands below. Step 1: Add ArangoDB Apt repository Run the following command to add an apt repository to your system. echo 'deb https://download.arangodb.com/arangodb34/DEBIAN/ /' | sudo tee /etc/apt/sources.list.d/arangodb.list Step 2: Import repository key The next step is to import GPG key used for signing the packages. wget -q https://download.arangodb.com/arangodb34/DEBIAN/Release.key -O- | sudo apt-key add - You should see the “OK” message as the output. Step 3: Update system and install ArangoDB Update your Apt package cache then install ArangoDB on Ubuntu 18.04. sudo apt update sudo apt -y install apt-transport-https sudo apt -y install arangodb3 You will be required to answer a couple questions during installation. 1. Set DB root password Enter the password and confirm it. 2. Select if to automatically upgrade database files. 3. Choose the database storage engine to use. 4. Set to backup database before doing an upgrade. Wait for the installation to finish then proceed to step 3 where you’ll start and manage ArangoDB service with systemd. If you had miss to set root password during installation, you can secure ArangoDB after installation by running: sudo arango-secure-installation Step 4: Start ArangoDB service The service should be automaticaly started upon installation. If it is not started, you can manually start it using systemctl service management command. sudo systemctl start arangodb3 sudo systemctl enable arangodb3 Check service with below command. $ systemctl status arangodb3 ● arangodb3.service - ArangoDB database server Loaded: loaded (/lib/systemd/system/arangodb3.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-05-02 22:39:24 UTC; 1min 14s ago Main PID: 29622 (arangod)

Tasks: 18 (limit: 131072) Memory: 221.9M CGroup: /system.slice/arangodb3.service └─29622 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb3/arangod.pid --temp.path /var/tmp/arangodb3 --log.for> May 02 22:39:24 ubuntu20 arangod[29622]: 2020-05-02T22:39:24Z [29622] INFO detected operating system: Linux version 5.4.0-26-generic (buildd@lcy01-amd6> May 02 22:39:24 ubuntu20 arangod[29622]: 2020-05-02T22:39:24Z [29622] WARNING threads --server.threads (64) is more than eight times the number of co> May 02 22:39:25 ubuntu20 arangod[29622]: 2020-05-02T22:39:25Z [29622] INFO authentication Jwt secret not specified, generating... May 02 22:39:25 ubuntu20 arangod[29622]: 2020-05-02T22:39:25Z [29622] INFO using storage engine rocksdb May 02 22:39:25 ubuntu20 arangod[29622]: 2020-05-02T22:39:25Z [29622] INFO cluster Starting up with role SINGLE May 02 22:39:25 ubuntu20 arangod[29622]: 2020-05-02T22:39:25Z [29622] INFO syscall file-descriptors (nofiles) hard limit is 131072, soft limit is 131> May 02 22:39:25 ubuntu20 arangod[29622]: 2020-05-02T22:39:25Z [29622] INFO authentication Authentication is turned on (system only), authentication f> May 02 22:39:25 ubuntu20 systemd[1]: /lib/systemd/system/arangodb3.service:28: PIDFile= references a path below legacy directory /var/run/, updating /v> May 02 22:39:25 ubuntu20 arangod[29622]: 2020-05-02T22:39:25Z [29622] INFO using endpoint 'http+tcp://127.0.0.1:8529' for non-encrypted requests May 02 22:39:26 ubuntu20 arangod[29622]: 2020-05-02T22:39:26Z [29622] INFO ArangoDB (version 3.4.9 [linux]) is ready for business. Have fun! The status should return “running“. Step 5: Launching ArangoDB shell on Ubuntu To launch ArangoDB shell, run: $ sudo arangosh Please specify a password: _ __ _ _ __ __ _ _ __ __ _ ___ ___| |__ / _` | '__/ _` | '_ \ / _` |/ _ \/ __| '_ \ | (_| | | | (_| | | | | (_| | (_) \__ \ | | | \__,_|_| \__,_|_| |_|\__, |\___/|___/_| |_| |___/ arangosh (ArangoDB 3.4.11 [linux] 64bit, using jemalloc, build tags/v3.4.11-0-gf077c18143, VPack 0.1.33, RocksDB 5.16.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0l 10 Sep 2019) Copyright (c) ArangoDB GmbH Command-line history will be persisted when the shell is exited. Could not connect to endpoint 'tcp://127.0.0.1:8529', database: '_system', username: 'root' Error message: 'Unauthorized' Please note that a new minor version '3.7.11' is available 127.0.0.1:8529@_system> Create a test database. 127.0.0.1:8529@_system> db._createDatabase("testdb"); true Create database user and grant access to the database created above. var users = require("@arangodb/users"); users.save("testuser@localhost", "dbuserpassword"); users.grantDatabase("testuser@example", "testdb"); Check complete user management guide for more information. List databases. 127.0.0.1:8529@_system> db._databases() [ "_system", "testdb" ] You can now connect to the test database using the user testuser@localhost. # arangosh --server.username "testuser@localhost" --server.database testdb Please specify a password: _ __ _ _ __ __ _ _ __ __ _ ___ ___| |__ / _` | '__/ _` | '_ \ / _` |/ _ \/ __| '_ \ | (_| | | | (_| | | | | (_| | (_) \__ \ | | | \__,_|_| \__,_|_| |_|\__, |\___/|___/_| |_| |___/ arangosh (ArangoDB 3.4.2-1 [linux] 64bit, using jemalloc, build tags/v3.4.2.1-0-g91afc6501f, VPack 0.1.33, RocksDB 5.16.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0j 20 Nov 2018) Copyright (c) ArangoDB GmbH Connected to ArangoDB 'http+tcp://127.0.0.1:8529' version: 3.4.2-1 [server], database: 'testdb', username: 'testuser@localhost' Type 'tutorial' for a tutorial or 'help' to see common examples 127.0.0.1:8529@testdb> To drop the database, use: 127.0.0.1:8529@_system> db._dropDatabase("testdb"); true Step 6: Accessing ArangoDB Admin Web Interface

ArangoDB server comes with a built-in web interface for administration. It lets you manage databases, collections, documents, users, graphs, run and explain queries, see server stats and much more. The service is set to listen on localhost address but it can be changed to listen on Local LAN IP. Comment the line endpoint = tcp://127.0.0.1:8529 and add your network IP. $ sudo vim /etc/arangodb3/arangod.conf ... #endpoint = tcp://127.0.0.1:8529 endpoint = tcp://192.168.10.11:8529 Restart ArangoDB service after making this change. sudo systemctl restart arangodb3 If you have UFW firewall enable, ensure port 8529 is allowed in the firewall. sudo ufw allow proto tcp from any to any port 8529 Access the Admin dashboard on http://[serverip or localhost]:8529. By default, authentication is enabled. Provide username and password to authenticate as and select database to connect to on the next screen. The database management web interface should look similar to below. Next Steps Visit the ArangoDB Documentation and Tutorials to learn more about it and how you can do integrations with your Applications. This cheat sheet may also come in handy. Our next guide will cover ArangoDB Cluster setup on Ubuntu 18.04. Until then, check other Database related guides on our website.

[tuckermatthew108]

Overview of SQL Server Replication Components

Modern businesses generate massive amounts of data with the IT departments expected to keep data online and accessible round the clock. This puts intense pressure on the systems to maintain, store, and manage the data. There is also the additional activity of replacing outdated data with new and more agile techniques of which SQL Server Replication is the most suitable to accommodate these demands. What is SQL Server Replication SQL Server Replication is a widely-used technology that helps to copy and distribute data and database from one source to another. After completing this activity the process synchronizes between the databases so that the consistency and the integrity of the data are maintained. This process can be programmed to run constantly or at pre-determined intervals. There are several replication techniques such as one-way, one-to-many, many-to-one, and bi-directional, all of which leads to datasets being in sync with each other.

Types of SQL Server Replication Tools There are three distinct types of SQL server replication tools. # Snapshot Replication This tool simply takes a “snapshot” of the data on one server and moves it to another or another database in the same server. After the first synchronization snapshot, it is possible to periodically refresh data in published tables as per the pre-programmed schedule. This is the easiest of all replication tools but comes with a rider - all data has to be copied each time a table is refreshed. Between scheduled refreshes in snapshot replication, the data on the publisher might be different from the data on the subscriber.   # Transactional Replication In transactional replication, data is copied from the publisher to the subscriber(s) once and then transactions are delivered to the subscriber(s) as and when they take place on the publisher. The initial copy of the data is transported in a similar process as snapshot replication where the SQL server takes a snapshot of data on the publisher and then forwards it to the subscriber(s). But here, as and when database users update, insert or delete records on the publisher, transactions are forwarded to the subscriber(s). # Merge Replication In this technique, data from multiple sources are combined into a single central database. As in transactional replication, merge replication also uses initial synchronization through the snapshot of data on the publisher and moves it to subscribers. But unlike transactional replication, this method permits modifications of the same data on publishers and subscribers, even though subscribers might not be connected to the network. Whenever subscribers connect to the network, merge replication tool will notice it and combine changes from all subscribers and change the existing data on the publisher. Of all SQL server replication tools, merger replication is very useful if there is a need to modify data on remote computers even in the absence of continuous connection to the network. There are some of the important concepts of SQL server replication tools.          

[anotherlauncher]

Performance Scalability Post:

Symptom:

Slowness observed. "node select queries are taking up majority of time resulting in slowness of the application"

in PDOStatement::execute called at /core/lib/Drupal/Core/Database/StatementWrapper.php (145)...Core\Database\StatementWrapper::execute called at /core/lib/Drupal/Core/Database/Connection.php (937)...Database\Connection::query called at /core/lib/Drupal/Core/Database/Driver/pgsql/Connection.php (183)...

These queries are taking longer than 5 seconds.

SELECT "base_table"."vid" AS "vid", "base_table"."nid" AS "nid" + FROM + "node" "base_table" + INNER JOIN "node" "node" ON "node"."nid" = "base_table"."nid" + INNER JOIN "node_field_data" "node_field_data" ON "node_field_data"."nid" = "base_table"."nid" + WHERE ((LOWER("node"."uuid") IN (LOWER('$1')))) AND ("node_field_data"."default_langcode" IN ('$2'))

Analysis:

Repeat the issue locally:

Edit postgresql.conf

log_statement = 'none' log_min_duration_statement = 5000

I also set `log_statement = 'all'` initally but that creates a huge file which you should be careful of not to overload your system. Though it can be helpful to see everything.

Otherwise setting `log_min_duration_statement = 5000` means only log queries that take over 5 seconds.

Then there was some time figuring out how to replicate the issue. But eventually I found it was related to saving an article based on some custom event we have.

The above codeblock points to the execute() function in Driver/pgsql

the execute() function uses the compile() function which uses translateCondition()

in translateCondition() we are seeing the `['where'] = 'LOWER('` . section which we can see in the slow logs. We saw this same function with similar behavior existing in Drupal 8. But for some reason it was not a performance issue then. Now with Drupal 9 the performance impact is now more visible.

public static function translateCondition(&$condition, SelectInterface  $sql_query, $case_sensitive) { if (is_array($condition['value']) && $case_sensitive === FALSE) { $condition['where'] = 'LOWER(' . $sql_query->escapeField($condition['real_field']) . ') ' . $condition['operator'] . ' ('; $condition['where_args'] = []; // Only use the array values in case an associative array is passed as an // argument following similar pattern in // \Drupal\Core\Database\Connection::expandArguments(). $where_prefix = str_replace('.', '_', $condition['real_field']); foreach ($condition['value'] as $key => $value) { $where_id = $where_prefix . $key; $condition['where'] .= 'LOWER(:' . $where_id . '),'; $condition['where_args'][':' . $where_id] = $value; } $condition['where'] = trim($condition['where'], ','); $condition['where'] .= ')'; } parent::translateCondition($condition, $sql_query, $case_sensitive); } }

In general we likely would not want to change the Core execute() function or the translateCondition() function since this is good behavior most of the time.

Though it is odd that "PostgreSQL is a case-sensitive database by default" and yet the default ends up fitting into the condition `$case_sensitive === FALSE`. Potentially core should be updated. But this should likely be a discussion with the maintainers.

['where'] = 'LOWER(' . is part of:

translateCondition() function which is called in core/lib/Drupal/Core/Entity/Query/Sql/pgsql/Condition.php

New Relic:

In New Relic we can see `Postgres node select` was consuming 84% of the db operations.

Cluster Control:

In ClusterControl we can see the top slowest Query was the `SELECT "base_table"."vid" AS` as seen above and it had a near 3 million count in just a few days.

Fix:

Alter the query that uses LOWER for the condition that matches 'LOWER("node"."uuid")

/** * Implements hook_query_TAG_alter(). * * @param \Drupal\Core\Database\Query\AlterableInterface $query */ function core_query_entity_query_node_alter(Drupal\Core\Database\Query\AlterableInterface $query) { $conditions = &$query->conditions(); if (isset($conditions[0]['field']) && $conditions[0]['field'] == 'LOWER("node"."uuid") IN (LOWER(:node_uuid0))') { $conditions[0]['field'] = '"node"."uuid" = :node_uuid0'; } }

Now after the fix, the LOWER still hits all other queries it normally would hit, except for the condition of node uuid

We now see it as :

SELECT "base_table"."vid" AS "vid", "base_table"."nid" AS "nid" + FROM + "node" "base_table" + INNER JOIN "node" "node" ON "node"."nid" = "base_table"."nid" + INNER JOIN "node_field_data" "node_field_data" ON "node_field_data"."nid" = "base_table"."nid" + WHERE ((("node"."uuid" = '$1'))) AND ("node_field_data"."default_langcode" IN ('$2'))

[cloudemind]

AWS Database Migration Service (DMS) Highlights

Có bài viết học luyện thi AWS mới nhất tại https://cloudemind.com/dms/ - Cloudemind.com

AWS Database Migration Service (DMS) Highlights

AWS Database Migration Service (DMS)

Support homogeneous migration such as Oracle to Oracle, as well as heterogeneous migration between database platforms such as Oracle or MSSQL to Aurora.

Support one-time data migration and continuous replication.

Replication between on-premise and on-premise is not supported. One source has to have AWS.

DMS provides end to end view of data replication process

AWS Schema Conversion Tool (SCT)

SCT to use for heterogeneous migration predictable by automatically converting the source database schema to a format compatible with the target database.

SCT also can look into your application code that embedded SQL statement and convert them as part of a database schema conversion project.

SCT Supported

Source DatabaseTarget DatabaseOracle, Azure SQL, MSSQL, PostgreSQL, IBM DB2 LUW, AuroraAmazon Aurora, MySQLOracle DBOracleOracle, Azure SQL, MSSQL, PostgreSQL, IBM DB2 LUW, MySQL, MariaDBPostgreSQLOracle DWH, MSSQL, Teradata, IBM Netezza, Greenplum, HPE VerticaAmazon RedshiftApache CassandraAmazon DynamoDB

Pricing

You only pay for resources used in migration process and additional log storage.

Behind the scene, DMS will use instances for the migration.

References

AWS DMS

AWS SCT

Pricing

Xem thêm: https://cloudemind.com/dms/

[mrkeu]

AWS Certified Solutions Architect – Associate

Giới thiệu

AWS Certification là chứng chỉ được câp bởi Amazon đánh giá mức độ hiểu biết về aws cloud, cụ thể là các dịch vụ của Amazon Web Services (AWS) cũng như việc áp dụng các dịch vụ đó 1 cách hiệu quả vào trong các bài toán thực tế. Bộ chứng chỉ này được chia thành các phần: Cloud Practitioner, Architect, Developer và Operations, ngoài ra có thêm Specialty. Về độ khó thì có 3 mức:

Foundational

Asociate

Professional

Chi tiết về các chứng chỉ các bạn tham khảo ở hình dưới đây: Tham khảo: https://aws.amazon.com/certification/

Kế hoạch học thi

Đầu tiên mình chọn mục tiêu là chứng chỉ: AWS Certified Solutions Architect – Associate. Chứng chỉ này có độ khó trung bình theo mình biết thì ngang tầm 1 năm kinh nghiệm sử dụng AWS, sở dĩ mình chọn chứng chỉ này vì các lý do sau:

Bản thân cũng có 2 năm kinnh nghiệm về Architect nhưng là một cloud khác, ngoài phần infrastrcuture ra thì team mình control toàn bộ, và không có nhiều service support như AWS.

Chứng chỉ với độ khó trung bình trước mắt phù hợp với năng lực bản thân.

Mình chọn khoá học ở https://linuxacademy.com/ và vì chưa từng có kinnh nghiêmh sử dụng AWS nên mình sẽ phải học qua khoá AWS Cloud Practitioner, khoá này mất khoảng 17 giờ học online và thời gian thực tế mình hoàn thành khoá học này mấy khoảng 10 ngày học. Tiếp theo mình sẽ học AWS Certified Solutions Architect – Associate - khoá học này có thời lượng 57 giờ học online, mình lên plan dự kiến hoàn thành trong một tháng. Mình đã hoàn thành khóa học AWS Cloud Practitioner và AWS Certified Solutions Architect – Associate trong khoảng 2 tháng, mỗi ngày 1-2h và 2-3h vào cuối tuần.

Đi thi

Sau 2 tháng học và khoảng 1 tuần để ôn tập kiến thức thì mình đi thi, mình đã book lịch thi từ thời điểm bắt đầu khóa Solutions Architect – Associate. Vì vậy ở thời điểm thi mình còn nhiều kiến thức chưa chuyên sâu, mập mờ giữa các service. Kết quả: 674/1000 => FAIL Đề thi có 65, 720/1000 trở lên là điểm pass. Hơi tiết một chút nhưng dù sao nó cũng năm trong dự đoán của mình (từ 650-700). Vì mình có làm một số đề thi thử nhưng cũng chỉ đạt khoảng 50-70%, trong khi đề thi thật thì dài và khó hơn 1 chút.

Các chủ đề ôn tập

Sau khi sumit bài thi thì bạn biết luôn kết quả là PASS hay FAIL, nhưng để biết điểm thì bạn phải vào trang aws training - nơi mà bạn đăng ký online. OK fail rồi, thế thì để rút kinh nghiệm cho lần thi sau mình đã nhớ lại các câu hỏi làm mình mập mờ giữa 2 đáp án, hay những câu mà mình thấy có nhiều phương án đúng nhưng lựa chọn thằng nào là tốt nhất cho trường hợp tiết kiệm chi phí hay là high availbility nhất v.v.., sau đó mình lục lại tài liệu ở khóa học trên trang https://linuxacademy.com/ bản pdf để lướt qua lại một lần và note lại những điểm cần phải tìm hiểu kỹ hơn cũng như so sánh những service giống nhau. Và dưới đây là các chủ đề mà mình note nhanh lại và sẽ lên kế hoạch để cường hóa kiến thức cho lần thi sắp tới. Mình đã book lịch thi vào đầu tháng 8/2020 nhưng do dịch covid-19 bùng phát tại Đà Nẵng nên mình đã dời sang đầu tháng 10/2020.

DataSync

VPN và Direct Connect

on-premise to Cloud

Snowball, Snowball Edge và Snowmobile (chú ý đến dung lượng)

Dung lượng data chuyển lên Cloud

NAT gatewate và NAT Instance

S3

SNS, SQS, (FIFO, Standard)

So sánh Amazon MQ, SQS, SNS

Data từ on-premeise to Cloud thông qua EFS, File Storage gatewate, Volume storage gatewate.

Storage Gateway

EBS Throughput Optimized HDD , provisined, popular purpose

Tìm hiểu về CloudFoudation và Elast ic Beanstalk

CloudFront hạn chế user theo khu vực địa lý : Geo.. và Geoxy...

Athena và Redshift

Tìm hiểu kỹ về DynamoDB, Aurora

Secure

Khi nào dùng NAT, VPC endpoint, VPC privatelink. Best Pract ice and HA

Kiness và Lampda trigger

DB nào cho phép AZ, hay replicate another region

EFS: performance method: General purpose and Max I/O, thoughput model: Bursting Throughput and Provisioned

I/O liên quan đến RAM hay CPU

Cloudfront và OAI

Encryption: S3, DB, bucket

S3: versioning and presigned URL, MFA Delete and Encrypt, Cross-Origin Resource Sharing (CORS)

Rout 53: Rout e 53 and Healt h Checks, các loại record

ELB

VPC: NAT gatewate and NAT Insrance, Egress-only int ernet gat eways, VPC Endpoint

Net work Access Cont rol List s (NACLs):

Sercurity group

Bast ion Hosts

ECS: EC2 mode và Fargate Mode

Lamda

các loại Instance trong Ec2

Role and policy

DynamoDB Accelerator (DAX)

auto scaling group is configured with default termination policy

EBS cross region or create snapshot

So sanh Dynamo Redshift Aurora

using standard SQL and existing business intelligence tools

Amazon GuardDuty

CloudWatch Event

VNP Cloudhub

Mình sẽ không viết kỹ về các service vì cái này có nhiều blog khác đã viết (tiếng Việt và tiếng Anh). Mình sẽ nêu ra những điểm cần lưu ý khi nói về các service đó, và đưa link tham khảo để các bạn tìm hiểu kỹ hơn.

S3

Khá nhiều câu hỏi liên quan đến service này, một service kinh điển của AWS. Nguồn: https://www.msp360.com/resources/blog/amazon-s3-storage-classes-guide/

Lưu ý

S3 Standard: Chi phí lưu trữ cao nhất, tốc độ truy cập nhanh S3 Standard Infrequent Access: Giống như cái tên của nó, khi chúng ta cần lưu trữ dữ liệu nhưng không truy cập thường xuyên, tuy nhiên khi cần truy cập phải được phản hồi ngay lập tức. S3 Intelligent Tiering: có thể nói đây không phải là một storage class, vậy khi nào thì bạn chọn loại này - khi chúng ta không dự đoán được tấn suất truy cập cho những dữ liệu này. AWS sẽ monitor và sẽ chuyển qua chuyển lại giữa các class tùy theo tần suất truy cập đến dữ liệu để đảm bảo tiết kiệm chi phí cho bạn. Amazon Glacier: Đây là class dành cho dữ liệu truy cập không thường xuyên và chi phí rẻ, và tùy thuộc vào thời gian bạn muốn lấy giữ liệu mà nó chia làm các retrieval options như sau (cái này hay hỏi bài thi):

expedite retrieval - 1-5 minutes

standard - 3-5 hours

bulk - 5-12 hours

Amazon Glacier Deep Archive: Đây là class có chi phí lưu trữ rẻ nhất, cùng với đó là thời gian để truy xuất dữ liệu cũng chậm 12 - 48 hours.

Câu hỏi mẫu

Tài liệu tham khảo

https://aws.amazon.com/s3/?nc1=h_ls https://aws.amazon.com/s3/storage-classes/?nc=sn&loc=3 https://www.msp360.com/resources/blog/amazon-s3-storage-classes-guide/

EC2

Một số lưu ý để ước tính chi phí Amazon EC2 là: Operating systems, Clock hours of server time, Pricing Model, Instance type and Number of instances

Pricing models for Amazon EC2:

On-Demand Instances

Reserved Instances

Spot Instances

Dedicated Hosts

On-Demand Instances

Không cần trả trước, bạn có thể tăng giảm khả năng compute để đáp ứng như cầu của bạn và bạn sẽ chi trả cho những gì bạn dùng. Phù hợp với các công việc thử nghiệm trong thời gian ngắn và không đoán trước được. Vd cho trường hợp sử dụng On-Demand Instances là khi bạn scale OUT instances để phục vụ lượng request lớn tăng đột ngột ramdom trong ngày mà bạn không đoán trước được thời gian nào.

Reserved Instances

Amazon EC2 Reserved cung cấp cho bạn mức chiếc khấu lên đến 75% so với On-Demand Instance. Khi chúng ta có một ứng dụng chạy dài hạn 1-3 năm và chúng ta có thể dự đoán được khả năng tính toán (compute) của instance thì loại instance này là phù hợp để tiết kiệm chi phí.

Spot Instances

Amazon EC2 Spot instances allow you to request spare Amazon EC2 computing capacity for up to 90% off the On-Demand price.

Amazon EC2 Spot Instances cho phép bạn tận dụng EC2 capacity chưa được sử dụng trên AWS cloud. Nói cách khác, là có thể sử dụng hàng ngon với giá rẻ. Khi sử dụng loại instance này, bạn phải chấp nhận việc các xử lý có thể bị gián đoạn (interrupted). và EC2 bị terminated. Tham khảo thêm tại: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-leveraging-ec2-spot-instances/when-to-use-spot-instances.html

Dedicated Hosts

Giống như cái tên, Nó là một physical EC2 server dành riêng cho bạn. Dedicated Hosts giúp bạn giảm chi phí bằng cách cho phép bạn sử dụng các software licenses như Windows server, SQL server v.v... đang có. Tìm hiểu thêm tại: https://aws.amazon.com/ec2/dedicated-hosts/

Tài liệu tham khảo

https://aws.amazon.com/ec2/pricing/?nc1=h_ls https://cuongquach.com/tim-hieu-ve-amazon-ec2-amazon-elastic-compute-cloud.html

Câu hỏi mẫu

Migrating Data to the Cloud

Trong rất nhiều tình huống được đưa ra, đa phần sẽ có các caai hỏi liên quan đến việc migrate data từ on-premise lên AWS cloud. Mình sẽ tổng hợp lại một số cách thức và chỉ ra điểm lưu ý ở các service này.

Online data transferAWS Virtual Private Network

Thiết lập nhanh

Chi phí thấp

Không đảm bảo băng thông ổn định, và mức độ bảo mật không cao

AWS Database Migration Service

Database của bạn vẫn hoạt động bình thường trong lúc tranfer

Chuyển và phân tích data nhạy cảm

Đơn giản, tự động và tăng tốc

AWS Direct Connect

Thiết lâp kết nối private chuyên dụng với cáp quang 1G hoặc 10G

Sử dụng khi chúng ta cần 1 kênh nhanh, tin cậy và bảo mật để truyền dữ liệu

Tốn chi phí và mất nhiều thời gian set up hơn so với VPN

Với chi phí và thời gian cần thiết để thiết lập, nó không phải là một tùy chọn truyền dữ liệu lý tưởng nếu bạn chỉ cần nó để thực hiện di chuyển một lần.

AWS S3 Transfer Acceleration

Nếu bạn đang chuyển dữ liệu của mình sang nền tảng lưu trữ AWS phổ biến S3 ở một khoảng cách xa, thì AWS S3 Transfer Acceleration giúp bạn thực hiện nhanh hơn: trung bình nhanh hơn 171%, theo AWS.

Nhanh chóng, an toàn thông qua public internet

Tiết kiệm thời gian khi chuyển data từ các location khác nhau

Dữ liệu được truyền thông qua Amazon CloudFront - tối đa hóa băng thông

AWS DataSync

Nguồn: https://aws.amazon.com/datasync/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc

Offline data transferAWS Snowball

AWS Snowball là em bé của gia đình Snowball: một thiết bị siêu bền, có kích thước bằng vali, có thể được tải lên tới 80TB dữ liệu.

50 - 80TB

Mã hóa 256 bit

Encryption keys được quản lý thông qua AWS Key Management Service (KMS)

AWS Snowball Edge

Có kích thước lớn lơn Snowball với khả năng tải lên tới 100TB. Snowball Edge cũng có thể chạy các ứng dụng dựa trên Lambda và EC2, ngay cả khi không có kết nối mạng.

Điều này làm cho nó lý tưởng cho các trường hợp sử dụng cần xử lý hoặc phân tích cục bộ trước khi dữ liệu được đưa lên AWS Cloud. Giá của nó dĩ nhiên sẽ đắt hơn Snowball.

AWS Snowmobile

AWS Snowmobile là một giải pháp di chuyển dữ liệu quy mô exabyte, đóng gói tương đương với 1.250 thiết bị AWS Snowball vào container vận chuyển dài 45ft.

AWS Snowmobile có thể vận chuyển tới 100PB dữ liệu trong một chuyến đi, với chi phí chỉ bằng 1/5 chi phí chuyển dữ liệu qua kết nối internet tốc độ cao. Snowmobile không chỉ là cách nhanh nhất và rẻ nhất để chuyển lượng dữ liệu khổng lồ lên cloud, nó còn có tính bảo mật cao. Nhân viên an ninh chuyên dụng, theo dõi GPS, giám sát báo động và giám sát video 24/7 phối hợp với nhau để giữ an toàn cho dữ liệu của bạn trong suốt hành trình. Dữ liệu được mã hóa bằng các khóa mã hóa 256 bit.

Tài liệu tham khảo

https://www.jeffersonfrank.com/aws-blog/aws-data-transfer-costs/#AWSDC https://www.slideshare.net/AmazonWebServices/migrating-data-to-the-cloud-exploring-your-options-from-aws-stg205r1-aws-reinvent-2018

Câu hỏi mẫu

Lời kết

Ở phần 1 này mình tạm thời memo lại bấy nhiêu thôi, mình không chắc những cái mình memo ở đây đầy đủ và chính xác, quan điểm của mình là đưa ra các chủ đề có trong bài thi và những điểm mập mờ cần làm rõ để phân biệt các service và khi nào nên sử dụng cái nào. Lượng kiến thức dành cho bài thi này không đến mức chuyên sâu nhưng trãi đều các nội dung. Nắm vững khái niệm, khi nào sử dụng và một số lưu ý để so sánh giữa các service giống nhau sẽ là chìa khóa để bạn chinh phục bài test này. Trên cả việc pass chứng chỉ hay không pass chứng chỉ đó là kiến thức bạn nắm được, hiểu được để có thể vận dụng trong thực tế, vì vậy sẽ hoàn hảo hơn khi chúng ta tìm hiểu các service này ở một mức độ tương đối chứ không qua loa. Nếu các bạn có gặp các câu hỏi nào hay service này cần thảo luận, hãy để lại comment và chúng ta cùng thảo luận nhé.