67 ShardingSphere 分表分庫

1,什么是垂直/水平拆分模式

2,基于客戶端與服務(wù)器端實(shí)現(xiàn)分表分庫的區(qū)別

3,單表達(dá)到多大的量開始分表分庫
4,數(shù)據(jù)庫分表分庫策略有哪些
5, shardingSphere 實(shí)戰(zhàn)分庫分表
6,為什么不推薦使用mycat 實(shí)現(xiàn)分庫分表
7,分表分庫后查詢存在哪些優(yōu)缺點(diǎn)
8,分表分庫后如何實(shí)現(xiàn)分頁查詢
9,分表分庫后如何實(shí)現(xiàn)排序查詢;

垂直/水平拆分模式
1,數(shù)據(jù)庫拆分主要指分庫分表,其目的主要是分散數(shù)據(jù)庫壓力,達(dá)到橫向發(fā)展,滿足均衡訪問等。
2, 數(shù)據(jù)庫拆分主要有2種形式: 垂直拆分和水平拆分。
A 垂直拆分:
將不同的業(yè)務(wù)功能相關(guān)的表放到不同的數(shù)據(jù)庫中,也就是類似于 微服務(wù)架構(gòu)中 會(huì)員數(shù)據(jù)庫/訂單數(shù)據(jù)庫/支付數(shù)據(jù)庫
B 水平拆分:
當(dāng)一張表的業(yè)務(wù)量行數(shù)如果超過 500 行,分頁/排序效率還是非常低,可以對(duì)同一個(gè)表數(shù)據(jù)實(shí)現(xiàn)拆分放到多個(gè)不同的數(shù)據(jù)庫表中存放。

Mayikt_memberdb01
Mayikt_user
Mayikt_memberdb02
Mayikt_user

基于客戶端與服務(wù)器端實(shí)現(xiàn)分表分庫的區(qū)別:

1,基于服務(wù)器端mycat 實(shí)現(xiàn)數(shù)據(jù)庫代理
優(yōu)點(diǎn): 能夠保證數(shù)據(jù)庫 的安全性
缺點(diǎn): 效率比較低

image.png

2,基于客戶端Shardingjdbc 實(shí)現(xiàn)數(shù)據(jù)庫代理:
優(yōu)點(diǎn): 效率比較高
缺點(diǎn): 不能夠保證數(shù)據(jù)庫的安全性,內(nèi)存溢出;


image.png

單表達(dá)到多大量開始分表分庫:
【推薦】單表行數(shù)超過 500 萬行或者單表容量超過 2GB,才推薦進(jìn)行分庫分表。
說明:如果預(yù)計(jì)三年后的數(shù)據(jù)量根本達(dá)不到這個(gè)級(jí)別,請(qǐng)不要在創(chuàng)建表時(shí)就分庫分表。
摘自:阿里巴巴java開發(fā)手冊(cè)

數(shù)據(jù)庫分表分庫策略有哪些:

取余/取模 均勻存放數(shù)據(jù) 缺點(diǎn): 不夠擴(kuò)容。
按照范圍分片: 1-500萬 501萬-10000萬;
按照日期進(jìn)行分片 日志,訂單信息 同居
按照枚舉值分片;
二進(jìn)制取模范圍分片:
一致性hash 分片,類似于HashMap 缺點(diǎn) 數(shù)據(jù)存放不均勻。
按照目前字段前綴指定的進(jìn)行分區(qū) mayikt wuhan
按照前綴ASCII 碼和值進(jìn)行取模范圍分片。

主流分片算法: 取余/取模 日期 一致性hash 分片
根據(jù)user_id
Userid=1%2=1
Userid=2%2=0
Userid=3%2=1
Userid=4%2=0

不能實(shí)現(xiàn)擴(kuò)容。

ShardingSphere 實(shí)戰(zhàn)分表分庫
ShardingSphere 是一套開源的分布式數(shù)據(jù)庫中間件解決方案組成的生態(tài)圈,它由Sharding-JDBC, Sharding-PROXY 和 Sahriding 這三款 相互獨(dú)立的產(chǎn)品組成。
他們均提供標(biāo)準(zhǔn)化的數(shù)據(jù)分片,分布式事務(wù) 和數(shù)據(jù)庫治理功能,可適用于java同構(gòu),已購語言,云原生等各種多樣化的應(yīng)用場(chǎng)景。

image.png

Sharding_JDBC 可以實(shí)現(xiàn)
分庫 & 分表

  • 讀寫分離
  • 分片策略定制化
  • 無中心化分布式主鍵

SharingdSphere 環(huán)境搭建:
數(shù)據(jù)庫準(zhǔn)備;

DROP TABLE IF EXISTS `mayikt_user_0`;
CREATE TABLE `mayikt_user_0` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `age` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;

-- ----------------------------
-- Records of mayikt_user_0
-- ----------------------------
INSERT INTO `mayikt_user_0` VALUES ('2', 'mayikt2', '2');
INSERT INTO `mayikt_user_0` VALUES ('4', 'mayikt4', '4');
INSERT INTO `mayikt_user_0` VALUES ('6', 'mayikt6', '6');
INSERT INTO `mayikt_user_0` VALUES ('8', 'mayikt8', '8');

-- ----------------------------
-- Table structure for mayikt_user_1
-- ----------------------------
DROP TABLE IF EXISTS `mayikt_user_1`;
CREATE TABLE `mayikt_user_1` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `age` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8;

-- ----------------------------
-- Records of mayikt_user_1
-- ----------------------------
INSERT INTO `mayikt_user_1` VALUES ('1', 'mayikt1', '1');
INSERT INTO `mayikt_user_1` VALUES ('3', 'mayikt3', '3');
INSERT INTO `mayikt_user_1` VALUES ('5', 'mayikt5', '5');
INSERT INTO `mayikt_user_1` VALUES ('7', 'mayikt7', '7');
INSERT INTO `mayikt_user_1` VALUES ('9', 'mayikt9', '9');



Maven 依賴:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.5.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.taotao</groupId>
    <artifactId>sharding-jdbc</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>sharding-jdbc</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
        </dependency>

        <dependency>
            <groupId>com.baomidou</groupId>
            <artifactId>mybatis-plus-boot-starter</artifactId>
            <version>3.3.2</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.20</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.62</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>

        <dependency>
            <groupId>io.shardingsphere</groupId>
            <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
            <version>3.1.0</version>
        </dependency>
        <dependency>
            <groupId>io.shardingsphere</groupId>
            <artifactId>sharding-jdbc-spring-namespace</artifactId>
            <version>3.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>


application.yml 配置文件

# 數(shù)據(jù)源 mayiktdb
sharding:
  jdbc:
    datasource:
      names: shard1,shard2
      # 第一個(gè)數(shù)據(jù)庫
      shard1:
        type: com.zaxxer.hikari.HikariDataSource
        driver-class-name: com.mysql.cj.jdbc.Driver
        jdbc-url: jdbc:mysql://127.0.0.1:3306/mayikt-member?useUnicode=true&characterEncoding=UTF-8&serverTimezone=GMT%2B8
        username: root
        password: root
      shard2:
        type: com.zaxxer.hikari.HikariDataSource
        driver-class-name: com.mysql.cj.jdbc.Driver
        jdbc-url: jdbc:mysql://127.0.0.1:3306/mayikt?useUnicode=true&characterEncoding=UTF-8&serverTimezone=GMT%2B8
        username: root
        password: root
    # 水平拆分的數(shù)據(jù)庫(表) 配置分庫 + 分表策略 行表達(dá)式分片策略
    config:
      sharding:
        #默認(rèn)數(shù)據(jù)源(未設(shè)置分表策略的表將默認(rèn)用此數(shù)據(jù)源)
        default-database-name: shard1
        #默認(rèn)分庫策略
        default-database-strategy:
           standard:
             sharding-column: id
             precise-algorithm-class-name: com.taotao.rfspringboot.config.DataBasePreciseRule
        tables:
          mayikt_user:  ##虛擬表名稱  mayikt_user_0 mayikt_user_1
            actual-data-nodes: shard$->{1..2}.mayikt_user_$->{0..2}  # 沒有帶上 分片字段
            table-strategy:
              standard:  ##SELECT * FROM  mayikt_user_1 where id =1
                precise-algorithm-class-name: com.taotao.rfspringboot.config.MayiktRangeShardingAlgorithm
                sharding-column: id
      # 打印執(zhí)行的數(shù)據(jù)庫
      props:
        sql:
          show: true

# 打印執(zhí)行的sql語句
spring:
  main:
    allow-bean-definition-overriding: true
server:
  port: 8081

mybatis-plus:
  # 如果是放在src/main/java目錄下 classpath:/com/yourpackage/*/com.exchange.mapper/*Mapper.com.exchange.mapper
  # 如果是放在resource目錄 classpath:/com.exchange.mapper/*Mapper.com.exchange.mapper
  mapper-locations: classpath:/mapper/*.xml
  #實(shí)體掃描,多個(gè)package用逗號(hào)或者分號(hào)分隔
  typeAliasesPackage: com.taotao.rfspringboot.entity
  global-config:
    #主鍵類型  0:"數(shù)據(jù)庫ID自增", 1:"用戶輸入ID",2:"全局唯一ID (數(shù)字類型唯一ID)", 3:"全局唯一ID UUID";
    #若采用雪花算法生成id,需要在生成的實(shí)體類中將id的type = IdType.AUTO去掉
    id-type: 0
    #字段策略 0:"忽略判斷",1:"非 NULL 判斷"),2:"非空判斷"
    field-strategy: 2
    #駝峰下劃線轉(zhuǎn)換
    db-column-underline: true
    #刷新mapper 調(diào)試神器
    #refresh-mapper: true
    #數(shù)據(jù)庫大寫下劃線轉(zhuǎn)換
    #capital-mode: true
    # Sequence序列接口實(shí)現(xiàn)類配置
    #邏輯刪除配置(下面3個(gè)配置)
    logic-delete-value: Y
    logic-not-delete-value: N
    #sql-injector: com.nky.pork.quality.standard.conf.MybatisPlusConfig
    configuration:
      #配置返回?cái)?shù)據(jù)庫(column下劃線命名&&返回java實(shí)體是駝峰命名),自動(dòng)匹配無需as(沒開啟這個(gè),SQL需要寫as: select user_id as userId)
      map-underscore-to-camel-case: true
      cache-enabled: false
      #配置JdbcTypeForNull, oracle數(shù)據(jù)庫必須配置
      jdbc-type-for-null: 'null'

entity:

package com.taotao.shardingjdbc.entity;

import com.baomidou.mybatisplus.annotation.TableName;
import lombok.Data;

/**
 *@author tom
 *Date  2020/11/6 0006 8:27
 *
 */
@Data
@TableName("mayikt_user")
public class Mayikt_user {
    private Integer id;
    private String name;
    private Integer age;

    public Mayikt_user(Integer id, String name, Integer age) {
        this.id = id;
        this.name = name;
        this.age = age;
    }
}



數(shù)據(jù)庫訪問:

package com.taotao.shardingjdbc.mapper;

import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.taotao.shardingjdbc.entity.MayiktUser;
import org.apache.ibatis.annotations.Select;

import java.util.List;

/**
 *@author tom
 *Date  2020/11/6 0006 8:29
 *
 */
public interface MayiktUserMapper  extends BaseMapper<MayiktUser> {
    /**
     * 查詢所有
     *
     * @return
     */
    @Select("SELECT * FROM mayikt_user")
    List<MayiktUser> userList();

    /**
     * 分頁查詢
     *
     * @return
     */
    @Select("SELECT * FROM mayikt_user limit 0,2")
    List<MayiktUser> userListPage();

    /**
     * 排序
     *
     * @return
     */
    @Select("SELECT * FROM mayikt_user order by id desc ")
    List<MayiktUser> userOrderBy();

    /**
     * get by id
     *
     * @return
     */
    @Select("SELECT * FROM mayikt_user where id =#{id} ")
    List<MayiktUser> getByUserId(Long id);
}



Controller:

package com.taotao.shardingjdbc.controller;

import com.taotao.shardingjdbc.entity.MayiktUser;
import com.taotao.shardingjdbc.mapper.MayiktUserMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

/**
 *@author tom
 *Date  2020/11/6 0006 8:41
 *
 */
@RestController
public class UserController {
    @Autowired
    private MayiktUserMapper mayiktUserMapper;


    @RequestMapping("/insertUser")
    public  String insertUser(){
        for (int i = 0; i <10 ; i++) {
            MayiktUser mayiktUser=new MayiktUser(i,"mykt"+i,i);
           try {
                mayiktUserMapper.insert(mayiktUser);
           }catch (Exception e){
               e.printStackTrace();
           }
        }
             return  "success";
    }

/**
 * 查詢所有
 */

@RequestMapping("/userList")
public List<MayiktUser> userList(){
    return  mayiktUserMapper.userList();
}

/**
 * 分頁查詢
 */
@RequestMapping("/userListPage")
public  List<MayiktUser> userListpage(){
    return  mayiktUserMapper.userListPage();
}
/**
 * 排序
 */

@RequestMapping("/userOrderBy")
    public  List<MayiktUser> userOrderBy(){
    return  mayiktUserMapper.userOrderBy();
}

@RequestMapping("/getByUserId")
    public  List<MayiktUser> getByUserId(Long id){
    return  mayiktUserMapper.getByUserId(id);
}
}






package com.taotao.rfspringboot.config;

import io.shardingsphere.api.algorithm.sharding.PreciseShardingValue;
import io.shardingsphere.api.algorithm.sharding.standard.PreciseShardingAlgorithm;

import java.util.Collection;

/**
 * 數(shù)據(jù)庫分片規(guī)則
 *
 * @author Mos
 */
public class DataBasePreciseRule  implements PreciseShardingAlgorithm<Long> {


  @Override
  public String doSharding(Collection<String> databaseNames, PreciseShardingValue<Long> shardingValue) {

    /**
     * databaseNames 所有分片庫的集合
     * shardingValue 為分片屬性,其中 logicTableName 為邏輯表,columnName 分片?。ㄗ侄危瑅alue 為從 SQL 中解析出的分片健的值
     */
    for (String databaseName : databaseNames) {

      String value = shardingValue.getValue() % databaseNames.size()+1 + "";
      if (databaseName.endsWith(value)) {
        System.out.println(databaseName+"&&&"+shardingValue.getValue() +"*********"+databaseNames.size());
        return databaseName;
      }
    }
    throw new IllegalArgumentException();


  }
}

package com.taotao.rfspringboot.config;

import io.shardingsphere.api.algorithm.sharding.PreciseShardingValue;
import io.shardingsphere.api.algorithm.sharding.standard.PreciseShardingAlgorithm;
import lombok.extern.slf4j.Slf4j;

import java.util.Collection;

@Slf4j
public class MayiktRangeShardingAlgorithm implements PreciseShardingAlgorithm<Long> {


    @Override
    public String doSharding(Collection<String> databaseNames, PreciseShardingValue<Long> shardingValue) {

        /**
         * databaseNames 所有分片庫的集合
         * shardingValue 為分片屬性,其中 logicTableName 為邏輯表,columnName 分片?。ㄗ侄危?,value 為從 SQL 中解析出的分片健的值
         */
      for (String databaseName : databaseNames) {
            String value = shardingValue.getValue() % databaseNames.size() + "";
            if (databaseName.endsWith(value)) {
                return databaseName;
            }
        }
        throw new IllegalArgumentException();


    }



}

歸并數(shù)據(jù)原理

1.SELECT * FROM mayikt_user
底層發(fā)出多條語句查詢每張表數(shù)據(jù),在本地實(shí)現(xiàn)數(shù)據(jù)合并

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡(jiǎn)書系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

友情鏈接更多精彩內(nèi)容