系列

• Sharding-JDBC 核心組件介紹
• Sharding-JDBC 配置分析
• Sharding-JDBC 執(zhí)行整體流程
• Sharding-JDBC 分庫配置解析過程
• Sharding-JDBC 分表配置解析過程
• Sharding-JDBC 分庫分表配置解析過程

開篇

• 案例代碼參考shardingsphere-example，版本基于4.0.0的tag版本。
• 這篇文章用來分析分表配置的整個(gè)解析過程，根據(jù)shardingRuleConfig來生成shardingRule對(duì)象。

分庫分表策略

支持策略

public enum ShardingType {
    // 分庫
    SHARDING_DATABASES,
    // 分表
    SHARDING_TABLES,
    // 分庫分表
    SHARDING_DATABASES_AND_TABLES,
    // 主從
    MASTER_SLAVE,
    // 分片下的主從
    SHARDING_MASTER_SLAVE,
    // 加密
    ENCRYPT
}

• 支持分庫、分表、分庫分表、主從、分片下的主從、加密等共6種場(chǎng)景。

精確值的分庫分表策略

public class DataSourceFactory {
    // 精確值的分庫分表策略
    public static DataSource newInstance(final ShardingType shardingType) throws SQLException {
        switch (shardingType) {
            // 分庫
            case SHARDING_DATABASES:
                return new ShardingDatabasesConfigurationPrecise().getDataSource();
            // 分表
            case SHARDING_TABLES:
                return new ShardingTablesConfigurationPrecise().getDataSource();
            // 分庫分表
            case SHARDING_DATABASES_AND_TABLES:
                return new ShardingDatabasesAndTablesConfigurationPrecise().getDataSource();
            // 主從
            case MASTER_SLAVE:
                return new MasterSlaveConfiguration().getDataSource();
            // 分片場(chǎng)景下的主從
            case SHARDING_MASTER_SLAVE:
                return new ShardingMasterSlaveConfigurationPrecise().getDataSource();
            default:
                throw new UnsupportedOperationException(shardingType.name());
        }
    }
}

• 按照精確值進(jìn)行分庫分表策略配置對(duì)象。
• 支持分庫、分表、分庫分表、主從、分片的主從等5類場(chǎng)景。

范圍值的分庫分表策略

public class RangeDataSourceFactory {
    // 范圍值的分庫分表策略
    public static DataSource newInstance(final ShardingType shardingType) throws SQLException {
        switch (shardingType) {
            // 分庫
            case SHARDING_DATABASES:
                return new ShardingDatabasesConfigurationRange().getDataSource();
            // 分表
            case SHARDING_TABLES:
                return new ShardingTablesConfigurationRange().getDataSource();
            // 分庫分表
            case SHARDING_DATABASES_AND_TABLES:
                return new ShardingDatabasesAndTablesConfigurationRange().getDataSource();
            // 主從
            case MASTER_SLAVE:
                return new MasterSlaveConfiguration().getDataSource();
            // 分片下的主從關(guān)系
            case SHARDING_MASTER_SLAVE:
                return new ShardingMasterSlaveConfigurationRange().getDataSource();
            default:
                throw new UnsupportedOperationException(shardingType.name());
        }
    }
}

• 按照范圍值進(jìn)行分庫分表策略配置對(duì)象。
• 支持分庫、分表、分庫分表、主從、分片的主從等5類場(chǎng)景。

精確值分表配置

配置的源碼實(shí)現(xiàn)

public final class ShardingTablesConfigurationPrecise implements ExampleConfiguration {
    
    @Override
    public DataSource getDataSource() throws SQLException {
        // 1、創(chuàng)建ShardingRuleConfiguration對(duì)象
        ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
        // 2、添加order表的分表配置規(guī)則
        shardingRuleConfig.getTableRuleConfigs().add(getOrderTableRuleConfiguration());
        // 3、添加order_item表的分表配置規(guī)則
        shardingRuleConfig.getTableRuleConfigs().add(getOrderItemTableRuleConfiguration());
        shardingRuleConfig.getBindingTableGroups().add("t_order, t_order_item");
        shardingRuleConfig.getBroadcastTables().add("t_address");
        // 4、設(shè)置分表策略
        shardingRuleConfig.setDefaultTableShardingStrategyConfig(new StandardShardingStrategyConfiguration("order_id", new PreciseModuloShardingTableAlgorithm()));
        // 5、創(chuàng)建DataSource返回
        return ShardingDataSourceFactory.createDataSource(createDataSourceMap(), shardingRuleConfig, new Properties());
    }
    
    private static TableRuleConfiguration getOrderTableRuleConfiguration() {
        TableRuleConfiguration result = new TableRuleConfiguration("t_order", "demo_ds.t_order_${[0, 1]}");
        result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_id", getProperties()));
        return result;
    }
    
    private static TableRuleConfiguration getOrderItemTableRuleConfiguration() {
        TableRuleConfiguration result = new TableRuleConfiguration("t_order_item", "demo_ds.t_order_item_${[0, 1]}");
        result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_item_id", getProperties()));
        return result;
    }
    
    private static Map<String, DataSource> createDataSourceMap() {
        Map<String, DataSource> result = new HashMap<>();
        result.put("demo_ds", DataSourceUtil.createDataSource("demo_ds"));
        return result;
    }
    
    private static Properties getProperties() {
        Properties result = new Properties();
        result.setProperty("worker.id", "123");
        return result;
    }
}

• 配置一個(gè)dataSource為demo_ds。
• 配置分表的邏輯StandardShardingStrategyConfiguration，分表的列為order_id。
• t_order表的分表規(guī)則為demo_ds.t_order_${[0, 1]}。
• t_order_item表的分表規(guī)則為demo_ds.t_order_item_${[0, 1]}。

配置的json格式

{
    "bindingTableGroups":[ "t_order, t_order_item"],
    "broadcastTables":["t_address"],
    "defaultTableShardingStrategyConfig":{
        "preciseShardingAlgorithm":{
        },
        "shardingColumn":"order_id"
    },
    "masterSlaveRuleConfigs":[
    ],
    "tableRuleConfigs":[
        {
            "actualDataNodes":"demo_ds.t_order_${[0, 1]}",
            "keyGeneratorConfig":{
                "column":"order_id",
                "properties":{
                    "worker.id":"123"
                },
                "type":"SNOWFLAKE"
            },
            "logicTable":"t_order"
        },
        {
            "actualDataNodes":"demo_ds.t_order_item_${[0, 1]}",
            "keyGeneratorConfig":{
                "column":"order_item_id",
                "properties":{
                    "worker.id":"123"
                },
                "type":"SNOWFLAKE"
            },
            "logicTable":"t_order_item"
        }]
}

• 包含t_order和t_order_item兩個(gè)tableRuleConfig。
• 邏輯表t_order的實(shí)際數(shù)據(jù)節(jié)點(diǎn)為demo_ds.t_order_${[0, 1]}。
• 邏輯表t_order_item的實(shí)際節(jié)點(diǎn)為demo_ds.t_order_item_${[0, 1]}。

配置規(guī)則的解析

public final class ShardingDataSourceFactory {
    
    public static DataSource createDataSource(
            final Map<String, DataSource> dataSourceMap, final ShardingRuleConfiguration shardingRuleConfig, final Properties props) throws SQLException {
        return new ShardingDataSource(dataSourceMap, new ShardingRule(shardingRuleConfig, dataSourceMap.keySet()), props);
    }
}

• 基于shardingRuleConfig和dataSourceMap生成ShardingRule對(duì)象。

public class ShardingRule implements BaseRule {

    private final ShardingRuleConfiguration ruleConfiguration;
    private final ShardingDataSourceNames shardingDataSourceNames;
    private final Collection<TableRule> tableRules;
    private final Collection<BindingTableRule> bindingTableRules;
    private final Collection<String> broadcastTables;
    private final ShardingStrategy defaultDatabaseShardingStrategy;
    private final ShardingStrategy defaultTableShardingStrategy;
    private final ShardingKeyGenerator defaultShardingKeyGenerator;
    private final Collection<MasterSlaveRule> masterSlaveRules;
    private final EncryptRule encryptRule;

    public ShardingRule(final ShardingRuleConfiguration shardingRuleConfig, final Collection<String> dataSourceNames) {
        Preconditions.checkArgument(null != shardingRuleConfig, "ShardingRuleConfig cannot be null.");
        Preconditions.checkArgument(null != dataSourceNames && !dataSourceNames.isEmpty(), "Data sources cannot be empty.");
        // 1、分片規(guī)則
        this.ruleConfiguration = shardingRuleConfig;
        // 2、解析生成數(shù)據(jù)源名字
        shardingDataSourceNames = new ShardingDataSourceNames(shardingRuleConfig, dataSourceNames);
        // 3、生成TableRule
        tableRules = createTableRules(shardingRuleConfig);
        // 4、生成broadcastTables
        broadcastTables = shardingRuleConfig.getBroadcastTables();
        // 5、生成BindingTableRule
        bindingTableRules = createBindingTableRules(shardingRuleConfig.getBindingTableGroups());
        // 6、生成分庫策略defaultDatabaseShardingStrategy
        defaultDatabaseShardingStrategy = createDefaultShardingStrategy(shardingRuleConfig.getDefaultDatabaseShardingStrategyConfig());
        // 7、生成分表策略defaultTableShardingStrategy
        defaultTableShardingStrategy = createDefaultShardingStrategy(shardingRuleConfig.getDefaultTableShardingStrategyConfig());
        // 8、生成分片key的生成器defaultShardingKeyGenerator
        defaultShardingKeyGenerator = createDefaultKeyGenerator(shardingRuleConfig.getDefaultKeyGeneratorConfig());
        // 9、生成MasterSlaveRule
        masterSlaveRules = createMasterSlaveRules(shardingRuleConfig.getMasterSlaveRuleConfigs());
        // 10、生成EncryptRule
        encryptRule = createEncryptRule(shardingRuleConfig.getEncryptRuleConfig());
    }
}

• 整體邏輯的注釋如上所示，核心關(guān)注tableRules和defaultTableShardingStrategy。
• ShardingDataSourceNames記錄分庫的數(shù)據(jù)源。
• ShardingRule包含tableRules，TableRule記錄分庫后的表規(guī)則。
• defaultTableShardingStrategy表示table的分表策略。

public final class ShardingDataSourceNames {
    
    private final ShardingRuleConfiguration shardingRuleConfig;
    
    @Getter
    private final Collection<String> dataSourceNames;
    
    public ShardingDataSourceNames(final ShardingRuleConfiguration shardingRuleConfig, final Collection<String> rawDataSourceNames) {
        Preconditions.checkArgument(null != shardingRuleConfig, "can not construct ShardingDataSourceNames with null ShardingRuleConfig");
        this.shardingRuleConfig = shardingRuleConfig;
        dataSourceNames = getAllDataSourceNames(rawDataSourceNames);
    }
    
    private Collection<String> getAllDataSourceNames(final Collection<String> dataSourceNames) {
        Collection<String> result = new LinkedHashSet<>(dataSourceNames);
        for (MasterSlaveRuleConfiguration each : shardingRuleConfig.getMasterSlaveRuleConfigs()) {
            result.remove(each.getMasterDataSourceName());
            result.removeAll(each.getSlaveDataSourceNames());
            result.add(each.getName());
        }
        return result;
    }
}

• ShardingDataSourceNames的生成規(guī)則為包含傳入的dataSourceNames，然后排除MasterSlaveRuleConfiguration的主從的DataSourceNames并添加對(duì)應(yīng)的name。
• 在分表的場(chǎng)景下dataSourceNames為["demo_ds"]。

public class ShardingRule implements BaseRule {

    private Collection<TableRule> createTableRules(final ShardingRuleConfiguration shardingRuleConfig) {
        // 1、getTableRuleConfigs的結(jié)果可以參考分庫的json格式配置內(nèi)容
        Collection<TableRuleConfiguration> tableRuleConfigurations = shardingRuleConfig.getTableRuleConfigs();
        // 每個(gè)表對(duì)應(yīng)一個(gè)TableRule對(duì)象
        Collection<TableRule> result = new ArrayList<>(tableRuleConfigurations.size());
        // 2、遍歷tableRuleConfigurations生成TableRule
        for (TableRuleConfiguration each : tableRuleConfigurations) {
            // 3、每個(gè)table對(duì)應(yīng)一個(gè)TableRuleConfiguration，每個(gè)TableRuleConfiguration生成TableRule對(duì)象
            result.add(new TableRule(each, shardingDataSourceNames, getDefaultGenerateKeyColumn(shardingRuleConfig)));
        }
        return result;
    }

    private String getDefaultGenerateKeyColumn(final ShardingRuleConfiguration shardingRuleConfig) {
        return null == shardingRuleConfig.getDefaultKeyGeneratorConfig() ? null : shardingRuleConfig.getDefaultKeyGeneratorConfig().getColumn();
    }
}


public final class TableRule {
    
    private final String logicTable;
    private final List<DataNode> actualDataNodes;
    private final Set<String> actualTables;
    private final Map<DataNode, Integer> dataNodeIndexMap;
    private final ShardingStrategy databaseShardingStrategy;
    private final ShardingStrategy tableShardingStrategy;
    private final String generateKeyColumn;
    private final ShardingKeyGenerator shardingKeyGenerator;
    private final Collection<String> actualDatasourceNames = new LinkedHashSet<>();
    private final Map<String, Collection<String>> datasourceToTablesMap = new HashMap<>();

    public TableRule(final TableRuleConfiguration tableRuleConfig, final ShardingDataSourceNames shardingDataSourceNames, final String defaultGenerateKeyColumn) {
        // 生成邏輯表名logicTable
        logicTable = tableRuleConfig.getLogicTable().toLowerCase();
        // 解析actualNodes生成dataNodes，分表的場(chǎng)景下這個(gè)值為demo_ds.t_order_0和demo_ds.t_order_1
        List<String> dataNodes = new InlineExpressionParser(tableRuleConfig.getActualDataNodes()).splitAndEvaluate();
        dataNodeIndexMap = new HashMap<>(dataNodes.size(), 1);
        // 生成實(shí)際的DataNode節(jié)點(diǎn)，不同場(chǎng)景使用不同的策略
        actualDataNodes = isEmptyDataNodes(dataNodes)
            ? generateDataNodes(tableRuleConfig.getLogicTable(), shardingDataSourceNames.getDataSourceNames()) : generateDataNodes(dataNodes, shardingDataSourceNames.getDataSourceNames());
        // 獲取實(shí)際的表名，獲取actualDataNode種DataNode的表名
        actualTables = getActualTables();
        // 分庫策略
        databaseShardingStrategy = null == tableRuleConfig.getDatabaseShardingStrategyConfig() ? null : ShardingStrategyFactory.newInstance(tableRuleConfig.getDatabaseShardingStrategyConfig());
        // 分表策略
        tableShardingStrategy = null == tableRuleConfig.getTableShardingStrategyConfig() ? null : ShardingStrategyFactory.newInstance(tableRuleConfig.getTableShardingStrategyConfig());
        // 生成key的列名
        generateKeyColumn = getGenerateKeyColumn(tableRuleConfig.getKeyGeneratorConfig(), defaultGenerateKeyColumn);
        // key的生成器
        shardingKeyGenerator = containsKeyGeneratorConfiguration(tableRuleConfig)
                ? new ShardingKeyGeneratorServiceLoader().newService(tableRuleConfig.getKeyGeneratorConfig().getType(), tableRuleConfig.getKeyGeneratorConfig().getProperties()) : null;
        checkRule(dataNodes);
    }
    
    private List<DataNode> generateDataNodes(final List<String> actualDataNodes, final Collection<String> dataSourceNames) {
        List<DataNode> result = new LinkedList<>();
        int index = 0;
        for (String each : actualDataNodes) {
            DataNode dataNode = new DataNode(each);
            if (!dataSourceNames.contains(dataNode.getDataSourceName())) {
                throw new ShardingException("Cannot find data source in sharding rule, invalid actual data node is: '%s'", each);
            }
            result.add(dataNode);
            dataNodeIndexMap.put(dataNode, index);
            actualDatasourceNames.add(dataNode.getDataSourceName());
            addActualTable(dataNode.getDataSourceName(), dataNode.getTableName());
            index++;
        }
        return result;
    }
}

public final class DataNode {
    private static final String DELIMITER = ".";
    // 所屬的數(shù)據(jù)源
    private final String dataSourceName;
    // 邏輯表名
    private final String tableName;

    public DataNode(final String dataNode) {
        if (!isValidDataNode(dataNode)) {
            throw new ShardingConfigurationException("Invalid format for actual data nodes: '%s'", dataNode);
        }
        // 通過符號(hào)"."進(jìn)行分割，獲取dataSourceName和tableName
        List<String> segments = Splitter.on(DELIMITER).splitToList(dataNode);
        dataSourceName = segments.get(0);
        tableName = segments.get(1);
    }
}

• TableRule記錄的是每個(gè)表的分庫后的結(jié)果。
• TableRule的核心變量包括logicTable邏輯、actualDataNodes實(shí)際數(shù)據(jù)存儲(chǔ)節(jié)點(diǎn)、tableShardingStrategy分表策略。
• 分表場(chǎng)景下t_order的dataNodes包含demo_ds.t_order_0和demo_ds.t_order_1。
• 分表場(chǎng)景下t_order_item的dataNodes包含demo_ds.t_order_item_0和demo_ds.t_order_item_1。
• InlineExpressionParser為根據(jù)配置生成實(shí)際物理節(jié)點(diǎn)的核心功能類。

public final class ShardingStrategyFactory {

    public static ShardingStrategy newInstance(final ShardingStrategyConfiguration shardingStrategyConfig) {
        // 分表場(chǎng)景下分表策略為StandardShardingStrategy對(duì)象
        if (shardingStrategyConfig instanceof StandardShardingStrategyConfiguration) {
            return new StandardShardingStrategy((StandardShardingStrategyConfiguration) shardingStrategyConfig);
        }
        if (shardingStrategyConfig instanceof InlineShardingStrategyConfiguration) {
            return new InlineShardingStrategy((InlineShardingStrategyConfiguration) shardingStrategyConfig);
        }
        if (shardingStrategyConfig instanceof ComplexShardingStrategyConfiguration) {
            return new ComplexShardingStrategy((ComplexShardingStrategyConfiguration) shardingStrategyConfig);
        }
        if (shardingStrategyConfig instanceof HintShardingStrategyConfiguration) {
            return new HintShardingStrategy((HintShardingStrategyConfiguration) shardingStrategyConfig);
        }
        return new NoneShardingStrategy();
    }
}

• 分表場(chǎng)景下分表策略為StandardShardingStrategy。

public interface ShardingStrategy {
    Collection<String> getShardingColumns();
    Collection<String> doSharding(Collection<String> availableTargetNames, Collection<RouteValue> shardingValues);
}


public final class StandardShardingStrategy implements ShardingStrategy {
    // 分表的列表
    private final String shardingColumn;
    // 按值精確分表的算法
    private final PreciseShardingAlgorithm preciseShardingAlgorithm;
    // 按指范圍分表的算法
    private final RangeShardingAlgorithm rangeShardingAlgorithm;
    
    public StandardShardingStrategy(final StandardShardingStrategyConfiguration standardShardingStrategyConfig) {
        Preconditions.checkNotNull(standardShardingStrategyConfig.getShardingColumn(), "Sharding column cannot be null.");
        Preconditions.checkNotNull(standardShardingStrategyConfig.getPreciseShardingAlgorithm(), "precise sharding algorithm cannot be null.");
        // 分表的列名 order_id
        shardingColumn = standardShardingStrategyConfig.getShardingColumn();
        preciseShardingAlgorithm = standardShardingStrategyConfig.getPreciseShardingAlgorithm();
        rangeShardingAlgorithm = standardShardingStrategyConfig.getRangeShardingAlgorithm();
    }
    
    @Override
    public Collection<String> doSharding(final Collection<String> availableTargetNames, final Collection<RouteValue> shardingValues) {
        // t_order維度下availableTargetNames為t_order_0和t_order_1
        // t_order維度下shardingValues為t_order.order_id = 479354537787240448
        RouteValue shardingValue = shardingValues.iterator().next();
        Collection<String> shardingResult = shardingValue instanceof ListRouteValue
                ? doSharding(availableTargetNames, (ListRouteValue) shardingValue) : doSharding(availableTargetNames, (RangeRouteValue) shardingValue);
        Collection<String> result = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
        result.addAll(shardingResult);
        return result;
    }

    @SuppressWarnings("unchecked")
    private Collection<String> doSharding(final Collection<String> availableTargetNames, final ListRouteValue<?> shardingValue) {
        Collection<String> result = new LinkedList<>();
        // t_order維度下availableTargetNames為t_order_0和t_order_1
        // t_order維度下shardingValue為t_order.order_id = 479354537787240448
        for (Comparable<?> each : shardingValue.getValues()) {
            // 通過preciseShardingAlgorithm來執(zhí)行分片操作
            String target = preciseShardingAlgorithm.doSharding(availableTargetNames, new PreciseShardingValue(shardingValue.getTableName(), shardingValue.getColumnName(), each));
            if (null != target) {
                result.add(target);
            }
        }
        return result;
    }
    
    @Override
    public Collection<String> getShardingColumns() {
        Collection<String> result = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
        result.add(shardingColumn);
        return result;
    }
}

// 自定義的精確分表算法實(shí)現(xiàn)
public final class PreciseModuloShardingTableAlgorithm implements PreciseShardingAlgorithm<Long> {
    
    @Override
    public String doSharding(final Collection<String> tableNames, final PreciseShardingValue<Long> shardingValue) {
        for (String each : tableNames) {
            // 數(shù)據(jù)庫的實(shí)際物理節(jié)點(diǎn)名和取模后的值比較相等直接返回
            if (each.endsWith(shardingValue.getValue() % 2 + "")) {
                return each;
            }
        }
        throw new UnsupportedOperationException();
    }
}

• StandardShardingStrategy需要自定義精確分表的算法，如例子中的PreciseModuloShardingTableAlgorithm。
• 自定義分片的邏輯按照具體的分片的列值和邏輯表的個(gè)數(shù)取模確定應(yīng)該落到具體那個(gè)表當(dāng)中。

public final class PreciseShardingValue<T extends Comparable<?>> implements ShardingValue {
    // 邏輯表名
    private final String logicTableName;
    // 列名
    private final String columnName;
    // 對(duì)應(yīng)的列值
    private final T value;

    @ConstructorProperties({"logicTableName", "columnName", "value"})
    public PreciseShardingValue(String logicTableName, String columnName, T value) {
        this.logicTableName = logicTableName;
        this.columnName = columnName;
        this.value = value;
    }
}

• 分表的場(chǎng)景下logicTableName為t_order，columnName為order_id，value為479354537787240448。

public final class ListRouteValue<T extends Comparable<?>> implements RouteValue {
    // 列名
    private final String columnName;
    // 表名
    private final String tableName;
    // 對(duì)應(yīng)的值
    private final Collection<T> values;
    
    @Override
    public String toString() {
        return tableName + "." + columnName + (1 == values.size() ? " = " + new ArrayList<>(values).get(0) : " in (" + Joiner.on(",").join(values) + ")");
    }
}

• 分表場(chǎng)景下t_order的ListRouteValue中tableName為t_order，columnName為order_id，values為479354537787240448。