不同阿里云賬號之間kafka元數(shù)據(jù)的遷移（topic和group）

使用阿里云的kafka-migration工具可以對阿里云上面的kafka元數(shù)據(jù)進行遷移
https://help.aliyun.com/document_detail/155520.html

topic的遷移
下載工具的jar包
https://aliware-images.oss-cn-hangzhou.aliyuncs.com/Kafka/migration%20tool/2021-08-11/kafka-migration.jar?spm=a2c4g.11186623.0.0.14e7217eegBzQm&file=kafka-migration.jar
安裝jdk1.8的環(huán)境

查看topic

[root@ops backup]# java -jar kafka-migration.jar TopicMigrationFromAliyun --sourceAk LTAI5tRiq8bAi3N89xxxxxx --sourceSk 3cUsQYCmKvzINzEtiPqxxxxxxx --sourceRegionId cn-beijing --sourceInstanceId alikafka_pre-cn-nxxxan2004 --destAk LTAI5tNmPSzKKxxxxxxxx --destSk MeA2MddZgkyk2qXcF3xxxxxxx --destRegionId cn-beijing --destInstanceId alikafka_post-cn-2xxxm1xp001 
16:12:18 INFO - Reflections took 119 ms to scan 1 urls, producing 6 keys and 14 values  
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
16:12:19 INFO - cmd=TopicMigrationFromAliyun, request=null, response={"total":61,"code":200,"requestId":"6B21589C-BC98-5A06-A444-F4C54D3D994F","success":true,"pageSize":10000,"currentPage":1,"message":"operation success.","topicList":[{"instanceId":"alikafka_pre-cn-n6w1wcan2004","localTopic":false,"createTime":1658391386000,"regionId":"cn-beijing","statusName":"服務(wù)中","topic":"GROUP_FINANCE_EXPORT_DATA_DEBUG","remark":"GROUP_FINANCE_EXPORT_DATA_DEBUG","partitionNum":6,"compactTopic":false,"status":0,"tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","localTopic":false,"createTime":1658391363000,"regionId":"cn-beijing","statusName":"服務(wù)中","topic":"GROUP_FINANCE_EXPORT_DATA","remark":"GROUP_FINANCE_EXPORT_DATA","partitionNum":6,"compactTopic":false,"status":0,"tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","localTopic":false,"createTime":1657535605000,"regionId":"cn-beijing","statusName":"服務(wù)中","
16:12:19 INFO - Will create topic:EMPLOYEE_ENTRANCE_AUTH, isCompactTopic:false, partition number:4 
16:12:19 INFO - Will create topic:RequestSuccessLogTopic, isCompactTopic:false, partition number:4 
16:12:19 INFO - Will create topic:JOIN_REWARD_ACTIVITY, isCompactTopic:false, partition number:4 
16:12:19 INFO - Will create topic:REGISTER, isCompactTopic:false, partition number:4 
16:12:19 INFO - Will create topic:NEW_FIRST_LOGIN, isCompactTopic:false, partition number:4

參數(shù) 描述
sourceAk 源消息隊列Kafka版實例所屬阿里云賬號的AccessKey ID
sourceSk 源消息隊列Kafka版實例所屬阿里云賬號的AccessKey Secret
sourceRegionId 源消息隊列Kafka版實例的地域ID
sourceInstanceId 源消息隊列Kafka版實例的ID
destAk 目標消息隊列Kafka版實例所屬阿里云賬號的AccessKey ID
destSk 目標消息隊列Kafka版實例所屬阿里云賬號的AccessKey Secret
destRegionId 目標消息隊列Kafka版實例的地域ID
destInstanceId 目標消息隊列Kafka版實例的ID
降查詢出的topic提交到新的kafka中

亞太地區(qū)
云服務(wù) 地域名稱 所在城市 Region ID
公共云 華東1 杭州 cn-hangzhou
華東2 上海 cn-shanghai
華北1 青島 cn-qingdao
華北2 北京 cn-beijing
華北3 張家口 cn-zhangjiakou
華南1 深圳 cn-shenzhen
中國香港 香港 cn-hongkong
亞太東南1 新加坡 ap-southeast-1
亞太東南5 雅加達 ap-southeast-5
亞太東北1 東京 ap-northeast-1
政務(wù)云 華北2 北京 cn-north-2-gov-1
金融云 華東1 杭州 cn-hangzhou-finance
華東2 上海 cn-shanghai-finance-1
華南1 深圳 cn-shenzhen-finance-1

歐洲與美洲
云服務(wù) 地域名稱 所在城市 Region ID
公共云 歐洲中部1 法蘭克福 eu-central-1
美國西部1 硅谷 us-west-1

提交topic到新kafka中

[root@ops backup]# java -jar kafka-migration.jar TopicMigrationFromAliyun --sourceAk LTAxxRiq8bAi3N89xxxxxx --sourceSk 3cUsQYCmKvzINzEtiPqxxxxxxx --sourceRegionId cn-beijing --sourceInstanceId alikafka_pre-cn-nxxxan2004 --destAk LTAI5tNmPSzKKxxxxxxxx --destSk MeA2MddZgkyk2qXcF3xxxxxxx --destRegionId cn-beijing --destInstanceId alikafka_post-cn-2xxxm1xp001 --commit

查詢group并提交到新kafak中

java -jar kafka-migration.jar ConsumerGroupMigrationFromAliyun --sourceAk LTAI5xRiq8bAi3Nxxxxxx --sourceSk 3cUsQYCmKxzINzEtxxxxxxd7wxxf --sourceRegionId cn-beijing --sourceInstanceId alikafka_pre-cn-n6w1xxxxxxx --destAk LTAI5tNxPSzKKULDcSxxxxxx --destSk MeA2Mddxxkyk2qXcF3k5lyxxxxxx --destRegionId cn-beijing --destInstanceId alikafka_post-cn-2xxx2ym1xp001
16:18:46 INFO - Reflections took 104 ms to scan 1 urls, producing 6 keys and 14 values  
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
16:18:47 INFO - cmd=ConsumerGroupMigrationFromAliyun, request=null, response={"code":200,"requestId":"247DF1C0-29E9-55D1-B3B5-9F062771CEA0","success":true,"consumerList":[{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"usetime","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"payBack","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workflowStatusChange","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"carSpaceBillReceivePayment","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"requestsuccesslog","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorder-wechat-notice","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"test","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"zlx_cs","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"householdAutoEntrance","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"updateUserPhone","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"faceMSynWhite","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"bindFaceDevice","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"buriedPoint","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"zipkin","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"silentCancel","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"collectCardPrize","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"pushFeedback","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"eventMessage","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"scoreStat","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorderEmergency","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorderTypeChange","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorderDispatchUser","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorderPress","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorderNotice","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"workorderStatusChange","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"houseAddressUpdate","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"preBuildBill","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"syncHousehold","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"yueheAccountRegisterMap","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"syncHouse","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"dataImport","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"areaImport","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"areaFlush","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"deleteControl","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"addControl","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"bindDevice","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"redPacketDeliverUser","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"advertStat","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"joinConsumeActivity","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"recommendPost","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"robotPost","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"exportData","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"goodsCatalogChange","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"oldHouseholdAutoEntrance","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"pushStationLetter","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"sendMarketingSms","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"communityInitPost","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"removeUserHouse","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"joinActivity","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"push","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"payRecordStatusChange","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"addUserHouse","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"employeeAutoEntrance","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"joinRewardActivity","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"register","tags":[]},{"instanceId":"alikafka_pre-cn-n6w1wcan2004","regionId":"cn-beijing","consumerId":"firstLoginAuth","tags":[]}],"message":"operation success."} 
16:18:47 INFO - Total consumer group number:56 
16:18:47 INFO - Check instance capacity success! 
16:18:47 INFO - Will create consumer groups:[usetime, payBack, workflowStatusChange, carSpaceBillReceivePayment, requestsuccesslog, workorder-wechat-notice, test, zlx_cs, householdAutoEntrance, updateUserPhone, faceMSynWhite, bindFaceDevice, buriedPoint, zipkin, silentCancel, collectCardPrize, pushFeedback, eventMessage, scoreStat, workorderEmergency, workorderTypeChange, workorderDispatchUser, workorderPress, workorderNotice, workorderStatusChange, houseAddressUpdate, preBuildBill, syncHousehold, yueheAccountRegisterMap, syncHouse, dataImport, areaImport, areaFlush, deleteControl, addControl, bindDevice, redPacketDeliverUser, advertStat, joinConsumeActivity, recommendPost, robotPost, exportData, goodsCatalogChange, oldHouseholdAutoEntrance, pushStationLetter, sendMarketingSms, communityInitPost, removeUserHouse, joinActivity, push, payRecordStatusChange, addUserHouse, employeeAutoEntrance, joinRewardActivity, register, firstLoginAuth]

提交

[root@ops backup]# java -jar kafka-migration.jar ConsumerGroupMigrationFromAliyun --sourceAk LTAI5xRiq8bAi3Nxxxxxx --sourceSk 3cUsQYCmKxzINzEtxxxxxxd7wxxf --sourceRegionId cn-beijing --sourceInstanceId alikafka_pre-cn-n6w1xxxxxxx --destAk LTAI5tNxPSzKKULDcSxxxxxx --destSk MeA2Mddxxkyk2qXcF3k5lyxxxxxx --destRegionId cn-beijing --destInstanceId alikafka_post-cn-2xxx2ym1xp001 --commit
15:15:03 INFO - cmd=ConsumerGroupMigrationFromAliyun, request=null, response={"code":200,"requestId":"1A87AE04-E357-5119-B93A-A0467A26A175","success":true,"message":"operation success"} 
15:15:03 INFO - ConsumerCreate success, consumer group=joinRewardActivity 
15:15:10 INFO - cmd=ConsumerGroupMigrationFromAliyun, request=null, response={"code":200,"requestId":"4EE9AC00-7B77-524C-BBA6-B0E0B639B244","success":true,"message":"operation success"} 
15:15:10 INFO - ConsumerCreate success, consumer group=register 
15:15:18 INFO - cmd=ConsumerGroupMigrationFromAliyun, request=null, response={"code":200,"requestId":"FD950C10-C140-5368-B05D-E3538F5E4004","success":true,"message":"operation success"} 
15:15:18 INFO - ConsumerCreate success, consumer group=firstLoginAuth 
15:15:23 INFO - Report meta data success, migrationType:consumer_from_aliyun, total:56, succeed:56, failed:0, progress:56/56
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容