Case-Ironic部署時No valid host

Ironic部署時，經(jīng)常會出現(xiàn)No valid host錯誤，這類錯誤常見的一個原因是hypervisor的資源不夠了。

Issue

hypervisor 可用資源為0

[root@cloud-b12-02 ~]# source  admin-openrc.sh 
[root@cloud-b12-02 ~(admin)]# nova hypervisor-list 
+----+----------------------------------------------+-------+----------+
| ID | Hypervisor hostname                          | State | Status   |
+----+----------------------------------------------+-------+----------+
| 4  | cloud-sz-compute-b11-04.sz.cloud.genomics.cn | up    | enabled  |
| 7  | cloud-sz-compute-b10-01.sz.cloud.genomics.cn | up    | enabled  |
| 10 | cloud-sz-compute-b11-03.sz.cloud.genomics.cn | up    | enabled  |
| 13 | cloud-sz-compute-b10-02.sz.cloud.genomics.cn | up    | enabled  |
| 16 | cloud-sz-compute-b11-01.sz.cloud.genomics.cn | up    | enabled  |
| 19 | cloud-sz-compute-b11-02.sz.cloud.genomics.cn | up    | enabled  |
| 25 | cloud-sz-compute-f18-03.sz.cloud.genomics.cn | down  | disabled |
| 55 | dbbda9bf-2fec-47ed-9a26-142f3d34c8d3         | up    | enabled  |
+----+----------------------------------------------+-------+----------+
[root@cloud-b12-02 ~(admin)]# nova hypervisor-show 55
+-------------------------+--------------------------------------+
| Property                | Value                                |
+-------------------------+--------------------------------------+
| cpu_info                | {}                                   |
| current_workload        | 0                                    |
| disk_available_least    | 0                                    |
| free_disk_gb            | 0                                    |
| free_ram_mb             | 0                                    |
| host_ip                 | 10.54.12.23                          |
| hypervisor_hostname     | dbbda9bf-2fec-47ed-9a26-142f3d34c8d3 |
| hypervisor_type         | ironic                               |
| hypervisor_version      | 1                                    |
| id                      | 55                                   |
| local_gb                | 0                                    |
| local_gb_used           | 0                                    |
| memory_mb               | 0                                    |
| memory_mb_used          | 0                                    |
| running_vms             | 0                                    |
| service_disabled_reason | None                                 |
| service_host            | cloud-b12-03-ironic       |
| service_id              | 154                                  |
| state                   | up                                   |
| status                  | enabled                              |
| vcpus                   | 0                                    |
| vcpus_used              | 0                                    |
+-------------------------+--------------------------------------+

可用資源為0，導(dǎo)致deploy裸機時報"no valid host"

Investigation

在該節(jié)點上，查一下resource_tracker

2017-11-14 22:09:54.297 7 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: d563d14f74ef47c9990bcbd147e576d1 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-11-14 22:09:54.298 7 INFO nova.compute.resource_tracker [req-d27d05d9-d132-40d4-81c7-503ca2a42849 - - - - -] Final resource view: name=dbbda9bf-2fec-47ed-9a26-142f3d34c8d3 phys_ram=0MB used_ram=0MB phys_disk=0GB used_disk=0GB total_vcpus=0 used_vcpus=0 pci_stats=[]
2017-11-14 22:09:54.299 7 DEBUG nova.compute.resource_tracker [req-d27d05d9-d132-40d4-81c7-503ca2a42849 - - - - -] Compute_service record updated for cloud-b12-03-ironic:dbbda9bf-2fec-47ed-9a26-142f3d34c8d3 _update_available_resource /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:626

定時任務(wù)
Running periodic task ComputeManager.update_available_resource

2017-11-14 22:07:52.018 7 DEBUG nova.compute.resource_tracker [req-d27d05d9-d132-40d4-81c7-503ca2a42849 - - - - -] Hypervisor: VCPU information unavailable _report_hypervisor_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:658
2017-11-14 22:07:52.019 7 DEBUG nova.compute.resource_tracker [req-d27d05d9-d132-40d4-81c7-503ca2a42849 - - - - -] Hypervisor/Node resource view: name=dbbda9bf-2fec-47ed-9a26-142f3d34c8d3 free_ram=0MB free_disk=0GB free_vcpus=unknown pci_devices=None _report_hypervisor_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:672

Analysis

在添加一個裸機節(jié)點（Ironic node-create）之后，就會生成對應(yīng)的一個hypervisor .

[root@cloud-b12-03 nova]#  openstack hypervisor list
+----+----------------------------------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname                          | Hypervisor Type | Host IP      | State |
+----+----------------------------------------------+-----------------+--------------+-------+
|  4 | cloud-sz-compute-b11-04.sz.cloud.genomics.cn | QEMU            | 10.54.12.27  | up    |
|  7 | cloud-sz-compute-b10-01.sz.cloud.genomics.cn | QEMU            | 10.54.12.28  | up    |
| 10 | cloud-sz-compute-b11-03.sz.cloud.genomics.cn | QEMU            | 10.54.12.26  | up    |
| 13 | cloud-sz-compute-b10-02.sz.cloud.genomics.cn | QEMU            | 10.54.12.29  | up    |
| 16 | cloud-sz-compute-b11-01.sz.cloud.genomics.cn | QEMU            | 10.54.12.24  | up    |
| 19 | cloud-sz-compute-b11-02.sz.cloud.genomics.cn | QEMU            | 10.54.12.25  | up    |
| 25 | cloud-sz-compute-f18-03.sz.cloud.genomics.cn | QEMU            | 10.54.12.183 | down  |
| 61 | ef89b610-96ab-473f-a0d8-9294d7efd4d8         | ironic          | 10.54.12.23  | up    |
+----+----------------------------------------------+-----------------+--------------+-------+
[root@cloud-b12-03 nova]# ironic node-list 
+--------------------------------------+-------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name  | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------+---------------+-------------+--------------------+-------------+
| ef89b610-96ab-473f-a0d8-9294d7efd4d8 | node1 | None          | power off   | available          | False       |
+--------------------------------------+-------+---------------+-------------+--------------------+-------------+
[root@cloud-sz-kolla-2 ironic-deploy-test]# openstack hypervisor show 61
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| aggregates           | [u'baremetal-hosts']                 |
| cpu_info             |                                      |
| current_workload     | 0                                    |
| disk_available_least | 10                                   |
| free_disk_gb         | 10                                   |
| free_ram_mb          | 4000                                 |
| host_ip              | 10.54.12.23                          |
| hypervisor_hostname  | ef89b610-96ab-473f-a0d8-9294d7efd4d8 |
| hypervisor_type      | ironic                               |
| hypervisor_version   | 1                                    |
| id                   | 61                                   |
| local_gb             | 10                                   |
| local_gb_used        | 0                                    |
| memory_mb            | 4000                                 |
| memory_mb_used       | 0                                    |
| running_vms          | 0                                    |
| service_host         | cloud-b12-03-ironic       |
| service_id           | 154                                  |
| state                | up                                   |
| status               | enabled                              |
| vcpus                | 1                                    |
| vcpus_used           | 0                                    |
+----------------------+--------------------------------------+


## log 
正常情況下：
```log
2017-11-14 22:40:26.081 7 DEBUG oslo_messaging._drivers.amqpdriver [req-d27d05d9-d132-40d4-81c7-503ca2a42849 - - - - -] CALL msg_id: 629e4b456d334e5ab4fd6b947b99ba5b exchange 'nova' topic 'conductor' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:442
2017-11-14 22:40:26.101 7 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 629e4b456d334e5ab4fd6b947b99ba5b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-11-14 22:40:26.139 7 DEBUG nova.virt.ironic.driver [req-d27d05d9-d132-40d4-81c7-503ca2a42849 - - - - -] Returning 1 available node(s) get_available_nodes /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:608

2017-11-14 23:38:41.266 6 DEBUG nova.servicegroup.drivers.db [req-3be3c1b2-64cc-4c74-97b2-150e8beb3458 - - - - -] DB_Driver: join new ServiceGroup member cloud-b12-03-ironic to the compute group, service = <Service: host=cloud-b12-03-ironic, binary=nova-compute, manager_class_name=nova.compute.manager.ComputeManager> join /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:47

2017-11-14 23:53:53.658 6 INFO nova.compute.resource_tracker [req-7c6362f1-4f6c-4ce0-9c45-2913d9b694c3 - - - - -] Final resource view: name=ef89b610-96ab-473f-a0d8-9294d7efd4d8 phys_ram=0MB used_ram=1024MB phys_disk=0GB used_disk=10GB total_vcpus=0 used_vcpus=0 pci_stats=[]

該節(jié)點部署完了之后，hypervisor的可用資源就變?yōu)?了。

hypervisor resouce statistic

日志中的resource tracker數(shù)據(jù)是“注冊數(shù)據(jù)-部署時使用的flavor的數(shù)據(jù)” 得出來的。

ironic node-update node1 add \
   properties/cpus=1 \
   properties/memory_mb=4000 \
   properties/local_gb=10 \
   properties/cpu_arch="x86_64" \
   properties/capabilities="boot_option:local"

[root@cloud-sz-kolla-b13-01 ironic]# openstack flavor show b1.half 
+----------------------------+----------------------------------------------------+
| Field                      | Value                                              |
+----------------------------+----------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                              |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                  |
| access_project_ids         | None                                               |
| disk                       | 10                                                 |
| id                         | 205bcb85-cf81-4911-83f6-65b392ac24f2               |
| name                       | b1.half                                            |
| os-flavor-access:is_public | True                                               |
| properties                 | baremetal='true', capabilities:boot_option='local' |
| ram                        | 1024                                               |
| rxtx_factor                | 1.0                                                |
| swap                       |                                                    |
| vcpus                      | 1                                                  |
+----------------------------+----------------------------------------------------+

KB

在Build階段刪除Instance

build時刪除Instance 之后，發(fā)現(xiàn)hypervisor可用資源為0, 過10秒之后忘記觀察結(jié)果了

[root@cloud-sz-kolla-2 ironic-deploy-centos6]# openstack server list
+------------------------+------------------------+--------+------------------------+-------------------------+
| ID                     | Name                   | Status | Networks               | Image Name              |
+------------------------+------------------------+--------+------------------------+-------------------------+
| 0f90dafe-21f6-42cf-979 | bare1                  | BUILD  | provision=10.54.0.103  | bm-user-half-centos6-os

對同一個baremetal node 反復(fù)deploy

deploy成功， 然后delete server, 然后查看hypervisor stats 發(fā)現(xiàn)剛開始是0， 過幾秒之后變正常。

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

  • Spring Cloud為開發(fā)人員提供了快速構(gòu)建分布式系統(tǒng)中一些常見模式的工具（例如配置管理，服務(wù)發(fā)現(xiàn)，斷路器，智...
    卡卡羅2017閱讀 136,569評論 19贊 139
  • 第一章 OpenStack基礎(chǔ) OpenStack管理的資源及提供的服務(wù)OpenStack做為一個操作系統(tǒng)，...
    sgt_tiger閱讀 13,239評論 4贊 72
  • 原文鏈接：http://www.dropwizard.io/1.2.0/docs/getting-started....
    Lance_Xu閱讀 1,156評論 0贊 0
  • 1. spark三種分布式部署模式 Spark支持的主要的三種分布式部署方式分別是standalone、spark...
    ZPPenny閱讀 10,445評論 0贊 5
  • 簡潔之美 當(dāng)上帝要懲罰人類的時候，他有時會給予我們一個荒年，有時會給予我們一次瘟疫或戰(zhàn)爭，有時甚至于還會創(chuàng)造出一個...
    維克君閱讀 728評論 1贊 3

友情鏈接更多精彩內(nèi)容