簡介

TextRank是受到Google的PageRank的啟發(fā)，通過把文本分割成若干組成單元(單詞、句子)并建立圖模型, 利用投票機制對文本中的重要成分進行排序, 僅利用單篇文檔本身的信息即可實現(xiàn)關(guān)鍵詞提取、文摘。和 LDA、HMM 等模型不同, TextRank不需要事先對多篇文檔進行學習訓練, 因其簡潔有效而得到廣泛應用。
TextRank 一般模型可以表示為一個有向有權(quán)圖 G =(V, E), 由點集合 V和邊集合 E 組成, E 是V ×V的子集。圖中任兩點 Vi , Vj 之間邊的權(quán)重為 w_ji , 對于一個給定的點 Vi, In(Vi) 為 指 向 該 點 的 點 集 合 , Out(Vi) 為點 Vi 指向的點集合。點 Vi 的得分定義如下:

textrankformula.png

我們可以通過TextRank算法，對文章做關(guān)鍵詞的提取以及自動文摘提取。
更詳細的內(nèi)容參見：和textrank4ZH代碼一模一樣的算法詳細解讀

相關(guān)組件安裝

相關(guān)組件為textrank4zh，其用于抽取中文文章的關(guān)鍵字以及關(guān)鍵句（作為文摘）
安裝方法很簡單：

pip install textrank4zh

中文文章摘要提取

以我之前寫的一篇文章：AWS認證備考心得為例（硬廣啊，O(∩_∩)O哈哈~）

2019-02-13 18_17_49-Start.png

我們通過非常簡單的代碼實現(xiàn)TextRank的使用：

# coding:utf-8

from textrank4zh import TextRank4Keyword, TextRank4Sentence
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl


if __name__ == '__main__':
    f = open('./aws.txt', mode='r', encoding='utf-8')
    text = f.read()
    f.close()

    tr4w = TextRank4Keyword()
    tr4w.analyze(text=text, lower=True, window=5)
    print('關(guān)鍵詞：')
    for item in tr4w.get_keywords(10, word_min_len=1):
        print(item['word'], item['weight'])

    tr4s = TextRank4Sentence()
    tr4s.analyze(text=text, lower=True, source = 'no_stop_words')
    data = pd.DataFrame(data=tr4s.key_sentences)
    mpl.rcParams['font.sans-serif'] = ['SimHei']
    mpl.rcParams['axes.unicode_minus'] = False
    plt.figure(facecolor='w')
    plt.plot(data['weight'], 'ro-', lw=2, ms=5, alpha=0.7, mec='#404040')
    plt.grid(b=True, ls=':', color='#606060')
    plt.xlabel('句子', fontsize=12)
    plt.ylabel('重要度', fontsize=12)
    plt.title('句子的重要度曲線', fontsize=15)
    plt.show()

    key_sentences = tr4s.get_key_sentences(num=10, sentence_min_len=2)
    for sentence in key_sentences:
        print(sentence['weight'], sentence['sentence'])

得到的結(jié)果如下：

關(guān)鍵字

關(guān)鍵字 + 權(quán)重的格式

考試 0.027935018564767485
相關(guān) 0.019789154134256207
aws 0.016636259077242657
題目 0.01407576775822367
看 0.013558049263992051
備考 0.01328776426934064
知識點 0.010689853883336327
認證 0.010566520363575575
會 0.009300281741320988
應該 0.009139419618958523

句子權(quán)重占比Top 10

權(quán)重 + 句子的格式

0.024624674608926533 如果這些題目在考試前能夠達到90%的準確率，說明基本掌握了備考的知識點了，如果不放心可以在考試前在AWS認證網(wǎng)站花費20美元預考，大概20個題目，65%及格
0.024016249587116383 不過有一些考點在這本書里并沒有，或者很少提及，比如：Lambda，ECS，Elastic Beanstalk，CloudFormation，API Gateway，我考試的題目里面，有幾題的知識點與它們相關(guān)，所以還是應該看一下相關(guān)的內(nèi)容，最起碼知道它們是做什么的
0.022585396746875635 根據(jù)備考經(jīng)驗，這本書應該是必看的，考試的內(nèi)容應該大都包含了，內(nèi)容編排很合理，如每章都有詳細的介紹，然后有考試需要注意的重點項目，還有相關(guān)內(nèi)容的仿真題型
0.021385870302766598 總結(jié)就是沒有捷徑，上述的各個內(nèi)容最好都認真看一遍，如果備考時間是一個月，每天備考時間大概晚上8點到10點，先半個月左右，即30個學時，把官方備考書先看完
0.02124521540684917 值得注意的是，AWS認證考試是連接美國那邊服務器的，考試之前最好打電話確認一下當天AWS認證考試的網(wǎng)絡情況如何，我去考試那天，網(wǎng)絡斷了四次，不過每次恢復網(wǎng)絡之后，做題進度是聯(lián)網(wǎng)保存的，考試時間也不會有影響
0.02081627266255728 因為工作需要，今年年初備考AWS Solution Architecture Associate的認證考試
0.02027674881999819 AWS資料中，有很多拓撲圖或架構(gòu)圖，先把架構(gòu)相關(guān)的知識點理解，然后再分別學習構(gòu)成相關(guān)架構(gòu)的各個元素，如學會讀懂下圖，掌握VPC, AZone, subnet, load balancer, auto scaling
0.01950681834056804 考試不僅考察應試者對AWS相關(guān)云服務的熟悉程度，題目也多來源于實際中要解決的問題，要求應試者有全面的架構(gòu)設(shè)計經(jīng)驗和積累，所以含金量很高
0.019032121092932345 3. 考試注冊：https://aws.amazon.com/cn/certification/certified-solutions-architect-associate/
0.018858895659387472 選擇AWS SAA相關(guān)的題目，題目很多，一共是500多道題，我在考試過程中，發(fā)現(xiàn)有5題是來自這里的

句子的重要度曲線

textrankaws.png

結(jié)果還是靠譜的，評判標準？畢竟我是作者啊~~~

英文文章摘要提取

事實上，這個textrank4zh做英文文章的摘要提取，也非常不錯。因為該算法本身應該與語言無關(guān)啊，只是textrank4zh的組件包，對中文分詞支持非常好而已。
以SEC公開披露的文章為例，

2019-02-13 18_22_20

直觀來看，與share merge什么的有關(guān)。
我們來看TextRank的結(jié)果，代碼同上，只是喂的文章是這篇英文文章而已。

關(guān)鍵字

關(guān)鍵字 + 權(quán)重的格式

portfolio 0.02184658057602816
policy 0.01727234660766818
merger 0.017022765571195125
fund 0.01688258532487905
date 0.01650937567134385
reorganization 0.014920320912647621
merging 0.014717803511731473
acquired 0.012681283504756821
prospectus 0.012258546110201685
surviving 0.010469987795383606

句子權(quán)重占比Top 10

看起來內(nèi)容聚焦在merge上，八九不離十了~~~~
權(quán)重 + 句子的格式

0.030845544238077883 After the Merger Date. Immediately following the Merger Date, the Acquired Portfolio will no longer be available as an investment option under the policies. In addition, for the sixty (60) days following the Merger Date, you may transfer all or a portion of your accumulation unit value out of the Investment Division for the Acquiring Portfolio to another investment option without any charge or limitation (except potentially harmful transfers (see the “Limits on Transfers” section in the Prospectus for your policy)) and without the transfer counting toward the number of free transfers that otherwise may be made in a given Policy Year. Such transfers will be based on the accumulation unit value of the Investment Division for the Acquiring Portfolio as of the close of the Business Day that we receive the transfer request. All other transfers are subject to limitations, and may be subject to charges, as described in the Prospectus for your policy. Please see the Prospectus for your policy for information on how to complete transfers from the Acquiring Portfolio to other investment options that we currently offer.
0.030211569807805698 Prior to the Merger Date. For forty-five (45) days before the Merger Date, if you have allocations to the Acquired Portfolio, you may transfer such allocations to any other available investment option without any charge or limitation (except potentially harmful transfers (see the “Limits on Transfers” section in the Prospectus for your policy)) and without the transfer counting toward the number of free transfers that otherwise may be made in a given Policy Year. Such transfers will be based on the accumulation unit value of the Investment Division for the Acquired Portfolio as of the close of the Business Day that we receive the transfer request. All other transfers are subject to limitations, and may be subject to charges, as described in the Prospectus for your policy. Please see the Prospectus for your policy for information on how to complete transfers from the Investment Division for the Acquired Portfolio to other investment options that we currently offer.
0.02840028243826242 Prior to the Reorganization Date. With the exception listed below, for forty-five (45) days before the Reorganization Date, if you have allocations to the Merging Fund, you may transfer such allocations to any other available investment option without any charge or limitation (except potentially harmful transfers (see the “Limits on Transfers” section in the Prospectus for your policy)) and without the transfer counting toward the number of free transfers that otherwise may be made in a given Policy Year. Such transfers will be based on the accumulation unit value of the Investment Division for the Merging Fund as of the close of the Business Day that we receive the transfer request. All other transfers are subject to limitations, and may be subject to charges, as described in the Prospectus for your policy. Please see the Prospectus for your policy for information on how to complete transfers from the Investment Division for the Merging Fund to other investment options that we currently offer.
0.0274712995401908 the sixty (60) days following the Reorganization Date, you may transfer all or a portion of your accumulation unit value out of the Investment Division for the Surviving Fund to another investment option without any charge or limitation (except potentially harmful transfers (see the “Limits on Transfers” section in the Prospectus for your policy)) and without the transfer counting toward the number of free transfers that otherwise may be made in a given Policy Year. Such transfers will be based on the accumulation unit value of the Investment Division for the Surviving Fund as of the close of the Business Day that we receive the transfer request. All other transfers are subject to limitations, and may be subject to charges, as described in the Prospectus for your policy. Please see the Prospectus for your policy for information on how to complete transfers from the Surviving Fund to other investment options that we currently offer.
0.02445596932569474 On the Merger Date. Any of your allocations that remain in the Acquired Portfolio will be redeemed. Those redemptions will then be used to purchase accumulation units in the Investment Division for the Acquiring Portfolio. All policyowners affected by the Merger will receive written confirmation of the transaction. The redemption and subsequent repurchase transactions required to effectuate the Merger will not be treated as transfers that count toward the number of free transfers that may otherwise be made in a given Policy Year.
0.021829344645832766 At a meeting held on December 13, 2018, the Board of Trustees of Neuberger Berman Advisers Management Trust approved a Plan of Reorganization and Dissolution (the “Plan”) as of on or about April 30, 2019 (the “Reorganization Date”). As a result of the Reorganization, on the Reorganization Date, policyowners that have allocated cash value to the Large Cap Value Portfolio (the “Merging Fund”) will transfer all of its assets to the Sustainable Equity Portfolio (the “Surviving Fund”) in exchange for shares of the Surviving Fund. In addition to the Reorganization of the Merging Fund, the Board of Trustees also approved a separate reorganization of Neuberger Berman AMT Guardian Portfolio with and into the Surviving Fund.
0.021154204652431365 Until the Merger Date, we will continue to process automatic transactions (such as dollar cost averaging and automatic asset rebalancing) involving the Acquired Portfolio, unless you provide us with alternate allocation instructions. Also note that the Acquired Portfolio will not accept new premium payment allocations or transfers as of the Merger Date.
0.02092614650125736 At a meeting held on December 10-12, 2018, the Board of Trustees of the MainStay VP Funds Trust (the “Board”) approved the Merger of the MainStay VP Epoch U.S. Small Cap Portfolio (the “Acquired Portfolio”) into the MainStay VP MacKay Small Cap Core Portfolio (the “Acquiring Portfolio”), followed by the complete liquidation of the Acquired Portfolio. Shareholder approval is required for the Merger to take place. If shareholder approval is obtained on or about April 22, 2019, the effective date established for the Merger is expected to be on or about May 1, 2019 (the “Merger Date”).
0.020559526950775668 Policy Proceeds are payable under the policy to the named Beneficiary when the Insured dies. Upon receiving due proof of death at our Service Office in Good Order, We will pay the Beneficiary the Life Insurance Benefit determined as of the date we receive due proof of death in Good Order as part of the Policy Proceeds. All or part of the Policy Proceeds can be paid in cash or applied under one or more of Our payment options described under ‘Policy Payment Information—When We Pay Policy Proceeds—Payment Options.’
0.02049239852496608 As a result of the Merger, the Acquiring Portfolio will be available as an Investment Division under your policy on the Merger Date.

句子的重要度曲線

textranksec.png

較大的英文文章摘要提取

這里的較大，是指超過3500句以上的文檔，36萬以上的單詞量

2019-02-13 18_31_25

粗略看上去，我也不清楚主要是說什么的。。。
那就直接看結(jié)果吧

關(guān)鍵字

關(guān)鍵字 + 權(quán)重的格式

policy 0.015148000545573168
insurance 0.0076696873841799375
investment 0.006696896916816165
life 0.0061127787815478205
amount 0.005691912888323185
benefit 0.005606466435793987
rider 0.005400878053318155
premium 0.004975816146068638
death 0.004657684469154609
portfolio 0.004598998468890753

感覺關(guān)鍵字一針見血的樣子~~~~

句子權(quán)重占比Top 10

看起來有幾句是重復的？因為在文章里面這些句子確實重復出現(xiàn)了幾次，而且也被TextRank算法認為很重要。是否真的應該權(quán)重這么高，還需要業(yè)務人員的專業(yè)意見。
不過建議對這種幾千句的文章，可以輸出Top 20或更多的句子，這樣更客觀一些。
權(quán)重 + 句子的格式

0.0009159391391327122 3 The mortality and expense risk face amount charge rate is based on the Age and Risk Class of the Insured and the Face Amount on the Policy Date. It also varies with the Death Benefit Option you choose. Each Coverage Segment will have a corresponding face amount charge related to the amount of the increase, based on the Age and Risk Class of the Insured at the time of the increase. The charge is level for 10 Policy Years from the effective date of the Coverage Segment, then is reduced in Policy Year 11 and thereafter. The mortality and expense risk face amount charges shown in the table may not be typical of the charges you will pay. Ask your life insurance producer for information regarding this charge for your Policy. The mortality and expense risk face amount charge for your Policy will be stated in the Policy Specifications.
0.0009159391391327117 3 The mortality and expense risk face amount charge rate is based on the Age and Risk Class of the Insured and the Face Amount on the Policy Date. It also varies with the Death Benefit Option you choose. Each Coverage Segment will have a corresponding face amount charge related to the amount of the increase, based on the Age and Risk Class of the Insured at the time of the increase. The charge is level for 10 Policy Years from the effective date of the Coverage Segment, then is reduced in Policy Year 11 and thereafter. The mortality and expense risk face amount charges shown in the table may not be typical of the charges you will pay. Ask your life insurance producer for information regarding this charge for your Policy. The mortality and expense risk face amount charge for your Policy will be stated in the Policy Specifications.
0.0009101920742634353 1 Cost of insurance rates apply uniformly to all members of the same Class. The cost of insurance charges shown in the table may not be typical of the charges you will pay. Your Policy Specifications will indicate the guaranteed cost of insurance charge applicable to your Policy, and more detailed information concerning your cost of insurance charges is available on request from your life insurance producer or us. Also, before you purchase the Policy, you may request personalized Illustrations of your future benefits under the Policy based upon the Insured?s Class, the Death Benefit Option, Face Amount, planned periodic premiums, and any Riders requested. Cost of insurance rates for your Policy will be stated in the Policy Specifications and calculated using the Net Amount At Risk.
0.0009085211753153032 1 Cost of insurance rates apply uniformly to all members of the same Class. The cost of insurance charges shown in the table may not be typical of the charges you will pay. Your Policy Specifications pages will indicate the guaranteed cost of insurance charge applicable to your Policy, and more detailed information concerning your cost of insurance charges is available on request from your life insurance producer or us. Also, before you purchase the Policy, you may request personalized Illustrations of your future benefits under the Policy based upon the Insured?s Class, the Death Benefit Option, Face Amount, planned periodic premiums, and any Riders requested. Cost of insurance rates for your Policy will be stated in the Policy Specifications and calculated using the Net Amount At Risk.
0.0009071944047640959 1 Cost of insurance rates apply uniformly to all members of the same Class. The cost of insurance charges shown in the table may not be typical of the charges you will pay. Your Policy Specifications pages will indicate the guaranteed cost of insurance charge applicable to your Policy, and more detailed information concerning your cost of insurance charges is available on request from your life insurance producer or us. Also, before you purchase the Policy, you may request personalized illustrations of your future benefits under the Policy based upon the Insured?s Class, the Death Benefit Option, Face Amount, planned periodic premiums, and any Riders requested. Cost of insurance rates for your Policy will be stated in the Policy Specifications and calculated per $1.00 of Net Amount At Risk.
0.0008978323334814181 · M&E risk asset charge We deduct a risk asset charge every month at a guaranteed maximum annual rate of 0.45% (0.0375% monthly) on the first $25,000 of your Policy?s Accumulated Value in the Investment Options plus an annual rate of 0.05% (0.0042% monthly) of the Accumulated Value in the Investment Options that exceeds $25,000. We may charge a lower annual rate for the M&E risk asset charge. For the purposes of this charge, the amount of Accumulated Value is calculated on the Monthly Payment Date before we deduct the monthly charge, but after we deduct any Policy Debt or allocate any new Net Premiums, withdrawals or loans. When the Insured reaches Age 100, the annual rate is reduced to 0%.
0.0008645495698352475 3 The mortality and expense risk face amount charge rate is based on the Age and Risk Class of the Insured and the Face Amount on the Policy Date. It also varies with the Death Benefit Option you choose. Each Coverage Segment will have a corresponding face amount charge related to the amount of the increase, based on the Age and Risk Class of the Insured at the time of the increase. The mortality and expense risk face amount charges shown in the table may not be typical of the charges you will pay. Ask your life insurance producer for information regarding this charge for your Policy. The mortality and expense risk face amount charge for your Policy will be stated in the Policy Specifications.
0.0008583041949445304 2 The surrender charge is based on the Age and Risk Class of the Insured, as well as the Death Benefit Option you choose. The surrender charge reduces to $0 after 10 years from the effective date of each Coverage Segment. The surrender charge shown in the table may not be typical of the surrender charge you will pay. Ask your life insurance producer for information on this charge for your Policy. The surrender charge and maximum surrender charge for your Policy will be stated in the Policy Specifications.
0.0008583041949445304 2 The surrender charge is based on the Age and Risk Class of the Insured, as well as the Death Benefit Option you choose. The surrender charge reduces to $0 after 10 years from the effective date of each Coverage Segment. The surrender charge shown in the table may not be typical of the surrender charge you will pay. Ask your life insurance producer for information on this charge for your Policy. The surrender charge and maximum surrender charge for your Policy will be stated in the Policy Specifications.
0.0008583041949445304 2 The surrender charge is based on the Age and Risk Class of the Insured, as well as the Death Benefit Option you choose. The surrender charge reduces to $0 after 10 years from the effective date of each Coverage Segment. The surrender charge shown in the table may not be typical of the surrender charge you will pay. Ask your life insurance producer for information on this charge for your Policy. The surrender charge and maximum surrender charge for your Policy will be stated in the Policy Specifications.

句子的重要度曲線

textranksecbig.png