HtmlExtractor網(wǎng)頁信息抽取組件

HtmlExtractor是一個Java實現(xiàn)的基于模板的網(wǎng)頁結(jié)構(gòu)化信息精準(zhǔn)抽取組件，本身并不包含爬蟲功能，但可被爬蟲或其他程序調(diào)用以便更精準(zhǔn)地對網(wǎng)頁結(jié)構(gòu)化信息進(jìn)行抽取。

HtmlExtractor是為大規(guī)模分布式環(huán)境設(shè)計的，采用主從架構(gòu)，主節(jié)點(diǎn)負(fù)責(zé)維護(hù)抽取規(guī)則，從節(jié)點(diǎn)向主節(jié)點(diǎn)請求抽取規(guī)則，當(dāng)抽取規(guī)則發(fā)生變化，主節(jié)點(diǎn)主動通知從節(jié)點(diǎn)，從而能實現(xiàn)抽取規(guī)則變化之后的實時動態(tài)生效。

如何使用？

HtmlExtractor由2個子項目構(gòu)成，html-extractor和html-extractor-web。
html-extractor實現(xiàn)了數(shù)據(jù)抽取邏輯，是從節(jié)點(diǎn)，html-extractor-web提供web界面來維護(hù)抽取規(guī)則，是主節(jié)點(diǎn)。
html-extractor是一個jar包，可通過maven引用：
<dependency>
    <groupId>org.apdplat</groupId>
    <artifactId>html-extractor</artifactId>
    <version>1.1</version>
</dependency>
html-extractor-web是一個war包，需要部署到Servlet/Jsp容器上。
在html-extractor-web目錄下運(yùn)行mvn jetty:run就可以啟動Servlet/Jsp容器jetty，之后打開瀏覽器訪問：
http://localhost:8080/html-extractor-web/api/ 查看自己定義的規(guī)則。

注意：頁面模板中定義的所有CSS路徑和抽取表達(dá)式全部抽取成功，才算抽取成功，
     只要有一個CSS路徑或抽取表達(dá)式失敗，就是抽取失敗。

單機(jī)集中式使用方法：

//1、構(gòu)造抽取規(guī)則
List<UrlPattern> urlPatterns = new ArrayList<>();
//1.1、構(gòu)造URL模式
UrlPattern urlPattern = new UrlPattern();
urlPattern.setUrlPattern("http://money.163.com/\\d{2}/\\d{4}/\\d{2}/[0-9A-Z]{16}.html");
//1.2、構(gòu)造HTML模板
HtmlTemplate htmlTemplate = new HtmlTemplate();
htmlTemplate.setTemplateName("網(wǎng)易財經(jīng)頻道");
htmlTemplate.setTableName("finance");
//1.3、將URL模式和HTML模板建立關(guān)聯(lián)
urlPattern.addHtmlTemplate(htmlTemplate);
//1.4、構(gòu)造CSS路徑
CssPath cssPath = new CssPath();
cssPath.setCssPath("h1");
cssPath.setFieldName("title");
cssPath.setFieldDescription("標(biāo)題");
//1.5、將CSS路徑和模板建立關(guān)聯(lián)
htmlTemplate.addCssPath(cssPath);
//1.6、構(gòu)造CSS路徑
cssPath = new CssPath();
cssPath.setCssPath("div#endText");
cssPath.setFieldName("content");
cssPath.setFieldDescription("正文");
//1.7、將CSS路徑和模板建立關(guān)聯(lián)
htmlTemplate.addCssPath(cssPath);
//可象上面那樣構(gòu)造多個URLURL模式
urlPatterns.add(urlPattern);

//2、獲取抽取規(guī)則對象
ExtractRegular extractRegular = ExtractRegular.getInstance(urlPatterns);
//注意：可通過如下3個方法動態(tài)地改變抽取規(guī)則
//extractRegular.addUrlPatterns(urlPatterns);
//extractRegular.addUrlPattern(urlPattern);
//extractRegular.removeUrlPattern(urlPattern.getUrlPattern());

//3、獲取HTML抽取工具
HtmlExtractor htmlExtractor = new DefaultHtmlExtractor(extractRegular);

//4、抽取網(wǎng)頁
String url = "http://money.163.com/08/1219/16/4THR2TMP002533QK.html";
HtmlFetcher htmlFetcher = new JSoupHtmlFetcher();
String html = htmlFetcher.fetch(url);
List<ExtractResult> extractResults = htmlExtractor.extract(url, html);

//5、輸出結(jié)果
int i = 1;
for (ExtractResult extractResult : extractResults) {
    System.out.println((i++) + "、網(wǎng)頁 " + extractResult.getUrl() + " 的抽取結(jié)果");
    if(!extractResult.isSuccess()){
        System.out.println("抽取失?。?);
        for(ExtractFailLog extractFailLog : extractResult.getExtractFailLogs()){
            System.out.println("\turl:"+extractFailLog.getUrl());
            System.out.println("\turlPattern:"+extractFailLog.getUrlPattern());
            System.out.println("\ttemplateName:"+extractFailLog.getTemplateName());
            System.out.println("\tfieldName:"+extractFailLog.getFieldName());
            System.out.println("\tfieldDescription:"+extractFailLog.getFieldDescription());
            System.out.println("\tcssPath:"+extractFailLog.getCssPath());
            if(extractFailLog.getExtractExpression()!=null) {
                System.out.println("\textractExpression:" + extractFailLog.getExtractExpression());
            }
        }
        continue;
    }
    Map<String, List<ExtractResultItem>> extractResultItems = extractResult.getExtractResultItems();
    for(String field : extractResultItems.keySet()){
        List<ExtractResultItem> values = extractResultItems.get(field);
        if(values.size() > 1){
            int j=1;
            System.out.println("\t多值字段:"+field);
            for(ExtractResultItem item : values){
                System.out.println("\t\t"+(j++)+"、"+field+" = "+item.getValue());   
            }
        }else{
            System.out.println("\t"+field+" = "+values.get(0).getValue());     
        }
    }
    System.out.println("\tdescription = "+extractResult.getDescription());
    System.out.println("\tkeywords = "+extractResult.getKeywords());
}

多機(jī)分布式使用方法：

1、運(yùn)行主節(jié)點(diǎn)，負(fù)責(zé)維護(hù)抽取規(guī)則：
方法一：在html-extractor-web目錄下運(yùn)行mvn jetty:run 。
方法二：在html-extractor-web目錄下運(yùn)行mvn install ，
      然后將target/html-extractor-web-1.0.war部署到Tomcat。

2、獲取一個HtmlExtractor的實例（從節(jié)點(diǎn)），示例代碼如下：
String allExtractRegularUrl = "http://localhost:8080/HtmlExtractorServer/api/all_extract_regular.jsp";
String redisHost = "localhost";
int redisPort = 6379;
ExtractRegular extractRegular = ExtractRegular.getInstance(allExtractRegularUrl, redisHost, redisPort);
HtmlExtractor htmlExtractor = new DefaultHtmlExtractor(extractRegular);

3、抽取信息，示例代碼如下：
String url = "http://money.163.com/08/1219/16/4THR2TMP002533QK.html";
HtmlFetcher htmlFetcher = new JSoupHtmlFetcher();
String html = htmlFetcher.fetch(url);
List<ExtractResult> extractResults = htmlExtractor.extract(url, html);

int i = 1;
for (ExtractResult extractResult : extractResults) {
    System.out.println((i++) + "、網(wǎng)頁 " + extractResult.getUrl() + " 的抽取結(jié)果");
    if(!extractResult.isSuccess()){
        System.out.println("抽取失敗：");
        for(ExtractFailLog extractFailLog : extractResult.getExtractFailLogs()){
            System.out.println("\turl:"+extractFailLog.getUrl());
            System.out.println("\turlPattern:"+extractFailLog.getUrlPattern());
            System.out.println("\ttemplateName:"+extractFailLog.getTemplateName());
            System.out.println("\tfieldName:"+extractFailLog.getFieldName());
            System.out.println("\tfieldDescription:"+extractFailLog.getFieldDescription());
            System.out.println("\tcssPath:"+extractFailLog.getCssPath());
            if(extractFailLog.getExtractExpression()!=null) {
                System.out.println("\textractExpression:" + extractFailLog.getExtractExpression());
            }
        }
        continue;
    }
    for(ExtractResultItem extractResultItem : extractResult.getExtractResultItems()){
        System.out.print("\t"+extractResultItem.getField()+" = "+extractResultItem.getValue());              
    }
    System.out.println("\tdescription = "+extractResult.getDescription());
    System.out.println("\tkeywords = "+extractResult.getKeywords());
}