一、介紹

WebMagic是一個(gè)簡(jiǎn)單靈活的Java爬蟲框架。基于WebMagic，你可以快速開發(fā)出一個(gè)高效、易維護(hù)的爬蟲。

二、如何學(xué)習(xí)

1.查看官網(wǎng)

官網(wǎng)地址為:http://webmagic.io/官網(wǎng)詳細(xì)文檔:http://webmagic.io/docs/zh/

2.跑通hello world示例(具體可以參考官網(wǎng)，也可以參考博客)

我下面寫的單元測(cè)試案例，可作為Hello World示例。

注意需要導(dǎo)入Maven依賴:

<dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-core</artifactId> <version>0.7.3</version></dependency><dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-extension</artifactId> <version>0.7.3</version></dependency>

3.帶著一個(gè)目的

說說我的目的，最近我開發(fā)的博客系統(tǒng)，其中有個(gè)導(dǎo)入第三方博客的插件，這個(gè)插件比較簡(jiǎn)單就是一個(gè)搜索框，在對(duì)應(yīng)的搜索框里面填寫URL，點(diǎn)擊搜索即可導(dǎo)入到自己的博客。

以導(dǎo)入博客園單篇文章為例:

下面是我的源代碼(單篇文章導(dǎo)入,我已經(jīng)將其封裝成一個(gè)工具類):

import cn.hutool.core.date.DateUtil;import com.blog.springboot.dto.CnBlogModelDTO;import com.blog.springboot.entity.Posts;import com.blog.springboot.service.PostsService;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.stereotype.Component;import us.codecraft.webmagic.Page;import us.codecraft.webmagic.Site;import us.codecraft.webmagic.Spider;import us.codecraft.webmagic.pipeline.ConsolePipeline;import us.codecraft.webmagic.processor.PageProcessor;import us.codecraft.webmagic.selector.Selectable;import javax.annotation.PostConstruct;/** * 導(dǎo)入博客園文章工具類 */@Componentpublic class WebMagicCnBlogUtils implements PageProcessor { @Autowired private PostsService postService; public static WebMagicCnBlogUtils magicCnBlogUtils; @PostConstruct public void init() { magicCnBlogUtils = this; magicCnBlogUtils.postService = this.postService; } private Site site = Site.me() .setDomain('https://www.cnblogs.com/') .setSleepTime(1000) .setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'); @Override public void process(Page page) { Selectable obj = page.getHtml().xpath('//div[@class=’post’]'); Selectable title = obj.xpath('//h1[@class=’postTitle’]//a'); Selectable content = obj.xpath('//div[@class=’blogpost-body’]'); System.out.println('title:' + title.replace('<[^>]*>', '')); System.out.println('content:' + content); CnBlogModelDTO blog = new CnBlogModelDTO(); blog.setTitle(title.toString()); blog.setContent(content.toString()); Posts post = new Posts(); String date = DateUtil.date().toString(); post.setPostAuthor(1L); post.setPostTitle(title.replace('<[^>]*>', '').toString()); post.setPostContent(content.toString()); post.setPostExcerpt(content.replace('<[^>]*>', '').toString()); post.setPostDate(date); post.setPostDate(date); post.setPostModified(date); boolean importPost = magicCnBlogUtils.postService.insert(post); if (importPost) { System.out.println('success'); } else { System.out.println('fail'); } } @Override public Site getSite() { return site; } /** * 導(dǎo)入單篇博客園文章數(shù)據(jù) * * @param url */ public static void importSinglePost(String url) { Spider.create(new WebMagicCnBlogUtils()).addUrl(url).addPipeline(new ConsolePipeline()).run(); }}

單元測(cè)試代碼:

import com.blog.springboot.dto.CnBlogModelDTO;import us.codecraft.webmagic.Page;import us.codecraft.webmagic.Site;import us.codecraft.webmagic.Spider;import us.codecraft.webmagic.pipeline.ConsolePipeline;import us.codecraft.webmagic.processor.PageProcessor;import us.codecraft.webmagic.selector.Selectable;public class WebMagicJunitTest implements PageProcessor { private Site site = Site.me() .setDomain('https://www.cnblogs.com/') .setSleepTime(1000) .setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'); @Override public void process(Page page) { Selectable obj = page.getHtml().xpath('//div[@class=’post’]'); Selectable title = obj.xpath('//h1[@class=’postTitle’]//a'); Selectable content = obj.xpath('//div[@class=’blogpost-body’]'); System.out.println('title:' + title.replace('<[^>]*>', '')); System.out.println('content:' + content); } @Override public Site getSite() { return site; } public static void importSinglePost(String url) { Spider.create(new WebMagicJunitTest()).addUrl(url).addPipeline(new ConsolePipeline()).run(); } public static void main(String[] args) { WebMagicJunitTest.importSinglePost('https://www.cnblogs.com/youcong/p/9404007.html'); }

另外我是怎么知道要爬取哪些數(shù)據(jù)呢?需求第一，然后通過Chrome或Firefox瀏覽器檢查元素，如圖:

到此這篇關(guān)于Java爬蟲框架之WebMagic實(shí)戰(zhàn)的文章就介紹到這了,更多相關(guān)Java WebMagic內(nèi)容請(qǐng)搜索好吧啦網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持好吧啦網(wǎng)！