C# + Selenium + ChromeDriver 爬取網(wǎng)頁(yè)

轉(zhuǎn)自：UP技術(shù)控

cnblogs.com/lyl6796910/p/14166876.html

背景

Selenium是一個(gè)用于Web應(yīng)用程序測(cè)試的工具。Selenium測(cè)試直接運(yùn)行在瀏覽器中，就像真正的用戶在操作一樣。而對(duì)于爬蟲(chóng)來(lái)說(shuō)，使用Selenium操控瀏覽器來(lái)爬取網(wǎng)上的數(shù)據(jù)那么肯定是爬蟲(chóng)中的殺手武器。這里，我將介紹selenium + 谷歌瀏覽器的一般使用。

需求

在平常的爬蟲(chóng)開(kāi)發(fā)中，有時(shí)候網(wǎng)頁(yè)是一堆js堆起來(lái)的代碼，涉及很多異步計(jì)算，如果是普通的http 控制臺(tái)請(qǐng)求，那么得到的源文件是一堆js ，需要自己在去組裝數(shù)據(jù)，很費(fèi)力；但是采用Selenium+ChromeDriver可以達(dá)到所見(jiàn)即所得的完美效果。

實(shí)現(xiàn)方式

項(xiàng)目結(jié)構(gòu)：為了方便使用，用的winform程序，附nuget包

以下是form1.cs的代碼，這里就只放關(guān)鍵方法代碼了。需要安裝最新的chrome瀏覽器+代碼中使用的chromedriver是 v2.9.248315

private void crawlingWebFunc()
{
    SetText("\r\n開(kāi)始嘗試...");
    List surls = new List();
    string path = System.Environment.CurrentDirectory + "\\圖片url\\";
    DirectoryInfo root = new DirectoryInfo(path);
    DirectoryInfo[] dics = root.GetDirectories();
    foreach (var itemdic in dics)
    {
        string txt = "";
        StreamReader sr = new StreamReader(itemdic.FullName + "\\data.txt");
        while (!sr.EndOfStream)
        {
            string str = sr.ReadLine();
            txt += str;// + "\n";
        }
        sr.Close();
        surls.Add(new testfold() { key = itemdic.FullName, picurl = txt });
    }
    ChromeDriverService service = ChromeDriverService.CreateDefaultService(System.Environment.CurrentDirectory);
    //  service.HideCommandPromptWindow = true;
    ChromeOptions options = new ChromeOptions();
    options.AddArguments("--test-type", "--ignore-certificate-errors");
    options.AddArgument("enable-automation");
    //   options.AddArgument("headless");
    //  options.AddArguments("--proxy-server=http://user:[email protected]:8080");
    using (IWebDriver driver = new OpenQA.Selenium.Chrome.ChromeDriver(service, options, TimeSpan.FromSeconds(120)))
    {
        driver.Url = "https://www.1688.com/";
        Thread.Sleep(200);
        try
        {
            int a = 1;
            foreach (var itemsurls in surls)
            {
                SetText("\r\n第" + a.ToString() + "個(gè)");
                driver.Navigate().GoToUrl(itemsurls.picurl);
                //登錄
                if (driver.Url.Contains("login.1688.com"))
                {
                    SetText("\r\n需要登錄，開(kāi)始嘗試...");
                    trylogin(driver); //嘗試登錄完成
                                      //再試試        driver.Navigate().GoToUrl("https://s.1688.com/youyuan/index.htm?tab=imageSearch&imageType=oss&imageAddress=cbuimgsearch/eWXC7XHHPN1607529600000&spm=");
                    if (driver.Url.Contains("login.1688.com"))
                    {
                        //沒(méi)辦法退出
                        SetText("\r\n退出，換ip重試...");
                        return;
                    }
                }
                //鼠標(biāo)放上去的內(nèi)容因?yàn)轫?yè)面自帶只能顯示一個(gè)的原因 沒(méi)辦法做到全部顯示 然后在下載 只能是其他方式下載
                //  var elements = document.getElementsByClassName('hover-container');
                //  Array.prototype.forEach.call(elements, function(element) {
                //  element.style.display = "block";
                //   console.log(element);
                //  });
                //   IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
                //    var sss = js.ExecuteScript(" var elements = document.getElementsByClassName('hover-container');  Array.prototype.forEach.call(elements, function(element) {  console.log(element); element.setAttribute(\"class\", \"測(cè)試title\");  element.style.display = \"block\";  console.log(element); });");
                Thread.Sleep(500);
                var responseModel = Write(itemsurls.key, driver.PageSource, Pagetypeenum.列表);
                Thread.Sleep(500);
                int i = 1;
                foreach (var offer in responseModel?.data?.offerList ?? new List())
                { driver.Navigate().GoToUrl(offer.information.detailUrl);
                    string responseDatadetail = driver.PageSource;
                    Write(itemsurls.key, driver.PageSource, Pagetypeenum.詳情);
                    SetText("\r\n第" + a.ToString() + "-" + i.ToString() + "個(gè)");
                    Thread.Sleep(500);
                    i++;
                }
            }
        }
        catch (Exception ex)
        {
            CloseChromeDriver(driver);
            throw;
        }
    }
}


#region 異常  退出chromedriver
[DllImport("user32.dll", EntryPoint = "FindWindow")]
private extern static IntPtr FindWindow(string lpClassName, string lpWindowName);
[DllImport("user32.dll", EntryPoint = "SendMessage")]
public static extern int SendMessage(IntPtr hWnd, int Msg, int wParam, int lParam);
public const int SW_HIDE = 0;
public const int SW_SHOW = 5;
[DllImport("user32.dll", EntryPoint = "ShowWindow")]
public static extern int ShowWindow(IntPtr hwnd, int nCmdShow);

/// 

/// 獲取窗口句柄
/// 
/// 
public IntPtr GetWindowHandle()
{
    string name = (Environment.CurrentDirectory + "\\chromedriver.exe");
    IntPtr hwd = FindWindow(null, name);
    return hwd;
}

/// 

/// 關(guān)閉chromedriver窗口
/// 
public void CloseWindow()
{
    try
    {
        IntPtr hwd = GetWindowHandle();
        SendMessage(hwd, 0x10, 0, 0);
    }
    catch { }
}
/// 

/// 退出chromedriver
/// 
/// 
public void CloseChromeDriver(IWebDriver driver)
{
    try
    {
        driver.Quit();
        driver.Dispose();
    }
    catch { }
    CloseWindow();
}

總結(jié)

說(shuō)一下思路:

1、跳轉(zhuǎn)到指定的網(wǎng)頁(yè)driver.Navigate().GoToUrl

2、確定數(shù)據(jù)源，從driver.PageSource讀取數(shù)據(jù)

3、對(duì)html數(shù)據(jù)進(jìn)行解析