httpclient4 · 网络爬虫知识汇总

# httpclient4 <div><div><div style="font-family:Verdana, Arial, Tahoma, sans-serif;font-size:large;font-weight:bold;"><span style="font-size:19px;">第四章 HTTP认证</span></div><div style="font-family:Verdana, Arial, Tahoma, sans-serif;font-size:9.5pt;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient提供对由HTTP标准规范定义的认证模式的完全支持。HttpClient的认证框架可以扩展支持非标准的认证模式，比如NTLM和SPNEGO。</div><h3>4.1 用户凭证</h3><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">任何用户身份验证的过程都需要一组可以用于建立用户身份的凭据。用户凭证的最简单的形式可以仅仅是用户名/密码对。 UsernamePasswordCredentials代表了一组包含安全规则和明文密码的凭据。这个实现对由HTTP标准规范中定义的标准认证模式是足够的</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">UsernamePasswordCredentials creds = new UsernamePasswordCredentials("user", "pwd");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(creds.getUserPrincipal().getName());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(creds.getPassword());</div></blockquote><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">输出内容为：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">user</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">pwd</div></blockquote><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">NTCredentials是微软Windows指定的实现，它包含了除了用户名/密码对外，一组额外的Windows指定的属性，比如用户域名的名字，比如在微软的Windows网络中，相同的用户使用不同设置的认证可以属于不同的域。</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">NTCredentials creds = new NTCredentials("user", "pwd", "workstation", "domain");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(creds.getUserPrincipal().getName());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(creds.getPassword());</div></blockquote><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">输出内容为：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">DOMAIN/user</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">pwd</div></blockquote><h3>4.2 认证模式</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">AuthScheme接口代表了抽象的，面向挑战-响应的认证模式。一个认证模式期望支持如下的功能：</div><ul><li>解析和处理由目标服务器在对受保护资源请求的响应中发回的挑战。</li><li>提供处理挑战的属性：认证模式类型和它的参数，如果可用，比如这个认证模型可应用的领域。</li><li>对给定的凭证组和HTTP请求对响应真实认证挑战生成认证字符串。</li></ul><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">要注意认证模式可能是有状态的，涉及一系列的挑战-响应交流。HttpClient附带了一些AuthScheme实现：</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><ul><li>Basic（基本）：Basic认证模式定义在RFC 2617中。这个认证模式是不安全的，因为凭据以明文形式传送。尽管它不安全，如果用在和TLS/SSL加密的组合中，Basic认证模式是完全够用的。</li><li>Digest（摘要）：Digest认证模式定义在RFC 2617中。Digest认证模式比Basic有显著的安全提升，对不想通过TLS/SL加密在完全运输安全上开销的应用程序来说也是很好的选择。</li><li>NTLM：NTLM是一个由微软开发的优化Windows平台的专有认证模式。NTLM被认为是比Digest更安全的模式。这个模式需要外部的NTLM引擎来工作。要获取更多详情请参考包含在HttpClient发布包中的NTLM_SUPPORT.txt文档。</li></ul><h3>4.3 HTTP认证参数</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">有一些可以用于定制HTTP认证过程和独立认证模式行为的参数：</div><ul><li>'http.protocol.handle-authentication'：定义了是否认证应该被自动处理。这个参数期望的得到一个java.lang.Boolean类型的值。如果这个参数没有被设置，HttpClient将会自动处理认证。</li><li>'http.auth.credential-charset'：定义了当编码用户凭证时使用的字符集。这个参数期望得到一个java.lang.String类型的值。如果这个参数没有被设置，那么就会使用US-ASCII。</li></ul><h3>4.4 认证模式注册表</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient使用AuthSchemeRegistry类维护一个可用的认证模式的注册表。对于每个默认的下面的模式是注册过的：</div><ul><li>Basic：基本认证模式</li><li>Digest：摘要认证模式</li></ul><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">请注意NTLM模式没有对每个默认的进行注册。NTLM不能对每个默认开启是应为许可和法律上的原因。要获取更详细的关于如何开启NTLM支持的内容请看这部分。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><h3>4.5 凭据提供器</h3><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">凭据提供器意来维护一组用户凭据，还有能够对特定认证范围生产用户凭据。认证范围包括主机名，端口号，领域名称和认证模式名称。当使用凭据提供器来注册凭据时，我们可以提供一个通配符（任意主机，任意端口，任意领域，任意模式）来替代确定的属性值。如果直接匹配没有发现，凭据提供器期望被用来发现最匹配的特定范围。</p><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient 可以和任意实现了CredentialsProvider接口的凭据提供器的物理代表一同工作。默认的CredentialsProvider实现被称为 BasicCredentialsProvider，它是简单的凭借java.util.HashMap的实现。</div><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CredentialsProvider credsProvider = new BasicCredentialsProvider();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">credsProvider.setCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("somehost", AuthScope.ANY_PORT),</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new UsernamePasswordCredentials("u1", "p1"));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">credsProvider.setCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("somehost", 8080),</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new UsernamePasswordCredentials("u2", "p2"));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">credsProvider.setCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("otherhost", 8080, AuthScope.ANY_REALM, "ntlm"),</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new UsernamePasswordCredentials("u3", "p3"));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(credsProvider.getCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("somehost", 80, "realm", "basic")));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(credsProvider.getCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("somehost", 8080, "realm", "basic")));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(credsProvider.getCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("otherhost", 8080, "realm", "basic")));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println(credsProvider.getCredentials(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">new AuthScope("otherhost", 8080, null, "ntlm")));</div></blockquote><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">输出内容为：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">[principal: u1]</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">[principal: u2]</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">null</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">[principal: u3]</div></blockquote><h3>4.6 HTTP认证和执行上下文</h3><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient 依赖于AuthState类来跟踪关于认证过程状态的详细信息。在HTTP请求执行过程中，HttpClient创建2个AuthState的实例：一个对于目标主机认证，另外一个对于代理认证。如果目标服务器或代理需要用户认证，那么各自的AuthState实例将会被在认证处理过程中使用的 AuthScope，AuthScheme和Crednetials来填充。AuthState可以被检查来找出请求的认证是什么类型的，是否匹配 AuthScheme的实现，是否凭据提供器对给定的认证范围去找用户凭据。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">在HTTP请求执行的过程中，HttpClient添加了下列和认证相关的对象到执行上下文中：</p><ul><li>'http.authscheme-registry'：AuthSchemeRegistry实例代表真实的认证模式注册表。在本地内容中设置的这个属性的值优先于默认的。</li><li>'http.auth.credentials-provider'：CookieSpec实例代表了真实的凭据提供器。在本地内容中设置的这个属性的值优先于默认的。</li><li>'http.auth.target-scope'：AuthState实例代表了真实的目标认证状态。在本地内容中设置的这个属性的值优先于默认的。</li><li>'http.auth.proxy-scope'：AuthState实例代表了真实的代理认证状态。在本地内容中设置的这个属性的值优先于默认的。</li></ul><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">本地的HttpContext对象可以用于定制HTTP认证内容，并先于请求执行或在请求被执行之后检查它的状态：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpContext localContext = new BasicHttpContext();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpGet httpget = new HttpGet("http://localhost:8080/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpResponse response = httpclient.execute(httpget, localContext);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">AuthState proxyAuthState = (AuthState) localContext.getAttribute(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientContext.PROXY_AUTH_STATE);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Proxy auth scope: " + proxyAuthState.getAuthScope());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Proxy auth scheme: " + proxyAuthState.getAuthScheme());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Proxy auth credentials: " + proxyAuthState.getCredentials());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">AuthState targetAuthState = (AuthState) localContext.getAttribute(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientContext.TARGET_AUTH_STATE);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Target auth scope: " + targetAuthState.getAuthScope());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Target auth scheme: " + targetAuthState.getAuthScheme());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Target auth credentials: " + targetAuthState.getCredentials());</div></blockquote><h3>4.7 抢占认证</h3><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient 不支持开箱的抢占认证，因为滥用或重用不正确的抢占认证可能会导致严重的安全问题，比如将用户凭据以明文形式发送给未认证的第三方。因此，用户期望评估抢占认证和在它们只能应用程序环境内容安全风险潜在的好处，而且要求使用如协议拦截器的标准HttpClient扩展机制添加对抢占认证的支持。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">这是一个简单的协议拦截器，如果没有企图认证，来抢先引入BasicScheme的实例到执行上下文中。请注意拦截器必须在标准认证拦截器之前加入到协议处理链中。</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpRequestInterceptor preemptiveAuth = new HttpRequestInterceptor() {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">public void process(final HttpRequest request,</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">final HttpContext context) throws HttpException, IOException {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">AuthState authState = (AuthState) context.getAttribute(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientContext.TARGET_AUTH_STATE);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(ClientContext.CREDS_PROVIDER);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpHost targetHost = (HttpHost) context.getAttribute(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ExecutionContext.HTTP_TARGET_HOST);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 如果没有初始化auth模式</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">if (authState.getAuthScheme() == null) {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">AuthScope authScope = new AuthScope(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">targetHost.getHostName(),</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">targetHost.getPort());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 获得匹配目标主机的凭据</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">Credentials creds = credsProvider.getCredentials(authScope);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 如果发现了，抢先生成BasicScheme</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">if (creds != null) {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">authState.setAuthScheme(new BasicScheme());</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">authState.setCredentials(creds);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">};</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">DefaultHttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 作为第一个拦截器加入到协议链中</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpclient.addRequestInterceptor(preemptiveAuth, 0);</div></blockquote></div></div><h3>4.8 NTLM 认证</h3><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">当前HttpClient没有提对开箱的NTLM认证模式的支持也可能永远也不会。这个原因是法律上的而不是技术上的。然而，NTLM认证可以使用外部的 NTLM引擎比如JCIFS[http://jcifs.samba.org/]来开启，类库由Samba[http://www.samba.org /]项目开发，作为它们Windows的交互操作程序套装的一部分。要获取详细内容请参考HttpClient发行包中包含的 NTLM_SUPPORT.txt文档。</p><h4>4.8.1 NTLM连接持久化</h4><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">NTLM 认证模式是在计算开销方面昂贵的多的，而且对标准的Basic和Digest模式的性能影响也很大。这很可能是为什么微软选择NTLM认证模式为有状态的主要原因之一。也就是说，一旦认证通过，用户标识是和连接的整个生命周期相关联的。NTLM连接的状态特性使得连接持久化非常复杂，对于明显的原因，持久化NTLM连接不能被使用不同用户标识的用户重用。标准的连接管理器附带HttpClient是完全能够管理状态连接的。而逻辑相关的，使用同一 session和执行上下文为了让它们了解到当前的用户标识的请求也是极为重要的。否则，HttpClient将会终止对每个基于NTLM保护资源的 HTTP请求创建新的HTTP连接。要获取关于有状态的HTTP连接的详细讨论，请参考这个部分。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">因为NTLM连接是有状态的，通常建议使用相对简单的方法触发NTLM认证，比如GET或HEAD，而重用相同的连接来执行代价更大的方法，特别是它们包含请求实体，比如POST或PUT。</p><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">DefaultHttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">NTCredentials creds = new NTCredentials("user", "pwd", "myworkstation", "microsoft.com");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpHost target = new HttpHost("www.microsoft.com", 80, "http");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 保证相同的内容来用于执行逻辑相关的请求</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpContext localContext = new BasicHttpContext();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 首先执行简便的方法。这会触发NTLM认证</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpGet httpget = new HttpGet("/ntlm-protected/info");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpResponse response1 = httpclient.execute(target, httpget, localContext);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpEntity entity1 = response1.getEntity();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">if (entity1 != null) {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">entity1.consumeContent();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">//之后使用相同的内容（和连接）执行开销大的方法。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpPost httppost = new HttpPost("/ntlm-protected/form");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httppost.setEntity(new StringEntity("lots and lots of data"));</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpResponse response2 = httpclient.execute(target, httppost, localContext);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpEntity entity2 = response2.getEntity();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">if (entity2 != null) {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">entity2.consumeContent();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div></blockquote></div><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"> </p></div></div></div> > 资源参考 http://www.cnblogs.com/loveyakamoz/archive/2011/07/21/2112804.html