2/14/12 6 essential cURL commands for dail use

1/4 www.codediesel.com/tools/6-essential-curl-commands/
Home About Archives WordPress Plugins Contact
Subscribe (RSS)
Recent Posts
Get latest posts b E-mail

About this blog
This site is a digital habitat of
Sameer, a freelance web developer
working in PHP & MySQL.More
FFmpeg: A beginners guide part 2
FFmpeg: A beginners guide part 1
Using the new Page visibility API in your apps
Generating clean URLs with javascript
Rejecting unwanted characters from input
Amazon Advertising API BrowseNodes
Create a infinite scroll page using jQuery
Comparing programming languages
Changing WordPress admin login error
message
WordPress plugins to check for security
threats
6 essential cURL commands Ior daily use
Posted in: tools | del.icio.us (102) saves. |
cURL is a command line tool for doing all sorts of interesting and essential URL
manipulations and data transfers. The original goal of the program was to transfer files
programmatically via protocols such as http, ftp, gopher, sftp, ftps, scp, tftp, and many
others, via a command line interface. The main benefit of using the command line
interface is that you can use the program in your Windows batch file or Linux shell
scripts to automate many URL related processes. In this post you will see some
essential things you can do using cURL.
1. Reading URLs
Read a plain URL.
cl hp://.google.com
Read a secured URL.
cl hp://.ece-ie.com
Get a web page and store it in a file. The following for example will store the index
page retrieved to the file savedpage.html
cl -o aedpage.hml hp://.eample.com/
Get a HTTP Basic authenticated page
cl - ename:paod hp://.eample.com/
Sometimes a page may redirect to another resource. By default CURL will not follow
page redirections. To make CURL follow redirections use the -L option.
cl -L hp://.eample.com/
2. Reading URLs with variable GET parameters
You can also download pages with a variable GET parameter. For e.g take the
following url:
hp://eample.com/page.php?pageNo=35
C++ Programming Classes
Embedded Sstems Certificate- UCI Etension. Hands-on. Learn More!
etensi on.uci .edu/engi neeri ng-tech
2/14/12 6 essential cURL commands for dail use
2/4 www.codediesel.com/tools/6-essential-curl-commands/
The variable here is the pageNo parameter. You can download all the pages by adding
a regular expression like parameter in the CURL url as given below.
cl -o page#1.hml hp://eample.com/page.php?pageNo=[1-12]
This will download all the pages from page no 1 to page no 12 and save it to a
corresponding file.
3. Reading document information
Show document headers only
cl --head hp://.google.com/
You can also use it on any specific resource.
cl --head hp://.google.com/logo_plain.jpg
Dump document headers to a file
cl --dmp-heade heade. hp://.google.com/
4. CURL and FTP
Get a FTP directory listing
cl fp://ename:paod@eample.com
To get the listing of a different directory append the directory name to the URL.
cl fp://ename:paod@eample.com/dieco/
Upload a file to a remote directory using FTP
cl -T ploadfilename - ename:paod fp://iename.com/mfile
The uploadfilename file will be copied to the remote site and named myfile. If the
destination filename is eliminated the file will be copied with the original name. By
default the file will be copied to the root directory. To copy to some other directory
specify the directory after the site name;e.g.
cl -T ploadfilename - ename:paod
fp://iename.com/dieco/mfile
5. To POST to a page.
You can also process a POST request using CURL. The data will use the application/x-
www-form-urlencoded encoding. Lets say you have the following POST form in your
page:
2/14/12 6 essential cURL commands for dail use
3/4 www.codediesel.com/tools/6-essential-curl-commands/
Lie
9 9
Get the latest posts

Possibl related posts:
Parallel cURL execution in PHP
Free Geolocation API tool
4 Repone
<fom mehod="POST" acion="poce.php">
<inp pe=e name="iem">
<inp pe=e name="caego">
<inp pe=bmi name="bmi" ale="ok">
</fom>
You can use the following CURL command to POST the request.
cl -d "iem=bole&caego=conme&bmi=ok"
.eample.com/poce.php
6. Referer & User Agent
HTTP requests may include a referer field, which is used to tell from which URL the
client got to this particular page. Many programs/scripts check the referer field of
requests to check the source of the request. You can simulate the referer field by the
following command.
cl -e hp://ome_efeing_ie.com hp://.eample.com/
All HTTP requests may set the User-Agent field. It names what user agent or client that
is being used. Many web applications use this information to decide how to display
web pages or use it to track browser usage. You can impersonate a particular browser
by the following method:
cl -A "Moilla/5.0 (compaible; MSIE 7.01; Windo NT 5.0)"
hp://.eample.com
There are many more options you can use with curl, the ones given above are just
some you may require on a regular basis.
Pro Log Analer
Proxy Log Analysis and Reporting Ior Squid and other Proxy Servers
www.manageengine.com/FWAnalyzer